Change serviceSubnet CIDR

Change IP range of your services.
The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.
Since I still did not find the solution, i had to reset the entire cluster with kubeadm reset and init it again.
Scenario
You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24.
Double check if your CIDR conflicts
It is a good idea to check if you new CIDR  does not overlap anything else in your network, for example, your pod subnet.
To do so, run the following python code:
import ipaddr
# Your pod subnet
n1 = ipaddr.IPNetwork('10.244.0.0/16')
# Your new service subnet
n2 = ipaddr.IPNetwork('10.5.0.0/24')
n1.overlaps(n2)
pip install ipaddr if you need.
Dump your current cluster config
SSH to your master node and run:
cd /etc
kubeadm config view > kubeadmconf-2019-06-07.yml
cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml
Add you new service CIDR
Edit your new config file.
cd /etc
nano kubeadmconf-2019-06-07-NEW.yml
Change serviceSubnet config:
...
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  # FROM serviceSubnet: 10.10.0.0/24
  serviceSubnet: 10.5.0.0/24
...
Update certificates
By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: https://github.com/kubernetes/kubeadm/issues/1447​
To do so, follow the steps below.
Check your current certificate
First of all, check your current certificate:
openssl x509 \
  -in /etc/kubernetes/pki/apiserver.crt \
  -text -noout
You will see a section like this one:
...
X509v3 Subject Alternative Name:
  DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88
...                
Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet.
Delete all certificates
mkdir /backup
cp /etc/kubernetes/pki/apiserver.* /backup/
rm /etc/kubernetes/pki/apiserver.*
Generate new certificates
kubeadm init phase certs apiserver \
  --config=/etc/kubeadmconf-2019-06-07-NEW.yml
Check your new certificate and note the first of your new service IP CIDR, in this case,  IP Address:10.5.0.1.
Restart services
Restart daemon and kubelet.
systemctl daemon-reload
systemctl restart kubelet
Restart kube-apiserver docker container.
docker ps | grep apiserver
# get the container name, for example.
docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0
Test it
You can test the new certificates running:
openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text
Change 10.0.0.4 with your server IP.
References
​https://github.com/kubernetes/kubeadm/issues/1447#issuecomment-490494999​
Redeploy kube-dns
Now you have two options. Both will throw an error. Both should have the same result.
Long story short, kubeadm upgrade apply will recreate the kube-dns service for you.
1) Delete kube-dns BEFORE upgrading the cluster.
kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded
Just run the upgrade command again.
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml
2) Delete kube-dns AFTER upgrading the cluster.
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
Delete the service and run the upgrade command again.
kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml
Check the service:
kubectl -n kube-system get service kube-dns
Your kube-dns your CLUSTER-IP should be in your new services CIDR.
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.5.0.10   <none>        53/UDP,53/TCP,9153/TCP   83m
Fix kubelet ConfigMap
Not completely sure how to do it yet.
Edit kubelet ConfigMap:
kubectl get cm -n kube-system kubelet-config-1.16 -o yaml
Fix the DNS IP with your new DNS IP.
Then run.
kubeadm upgrade node
systemctl restart kubelet
Redeploy kubernetes service
This service exposes the API.
Delete it and it should be recreated automatically:
kubectl -n default delete service kubernetes
sleep 3
kubectl -n default get service kubernetes
Your kube-dns your CLUSTER-IP should be in your new services CIDR.
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.5.0.1    <none>        443/TCP   85m
Redeploy the ingress
If you do not have and ingress, go to the next section.
Delete and redeploy it.
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
sleep 3
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
This example is for Azure.
Check the service.
kubectl -n ingress-nginx get service ingress-nginx
The CLUSTER-IP should be in new services CIDR.
NAME            TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.5.0.77   191.238.222.97   80:31022/TCP,443:31035/TCP   67m
References
​https://kubernetes.github.io/ingress-nginx/deploy/#azure​
Redeploy the dashboard
kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
Run kube proxy and test it.
​http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=default​
References
​https://github.com/kubernetes/dashboard​
Redeploy helm and tiller
kubectl -n kube-system delete service tiller-deploy
sleep 3
# Make to force and upgrade
helm init --upgrade --service-account=test
sleep 3
helm init --upgrade --service-account=tiller
Redeploy all your services
Backup and restore all your services, so they will get a new IP in your CIDR.
kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml
Edit the manifest and delete all imutable fields, like uid.
Delete the field clusterIP.
Delete the section status.
Delete and recreate your service.
kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE
kubectl apply YOUR-SERVICE.yml
Backup all you need like Load Balancer info, etc.
Kubeadm - Previous
Cheat Sheet
Next - Kubeadm
Setup cluster
Last modified 3yr ago