> For the complete documentation index, see [llms.txt](https://www.devops.buzz/public/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.devops.buzz/public/kubeadm/change-servicesubnet-cidr.md). # Change serviceSubnet CIDR {% hint style="danger" %} The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf. Since I still did not find the solution, i had to reset the entire cluster with `kubeadm reset` and `init` it again. {% endhint %} ## Scenario You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24. ## Double check if your CIDR conflicts It is a good idea to check if you new CIDR does not overlap anything else in your network, for example, your pod subnet. To do so, run the following python code: ```python import ipaddr # Your pod subnet n1 = ipaddr.IPNetwork('10.244.0.0/16') # Your new service subnet n2 = ipaddr.IPNetwork('10.5.0.0/24') n1.overlaps(n2) ``` {% hint style="info" %} `pip install ipaddr` if you need. {% endhint %} ## Dump your current cluster config SSH to your master node and run: ``` cd /etc kubeadm config view > kubeadmconf-2019-06-07.yml cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml ``` ## Add you new service CIDR Edit your new config file. ``` cd /etc nano kubeadmconf-2019-06-07-NEW.yml ``` Change `serviceSubnet` config: ``` ... networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 # FROM serviceSubnet: 10.10.0.0/24 serviceSubnet: 10.5.0.0/24 ... ``` ## Update certificates By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: To do so, follow the steps below. ### Check your current certificate First of all, check your current certificate: ```bash openssl x509 \ -in /etc/kubernetes/pki/apiserver.crt \ -text -noout ``` You will see a section like this one: ```bash ... X509v3 Subject Alternative Name: DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88 ... ``` Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet. ### Delete all certificates ```bash mkdir /backup cp /etc/kubernetes/pki/apiserver.* /backup/ rm /etc/kubernetes/pki/apiserver.* ``` ### Generate new certificates ```bash kubeadm init phase certs apiserver \ --config=/etc/kubeadmconf-2019-06-07-NEW.yml ``` Check your new certificate and note the first of your new service IP CIDR, in this case, `IP Address:10.5.0.1`. ### Restart services Restart daemon and kubelet. ```bash systemctl daemon-reload systemctl restart kubelet ``` Restart kube-apiserver docker container. ```bash docker ps | grep apiserver # get the container name, for example. docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0 ``` ### Test it You can test the new certificates running: ```bash openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text ``` {% hint style="warning" %} Change `10.0.0.4` with your server IP. {% endhint %} ### References ## Redeploy kube-dns Now you have two options. Both will throw an error. Both should have the same result. Long story short, `kubeadm upgrade apply` will recreate the kube-dns service for you. ### 1) Delete kube-dns BEFORE upgrading the cluster. ```bash kubectl -n kube-system delete service kube-dns kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` You will see this error: {% hint style="warning" %} \[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded {% endhint %} Just run the upgrade command again. ```bash kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` ### 2) Delete kube-dns AFTER upgrading the cluster. ```bash kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` You will see this error: {% hint style="warning" %} \[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable {% endhint %} Delete the service and run the upgrade command again. ```bash kubectl -n kube-system delete service kube-dns kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` Check the service: ```bash kubectl -n kube-system get service kube-dns ``` Your `kube-dns` your `CLUSTER-IP` should be in your new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.5.0.10 53/UDP,53/TCP,9153/TCP 83m ``` ## Fix kubelet ConfigMap {% hint style="warning" %} Not completely sure how to do it yet. {% endhint %} Edit kubelet ConfigMap: ``` kubectl get cm -n kube-system kubelet-config-1.16 -o yaml ``` Fix the DNS IP with your new DNS IP. Then run. ``` kubeadm upgrade node systemctl restart kubelet ``` ## Redeploy kubernetes service This service exposes the API. Delete it and it should be recreated automatically: ``` kubectl -n default delete service kubernetes sleep 3 kubectl -n default get service kubernetes ``` Your `kube-dns` your `CLUSTER-IP` should be in your new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.5.0.1 443/TCP 85m ``` ## Redeploy the ingress If you do not have and ingress, go to the next section. Delete and redeploy it. ```bash kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml sleep 3 kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml ``` {% hint style="info" %} This example is for Azure. {% endhint %} Check the service. ```bash kubectl -n ingress-nginx get service ingress-nginx ``` The `CLUSTER-IP` should be in new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx LoadBalancer 10.5.0.77 191.238.222.97 80:31022/TCP,443:31035/TCP 67m ``` ### References ## Redeploy the dashboard ```bash kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml ``` Run `kube proxy` and test it. ### References Redeploy helm and tiller ```bash kubectl -n kube-system delete service tiller-deploy sleep 3 # Make to force and upgrade helm init --upgrade --service-account=test sleep 3 helm init --upgrade --service-account=tiller ``` ## Redeploy all your services Backup and restore all your services, so they will get a new IP in your CIDR. ``` kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml ``` Edit the manifest and delete all imutable fields, like `uid`. Delete the field `clusterIP`. Delete the section `status`. Delete and recreate your service. ``` kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE kubectl apply YOUR-SERVICE.yml ``` {% hint style="warning" %} Backup all you need like Load Balancer info, etc. {% endhint %}