# Change serviceSubnet CIDR {% hint style="danger" %} The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf. Since I still did not find the solution, i had to reset the entire cluster with `kubeadm reset` and `init` it again. {% endhint %} ## Scenario You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24. ## Double check if your CIDR conflicts It is a good idea to check if you new CIDR does not overlap anything else in your network, for example, your pod subnet. To do so, run the following python code: ```python import ipaddr # Your pod subnet n1 = ipaddr.IPNetwork('10.244.0.0/16') # Your new service subnet n2 = ipaddr.IPNetwork('10.5.0.0/24') n1.overlaps(n2) ``` {% hint style="info" %} `pip install ipaddr` if you need. {% endhint %} ## Dump your current cluster config SSH to your master node and run: ``` cd /etc kubeadm config view > kubeadmconf-2019-06-07.yml cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml ``` ## Add you new service CIDR Edit your new config file. ``` cd /etc nano kubeadmconf-2019-06-07-NEW.yml ``` Change `serviceSubnet` config: ``` ... networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 # FROM serviceSubnet: 10.10.0.0/24 serviceSubnet: 10.5.0.0/24 ... ``` ## Update certificates By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: To do so, follow the steps below. ### Check your current certificate First of all, check your current certificate: ```bash openssl x509 \ -in /etc/kubernetes/pki/apiserver.crt \ -text -noout ``` You will see a section like this one: ```bash ... X509v3 Subject Alternative Name: DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88 ... ``` Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet. ### Delete all certificates ```bash mkdir /backup cp /etc/kubernetes/pki/apiserver.* /backup/ rm /etc/kubernetes/pki/apiserver.* ``` ### Generate new certificates ```bash kubeadm init phase certs apiserver \ --config=/etc/kubeadmconf-2019-06-07-NEW.yml ``` Check your new certificate and note the first of your new service IP CIDR, in this case, `IP Address:10.5.0.1`. ### Restart services Restart daemon and kubelet. ```bash systemctl daemon-reload systemctl restart kubelet ``` Restart kube-apiserver docker container. ```bash docker ps | grep apiserver # get the container name, for example. docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0 ``` ### Test it You can test the new certificates running: ```bash openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text ``` {% hint style="warning" %} Change `10.0.0.4` with your server IP. {% endhint %} ### References ## Redeploy kube-dns Now you have two options. Both will throw an error. Both should have the same result. Long story short, `kubeadm upgrade apply` will recreate the kube-dns service for you. ### 1) Delete kube-dns BEFORE upgrading the cluster. ```bash kubectl -n kube-system delete service kube-dns kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` You will see this error: {% hint style="warning" %} \[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded {% endhint %} Just run the upgrade command again. ```bash kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` ### 2) Delete kube-dns AFTER upgrading the cluster. ```bash kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` You will see this error: {% hint style="warning" %} \[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable {% endhint %} Delete the service and run the upgrade command again. ```bash kubectl -n kube-system delete service kube-dns kubeadm upgrade apply \ --config /etc/kubeadmconf-2019-06-07-NEW.yml ``` Check the service: ```bash kubectl -n kube-system get service kube-dns ``` Your `kube-dns` your `CLUSTER-IP` should be in your new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.5.0.10 53/UDP,53/TCP,9153/TCP 83m ``` ## Fix kubelet ConfigMap {% hint style="warning" %} Not completely sure how to do it yet. {% endhint %} Edit kubelet ConfigMap: ``` kubectl get cm -n kube-system kubelet-config-1.16 -o yaml ``` Fix the DNS IP with your new DNS IP. Then run. ``` kubeadm upgrade node systemctl restart kubelet ``` ## Redeploy kubernetes service This service exposes the API. Delete it and it should be recreated automatically: ``` kubectl -n default delete service kubernetes sleep 3 kubectl -n default get service kubernetes ``` Your `kube-dns` your `CLUSTER-IP` should be in your new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.5.0.1 443/TCP 85m ``` ## Redeploy the ingress If you do not have and ingress, go to the next section. Delete and redeploy it. ```bash kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml sleep 3 kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml ``` {% hint style="info" %} This example is for Azure. {% endhint %} Check the service. ```bash kubectl -n ingress-nginx get service ingress-nginx ``` The `CLUSTER-IP` should be in new services CIDR. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx LoadBalancer 10.5.0.77 191.238.222.97 80:31022/TCP,443:31035/TCP 67m ``` ### References ## Redeploy the dashboard ```bash kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml sleep 3 kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml ``` Run `kube proxy` and test it. ### References Redeploy helm and tiller ```bash kubectl -n kube-system delete service tiller-deploy sleep 3 # Make to force and upgrade helm init --upgrade --service-account=test sleep 3 helm init --upgrade --service-account=tiller ``` ## Redeploy all your services Backup and restore all your services, so they will get a new IP in your CIDR. ``` kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml ``` Edit the manifest and delete all imutable fields, like `uid`. Delete the field `clusterIP`. Delete the section `status`. Delete and recreate your service. ``` kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE kubectl apply YOUR-SERVICE.yml ``` {% hint style="warning" %} Backup all you need like Load Balancer info, etc. {% endhint %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://www.devops.buzz/public/kubeadm/change-servicesubnet-cidr.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.