DevOps Buzz
Search
K

Change serviceSubnet CIDR

Change IP range of your services.
The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.
Since I still did not find the solution, i had to reset the entire cluster with kubeadm reset and init it again.

Scenario

You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24.

Double check if your CIDR conflicts

It is a good idea to check if you new CIDR does not overlap anything else in your network, for example, your pod subnet.
To do so, run the following python code:
import ipaddr
# Your pod subnet
n1 = ipaddr.IPNetwork('10.244.0.0/16')
# Your new service subnet
n2 = ipaddr.IPNetwork('10.5.0.0/24')
n1.overlaps(n2)
pip install ipaddr if you need.

Dump your current cluster config

SSH to your master node and run:
cd /etc
kubeadm config view > kubeadmconf-2019-06-07.yml
cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml

Add you new service CIDR

Edit your new config file.
cd /etc
nano kubeadmconf-2019-06-07-NEW.yml
Change serviceSubnet config:
...
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
# FROM serviceSubnet: 10.10.0.0/24
serviceSubnet: 10.5.0.0/24
...

Update certificates

By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: https://github.com/kubernetes/kubeadm/issues/1447
To do so, follow the steps below.

Check your current certificate

First of all, check your current certificate:
openssl x509 \
-in /etc/kubernetes/pki/apiserver.crt \
-text -noout
You will see a section like this one:
...
X509v3 Subject Alternative Name:
DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88
...
Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet.

Delete all certificates

mkdir /backup
cp /etc/kubernetes/pki/apiserver.* /backup/
rm /etc/kubernetes/pki/apiserver.*

Generate new certificates

kubeadm init phase certs apiserver \
--config=/etc/kubeadmconf-2019-06-07-NEW.yml
Check your new certificate and note the first of your new service IP CIDR, in this case, IP Address:10.5.0.1.

Restart services

Restart daemon and kubelet.
systemctl daemon-reload
systemctl restart kubelet
Restart kube-apiserver docker container.
docker ps | grep apiserver
# get the container name, for example.
docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0

Test it

You can test the new certificates running:
openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text
Change 10.0.0.4 with your server IP.

References

Redeploy kube-dns

Now you have two options. Both will throw an error. Both should have the same result.
Long story short, kubeadm upgrade apply will recreate the kube-dns service for you.

1) Delete kube-dns BEFORE upgrading the cluster.

kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded
Just run the upgrade command again.
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml

2) Delete kube-dns AFTER upgrading the cluster.

kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
Delete the service and run the upgrade command again.
kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
Check the service:
kubectl -n kube-system get service kube-dns
Your kube-dns your CLUSTER-IP should be in your new services CIDR.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.5.0.10 <none> 53/UDP,53/TCP,9153/TCP 83m

Fix kubelet ConfigMap

Not completely sure how to do it yet.
Edit kubelet ConfigMap:
kubectl get cm -n kube-system kubelet-config-1.16 -o yaml
Fix the DNS IP with your new DNS IP.
Then run.
kubeadm upgrade node
systemctl restart kubelet

Redeploy kubernetes service

This service exposes the API.
Delete it and it should be recreated automatically:
kubectl -n default delete service kubernetes
sleep 3
kubectl -n default get service kubernetes
Your kube-dns your CLUSTER-IP should be in your new services CIDR.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.5.0.1 <none> 443/TCP 85m

Redeploy the ingress

If you do not have and ingress, go to the next section.
Delete and redeploy it.
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
sleep 3
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
This example is for Azure.
Check the service.
kubectl -n ingress-nginx get service ingress-nginx
The CLUSTER-IP should be in new services CIDR.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.5.0.77 191.238.222.97 80:31022/TCP,443:31035/TCP 67m

References

Redeploy the dashboard

kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
Run kube proxy and test it.

References

Redeploy helm and tiller
kubectl -n kube-system delete service tiller-deploy
sleep 3
# Make to force and upgrade
helm init --upgrade --service-account=test
sleep 3
helm init --upgrade --service-account=tiller

Redeploy all your services

Backup and restore all your services, so they will get a new IP in your CIDR.
kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml
Edit the manifest and delete all imutable fields, like uid.
Delete the field clusterIP.
Delete the section status.
Delete and recreate your service.
kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE
kubectl apply YOUR-SERVICE.yml
Backup all you need like Load Balancer info, etc.