Change serviceSubnet CIDR

Change IP range of your services.

The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.

Since I still did not find the solution, i had to reset the entire cluster with kubeadm reset and init it again.

Scenario

You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24.

Double check if your CIDR conflicts

It is a good idea to check if you new CIDR does not overlap anything else in your network, for example, your pod subnet.

To do so, run the following python code:

import ipaddr
# Your pod subnet
n1 = ipaddr.IPNetwork('10.244.0.0/16')
# Your new service subnet
n2 = ipaddr.IPNetwork('10.5.0.0/24')
n1.overlaps(n2)

pip install ipaddr if you need.

Dump your current cluster config

SSH to your master node and run:

cd /etc
kubeadm config view > kubeadmconf-2019-06-07.yml
cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml

Add you new service CIDR

Edit your new config file.

cd /etc
nano kubeadmconf-2019-06-07-NEW.yml

Change serviceSubnet config:

...
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  # FROM serviceSubnet: 10.10.0.0/24
  serviceSubnet: 10.5.0.0/24
...

Update certificates

By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: https://github.com/kubernetes/kubeadm/issues/1447

To do so, follow the steps below.

Check your current certificate

First of all, check your current certificate:

openssl x509 \
  -in /etc/kubernetes/pki/apiserver.crt \
  -text -noout

You will see a section like this one:

...
X509v3 Subject Alternative Name:
  DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88
...                

Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet.

Delete all certificates

mkdir /backup
cp /etc/kubernetes/pki/apiserver.* /backup/
rm /etc/kubernetes/pki/apiserver.*

Generate new certificates

kubeadm init phase certs apiserver \
  --config=/etc/kubeadmconf-2019-06-07-NEW.yml

Check your new certificate and note the first of your new service IP CIDR, in this case, IP Address:10.5.0.1.

Restart services

Restart daemon and kubelet.

systemctl daemon-reload
systemctl restart kubelet

Restart kube-apiserver docker container.

docker ps | grep apiserver
# get the container name, for example.
docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0

Test it

You can test the new certificates running:

openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text

Change 10.0.0.4 with your server IP.

References

https://github.com/kubernetes/kubeadm/issues/1447#issuecomment-490494999

Redeploy kube-dns

Now you have two options. Both will throw an error. Both should have the same result.

Long story short, kubeadm upgrade apply will recreate the kube-dns service for you.

1) Delete kube-dns BEFORE upgrading the cluster.

kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml

You will see this error:

[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded

Just run the upgrade command again.

kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml

2) Delete kube-dns AFTER upgrading the cluster.

kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml

You will see this error:

[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable

Delete the service and run the upgrade command again.

kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
  --config /etc/kubeadmconf-2019-06-07-NEW.yml

Check the service:

kubectl -n kube-system get service kube-dns

Your kube-dns your CLUSTER-IP should be in your new services CIDR.

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.5.0.10   <none>        53/UDP,53/TCP,9153/TCP   83m

Fix kubelet ConfigMap

Not completely sure how to do it yet.

Edit kubelet ConfigMap:

kubectl get cm -n kube-system kubelet-config-1.16 -o yaml

Fix the DNS IP with your new DNS IP.

Then run.

kubeadm upgrade node
systemctl restart kubelet

Redeploy kubernetes service

This service exposes the API.

Delete it and it should be recreated automatically:

kubectl -n default delete service kubernetes
sleep 3
kubectl -n default get service kubernetes

Your kube-dns your CLUSTER-IP should be in your new services CIDR.

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.5.0.1    <none>        443/TCP   85m

Redeploy the ingress

If you do not have and ingress, go to the next section.

Delete and redeploy it.

kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
sleep 3
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml

This example is for Azure.

Check the service.

kubectl -n ingress-nginx get service ingress-nginx

The CLUSTER-IP should be in new services CIDR.

NAME            TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.5.0.77   191.238.222.97   80:31022/TCP,443:31035/TCP   67m

References

https://kubernetes.github.io/ingress-nginx/deploy/#azure

Redeploy the dashboard

kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

Run kube proxy and test it.

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/overview?namespace=default

References

https://github.com/kubernetes/dashboard

Redeploy helm and tiller

kubectl -n kube-system delete service tiller-deploy
sleep 3
# Make to force and upgrade
helm init --upgrade --service-account=test
sleep 3
helm init --upgrade --service-account=tiller

Redeploy all your services

Backup and restore all your services, so they will get a new IP in your CIDR.

kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml

Edit the manifest and delete all imutable fields, like uid.

Delete the field clusterIP.

Delete the section status.

Delete and recreate your service.

kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE
kubectl apply YOUR-SERVICE.yml

Backup all you need like Load Balancer info, etc.