Change serviceSubnet CIDR
Change IP range of your services.
The following process has a problem: after everything configured, the pods come up with the old IP as a DNS nameserver in /etc/resolv.conf.
Since I still did not find the solution, i had to reset the entire cluster with
kubeadm reset
and init
it again.You have your cluster up and running services, let's say, on CIDR 10.10.0.0/24 and you want to change it to 10.5.0.0/24.
It is a good idea to check if you new CIDR does not overlap anything else in your network, for example, your pod subnet.
To do so, run the following python code:
import ipaddr
# Your pod subnet
n1 = ipaddr.IPNetwork('10.244.0.0/16')
# Your new service subnet
n2 = ipaddr.IPNetwork('10.5.0.0/24')
n1.overlaps(n2)
pip install ipaddr
if you need.SSH to your master node and run:
cd /etc
kubeadm config view > kubeadmconf-2019-06-07.yml
cp kubeadmconf-2019-06-07.yml kubeadmconf-2019-06-07-NEW.yml
Edit your new config file.
cd /etc
nano kubeadmconf-2019-06-07-NEW.yml
Change
serviceSubnet
config:...
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
# FROM serviceSubnet: 10.10.0.0/24
serviceSubnet: 10.5.0.0/24
...
By the time of this writing (2019-06-07), "kubeadm upgrade" does not support updating API server certSANs: https://github.com/kubernetes/kubeadm/issues/1447
To do so, follow the steps below.
First of all, check your current certificate:
openssl x509 \
-in /etc/kubernetes/pki/apiserver.crt \
-text -noout
You will see a section like this one:
...
X509v3 Subject Alternative Name:
DNS:k8s-non-prod-001-master-001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:k8s-non-prod-001-master.brazilsouth.cloudapp.azure.com, IP Address:10.10.0.1, IP Address:10.0.0.4, IP Address:10.0.0.4, IP Address:191.234.160.212, IP Address:191.238.210.88
...
Check all "IP Address" sections. You will note your new service CIDR IPs are not there yet.
mkdir /backup
cp /etc/kubernetes/pki/apiserver.* /backup/
rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs apiserver \
--config=/etc/kubeadmconf-2019-06-07-NEW.yml
Check your new certificate and note the first of your new service IP CIDR, in this case,
IP Address:10.5.0.1
.Restart daemon and kubelet.
systemctl daemon-reload
systemctl restart kubelet
Restart kube-apiserver docker container.
docker ps | grep apiserver
# get the container name, for example.
docker restart k8s_kube-apiserver_kube-apiserver-k8s-non-prod-001-master-001_kube-system_fc4ca5d2a58c3647572c064b74f7c5a4_0
You can test the new certificates running:
openssl s_client -connect 10.0.0.4:6443 | openssl x509 -noout -text
Change
10.0.0.4
with your server IP.Now you have two options. Both will throw an error. Both should have the same result.
Long story short,
kubeadm upgrade apply
will recreate the kube-dns service for you.kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/apply] FATAL: failed to retrieve the current etcd version: context deadline exceeded
Just run the upgrade command again.
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
You will see this error:
[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
Delete the service and run the upgrade command again.
kubectl -n kube-system delete service kube-dns
kubeadm upgrade apply \
--config /etc/kubeadmconf-2019-06-07-NEW.yml
Check the service:
kubectl -n kube-system get service kube-dns
Your
kube-dns
your CLUSTER-IP
should be in your new services CIDR.NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.5.0.10 <none> 53/UDP,53/TCP,9153/TCP 83m
Not completely sure how to do it yet.
Edit kubelet ConfigMap:
kubectl get cm -n kube-system kubelet-config-1.16 -o yaml
Fix the DNS IP with your new DNS IP.
Then run.
kubeadm upgrade node
systemctl restart kubelet
This service exposes the API.
Delete it and it should be recreated automatically:
kubectl -n default delete service kubernetes
sleep 3
kubectl -n default get service kubernetes
Your
kube-dns
your CLUSTER-IP
should be in your new services CIDR.NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.5.0.1 <none> 443/TCP 85m
If you do not have and ingress, go to the next section.
Delete and redeploy it.
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
sleep 3
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
This example is for Azure.
Check the service.
kubectl -n ingress-nginx get service ingress-nginx
The
CLUSTER-IP
should be in new services CIDR.NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.5.0.77 191.238.222.97 80:31022/TCP,443:31035/TCP 67m
kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
sleep 3
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
Run
kube proxy
and test it.Redeploy helm and tiller
kubectl -n kube-system delete service tiller-deploy
sleep 3
# Make to force and upgrade
helm init --upgrade --service-account=test
sleep 3
helm init --upgrade --service-account=tiller
Backup and restore all your services, so they will get a new IP in your CIDR.
kubectl -n YOUR-NAMESPACE get service YOUR-SERVICE -o yaml > YOUR-SERVICE.yml
Edit the manifest and delete all imutable fields, like
uid
.Delete the field
clusterIP
.Delete the section
status
.Delete and recreate your service.
kubectl -n YOUR-NAMESPACE delete service YOUR-SERVICE
kubectl apply YOUR-SERVICE.yml
Backup all you need like Load Balancer info, etc.
Last modified 3yr ago