DevOps Buzz


Welcome to the jungle!


Custom DNS nameservers

To specify a nameserver use this following template:
- name: ........
image: ..........
dnsPolicy: "None" #overide all configs

DNS troubleshoot

kubectl create namespace my-namespace
kubectl -n my-namespace create -f - <<EOF
apiVersion: v1
kind: Pod
name: busybox
- name: busybox
image: busybox
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
Then run.
kubectl -n my-namespace exec -ti busybox -- nslookup kubernetes.default

Port forwarding


kubectl -n MY-NAMESPACE port-forward deployment/MY-DEPLOYMENT 8001:8001


kubectl proxy
Then access:

Known errors and solutions

Calico node 'XXXX' is already using the IPv4 address X.X.X.X


When you using kubespray, you remove a node:
ansible-playbook -i inventory/custom/hosts.ini remove-node.yml \
-b -v --extra-vars "node=worker-006"
Then you try to scale the cluster again, using the same node IP but with a different host name.
"calico-node" pod fails with the following error:
2019-03-28 19:53:36.194 [INFO][10] startup.go 251: Early log level set to info
2019-03-28 19:53:36.194 [INFO][10] startup.go 267: Using NODENAME environment for node name
2019-03-28 19:53:36.194 [INFO][10] startup.go 279: Determined node name: fm-mcd-006
2019-03-28 19:53:36.472 [INFO][10] startup.go 101: Skipping datastore connection test
2019-03-28 19:53:36.608 [INFO][10] startup.go 352: Building new node resource Name="fm-mcd-006"
2019-03-28 19:53:36.608 [INFO][10] startup.go 367: Initialize BGP data
2019-03-28 19:53:36.608 [INFO][10] startup.go 456: Using IPv4 address from environment: IP=
2019-03-28 19:53:36.610 [INFO][10] startup.go 489: IPv4 address discovered on interface fm-k8s
2019-03-28 19:53:36.610 [INFO][10] startup.go 432: Node IPv4 changed, will check for conflicts
2019-03-28 19:53:36.701 [WARNING][10] startup.go 861: Calico node 'worker-006' is already using the IPv4 address
2019-03-28 19:53:36.701 [WARNING][10] startup.go 1058: Terminating
Calico node failed to start


SSH to the master node, query etcd and check for the registries related to the "old" host name:
export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/ssl/etcd/ssl/ca.pem
export ETCDCTL_CERT=/etc/ssl/etcd/ssl/node-k8smaster.pem
export ETCDCTL_KEY=/etc/ssl/etcd/ssl/node-k8smaster-key.pem
etcdctl get / --prefix --keys-only | grep "fm-doc-001"
Sample output:
Analyse the registries than delete them:
etcdctl get / --prefix --keys-only | grep "fm-doc-001" | xargs -I {} etcdctl del {}
Delete "calico-node" pod on the new node, it should start normally.
If it does not work, try the following steps:
  • Remove the "old" node;
  • Delete its etcd registries;
  • Scale the cluster with the new node host name.

network plugin is not ready: cni config uninitialized


When trying to scale the cluster, the new node got disconnected in the middle of the process. When it came back, one the the CNI containers was returning the following error(s):
network plugin is not ready: cni config uninitialized
/var/lib/calico/nodename: no such file or directory


SSH to the node you are trying to add, get root, create the following file:
nano /var/lib/calico/nodename
The content of the file should be its node hostname.
The CNI container should come up. If not, try deleting the pod (the replica set will recreated it automatically).