Bare-metal

Install a cluster using Kubeadm on bare-metal servers

Install docker

Check which Docker version Kubernetes supports: https://kubernetes.io/docs/setup/cri/#docker

By the time of this writing, the latest supported docker version is 18.06.2.

Install dependencies:

sudo apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

Add the repo:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

Install Docker:

sudo apt-get update
sudo apt-get -y install docker-ce=18.06.2~ce~3-0~ubuntu

Replace 18.06.2 with the latest supported docker version.

Setup firewall

Master

Nodes

References

https://kubernetes.io/docs/setup/independent/install-kubeadm/

Install kubelet, kubeadm and kubectl

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

Init cluster

Make sure you server's host name is configured (avoid changing the host name after the cluster is created).

kubeadm init \
  --pod-network-cidr=172.31.0.0/20 \
  --apiserver-advertise-address=0.0.0.0 \
  --apiserver-cert-extra-sans=13.211.85.85

Copy and save the kubeadm join command.

Configure your user to run kubectl:

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Deploy Weave Net:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

By the time of this writing, Kubeadm has an issue. Coredns will remaing "pending" until you deploy Weave Net: https://github.com/kubernetes/kubeadm/issues/980

Confirm that all of the pods are running with the following command.

watch kubectl get pods --all-namespaces

Wait until each pod has the STATUS of Running.

Confirm that master node is ready:

kubectl get nodes

If you want to deploy Calico instead, check the latest documentation: https://docs.projectcalico.org/v3.5/getting-started/kubernetes/

References

https://dzone.com/articles/deploying-kubernetes-dashboard-to-a-kubeadm-create https://zihao.me/post/creating-a-kubernetes-cluster-from-scratch-with-kubeadm/ https://chrislovecnm.com/kubernetes/cni/choosing-a-cni-provider/ https://docs.projectcalico.org/v3.5/getting-started/kubernetes/

Dashboard

Deploy the dashboard:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

In your workstation...

kubectl proxy

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

Create dashboard user: https://devops-buzz.gitbook.io/public/kubernetes/dashboard#create-user

Join node

Install kubelet, kubeadm and kubectl.

Setup Docker cgroupdriver systemd

Check docker systemd service config file:

systemctl cat docker

Sample output:

/lib/systemd/system/docker.service

Edit the file:

nano /lib/systemd/system/docker.service

Add --exec-opt native.cgroupdriver=systemd to the ExecStart option:

ExecStart=/usr/bin/dockerd -H fd:// --exec-opt native.cgroupdriver=systemd

Restart daemon:

systemctl daemon-reload
systemctl restart docker

Add node to the cluster

The worker node must have exactly the same master versions of the following packages: kubelet, kubeadm, kubectl, kubernetes-cni.

Reference here.

Install packages, for example:

apt-get install \
  kubelet=1.13.4-00 \
  kubeadm=1.13.4-00 \
  kubectl=1.13.4-00 \
  kubernetes-cni=0.6.0-00

Hold packages versions:

apt-mark hold kubelet kubeadm kubectl kubernetes-cni

Make sure the master node firewall allows the new node to access it on port 6443.

Make sure you server's host name is configured (avoid changing the host name after).

Run the joint command:

kubeadm join 172.31.7.247:6443 --token g651e8.p09een664u224v76 --discovery-token-ca-cert-hash sha256:b589d4690ac427c5b7046233963d058e2e2cb99f168f90d58ec15a351aa4243b

Use your cluster join command.

If by any chance you need to run the join command more than one time and you the certificate FileAvailable--etc-kubernetes-pki-ca.crt, run this command before:

rm -fr /etc/kubernetes/*

You should see this output:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

Go to your master server and label the new node:

kubectl label node PUT-YOURNODE-HOSTNAME-HERE node-role.kubernetes.io/worker=worker

Remove node

List your current nodes:

kubectl get nodes

Output example:

NAME              STATUS   ROLES    AGE   VERSION
ip-172-31-7-247   Ready    master   46h   v1.13.4
ip-172-31-9-145   Ready    worker   45h   v1.13.4

Let's suppose you want to remove the node ip-172-31-9-145:

kubectl drain ip-172-31-9-145 --delete-local-data --force --ignore-daemonsets

Check node status:

kubectl get nodes

Output example:

NAME              STATUS                     ROLES    AGE   VERSION
ip-172-31-7-247   Ready                      master   46h   v1.13.4
ip-172-31-9-145   Ready,SchedulingDisabled   worker   45h   v1.13.4

Then, on the node being removed, reset all kubeadm installed state::

kubeadm reset

The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually:

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If you want to reset the IPVS tables, you must run the following command:

ipvsadm -C

Kubeadm over VPN

kubeadm init \
  --apiserver-advertise-address=10.187.216.232 \
  --pod-network-cidr=10.32.0.0/12 \
  --service-cidr=10.96.0.0/12

https://github.com/kubernetes/kubeadm/issues/113