Azure

Setup kubeadm K8s cluster on Azure provider.

Requirements

You should have defined (or created) the following resources:

• Azure subscription;

• Resource Group (for all resources like VMs, disks, etc);

• VNET and a VNET Resource Group (if different from the VMs RG);

• Network Security Group.

All steps bellow describe the process to create a single-master and single-node cluster.

Keep in mind you must replace all resources' names with your own definitions.

Select your Azure subscription

We will need to run some AZ CLI command, to do so, make sure you are using your correct subscription during the entire process.

az account set --subscription "MY-SUBSCRIPTION"
az account list

You will see your subscription as "Default: true".

Create an Availability Set

It will be used to configure the Load Balancer later.

az vm availability-set create `
  --name MY-RESOURCE-GROUP `
  --resource-group MY-RESOURCE-GROUP

You can also create a Scalability Set if you are going to use auto scaling or Application Gateway endpoints.

Create a Public IP

This IP will be used in the Load Balancer to connect to the master node.

Using Azure Portal, create a Public IP with the following options.

SKU: Basic
IP address assignment: Static

Create Virtual Machines

First of all, you have to get the ID of the subnet where the VMs will be created.

az network vnet list -g MY-VNET-RESOURCE-GROUP

Identify and get subnet ID. It will be something like:

/subscriptions/d66a4ccf-4568-41fc-8b2f-3d930a88486e/resourceGroups/MY-VNET-RESOURCE-GROUP/providers/Microsoft.Network/virtualNetworks/MY-VNET/subnets/MY-SUBNET

Then, create your VMs.

az vm create `  
  --name vm-master-001 `
  --resource-group MY-RESOURCE-GROUP `
  --image UbuntuLTS `
  --size Standard_F4s `
  --authentication-type password `
  --admin-username ambev `
  --admin-password MySecretPWD! `
  --nsg MY-NSG `
  --nsg-rule SSH `
  --os-disk-size-gb 100 `
  --availability-set MY-AV `
  --subnet /subscriptions/d66a4ccf-4568-41fc-8b2f-3d930a88486e/resourceGroups/MY-VNET-RESOURCE-GROUP/providers/Microsoft.Network/virtualNetworks/MY-VNET/subnets/MY-SUBNET

Create one "master" and one "worker".

Install Docker

SSH to both your servers, master and worker.

Install docker dependencies.

sudo apt-get -y install \    
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common

Setup the repository.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

Install Docker.

sudo apt-get update
sudo apt-get -y install docker-ce=18.06.2~ce~3-0~ubuntu

Check with docker version you should install to support you K8s version: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

Edit Docker service file.

nano /lib/systemd/system/docker.service

Edit Docker service file.

nano /lib/systemd/system/docker.service

Add the --exec-opt native.cgroupdriver=systemdconfig at the end of the line "ExecStart":

ExecStart=/usr/bin/dockerd -H fd:// --exec-opt native.cgroupdriver=systemd

Restart services:

systemctl daemon-reload
systemctl restart docker

Install kubeadm, kubectl and kubelet

SSH to your master node.

Follow the instructions here: https://app.gitbook.com/@devops-buzz/s/public/kubeadm/cheat-sheet#install-kubeadm-kubectl-and-kubelet

Then get the packages versions and copy them. They will be used to install the worker node with exactly the same versions: https://app.gitbook.com/@devops-buzz/s/public/kubeadm/cheat-sheet#get-kubeadm-dependencies-versions

SSH to your worker node, and install the same version: https://app.gitbook.com/@devops-buzz/s/public/~/drafts/-LhMcXSw47Ajn24zlx5O/primary/kubeadm/cheat-sheet#specific-version

Create an AD App registrations

Register an application

On Azure Portal, go to "Azure Active Directory", "App registrations".

Select "New registration". Fill up the form.

Add role assignment

Go to "All services", "Subscriptions" and select your subscription.

Go to "Access control (IAM), "Add", "Add role assignment".

Go to "Role" and "Assign access to". "Select" the application you created.

Go to "Azure Active Directory", "App registrations" and find your application.

Get the "Application (client) ID".

New client secret

Go to "Certificates & secrets", "New client secret".

Fill up the form.

Copy and save the secret value.

The secret value will be displayed just once. Copy and save it now!

Setup cluster

SSH to your master server.

Create config files

Create kubeadm config file.

nano /etc/kubeadm.conf

With the following content.

apiServer:
  certSANs:
  - MY-SERVER-PUBLIC-IP
  - MY-PUBLIC-IP-URL.com
  - MY-CREATED-PUBLIC-IP
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: MY-SERVER-INTERNAL-IP:6443
controllerManager:
  extraArgs:
    cloud-config: /etc/kubernetes/cloud.conf
    cloud-provider: azure
  extraVolumes:
  - hostPath: /etc/kubernetes/cloud.conf
    mountPath: /etc/kubernetes/cloud.conf
    name: cloud
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.14.2
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.240.0.0/16
scheduler: {}

Create cloud config file.

nano /etc/kubernetes/cloud.conf

With the following content.

{
    "cloud":"AzurePublicCloud",
    "tenantId": "Def04b19-7776-4a94-b45b-375c77a8f966",
    "subscriptionId": "d44a4ccf-4578-87fc-8F2d-3d340a88486e",
    "aadClientId": "CCef9bf7-b34e-69be-a1c9-d5be47b846d2",
    "aadClientSecret": "s]ksASDFffaseWuT2MJsafdfdsf18OsdfJzc1",
    "resourceGroup": "MY-RESOURCE-GROUP",
    "location": "brazilsouth",
    "vmType": "Standard_F4s",
    "subnetName": "MY-SUBNET-NAME",
    "securityGroupName": "MY-RESOURCE-GROUP",
    "vnetName": "MY-VNET",
    "vnetResourceGroup": "MY-VNET-RESOURCE-GROUP",
    "routeTableName": "MY-RESOURCE-GROUP",
    "primaryAvailabilitySetName": "MY-RESOURCE-GROUP",
    "primaryScaleSetName": "",
    "cloudProviderBackoff": true,
    "cloudProviderBackoffRetries": 6,
    "cloudProviderBackoffExponent": 1.5,
    "cloudProviderBackoffDuration": 5,
    "cloudProviderBackoffJitter": 1,
    "cloudProviderRatelimit": true,
    "cloudProviderRateLimitQPS": 3,
    "cloudProviderRateLimitBucket": 10,
    "useManagedIdentityExtension": false,
    "userAssignedIdentityID": "",
    "useInstanceMetadata": true,
    "loadBalancerSku": "Basic",
    "excludeMasterFromStandardLB": false,
    "providerVaultName": "",
    "maximumLoadBalancerRuleCount": 250,
    "providerKeyName": "k8s",
    "providerKeyVersion": ""
}

• To get the "tenantId", run az account show and get "tenantId" field.

• To get the "subscriptionId", run az account show and get "id" field.

• "aadClientId" is the "Application (client) ID" from Azure AD Application created above.

• "aadClientSecret" is the Application secret created above.

Init cluster

kubeadm init --config kubeadm.conf

The command above will return the "join" command, like:

kubeadm join YOUR-SERVER-INTERNAL-IP:6443 --token 2m8oey.aeejkbpfgpg0izi0 \
    --discovery-token-ca-cert-hash sha256:bb4527127bbe01f049532c55f681dcf25ce31a1b4ff81797b615b2a247862e75

Save this command in safe place. It will be used to add nodes to your cluster.

Get kubeconfig

SSH to your master node and run as root.

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Deploy Weave CNI

SSH to your master node and run as root.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Azure does not support Calico.

Wait until the master node is "READY".

watch kubectl get pods --all-namespaces

Add nodes

SSH to your node and run the "join" command you got above.

Label nodes (optional)

kubectl label node \
  MY-node-001 node-role.kubernetes.io/worker=worker

Setup Load Balancer

Create any service type "LoadBalancer". This will create create a Load Balancer on Azure called "kubernetes".

Create the manifest file.

nano service.yml

With the following content.

kind: Service
apiVersion: v1
metadata:
  name: first-service
spec:
  selector:
    app: first-deployment
  ports:
  - protocol: TCP
    port: 80
  type: LoadBalancer

Deploy the service.

kubectl create -f service.yml

Go to Azure portal, find the Load Balancer "kuberntes" in your Resource Group.

Wait a few minutes. You should have a "Backend pool" called "kubernets" with your node in it.

Add a "Frontend IP Configuration" and select the public IP you created above.

Create a "Backend pool" add your master server.

Create a "health probe" for you master server on port 6443.

Create a "Load Balancing Rule" using your Public IP on port 6443 to your master pool.

At this point you should be able to use your kubeconfig your public DNS.

Setup static IPs

Go to Azure portal, find your VMs, go to "Networking", lick on the NIC, go to "IP configurations", click on the IP, on "Assignment", set it to "Static".

Find the VMs Public IPs and double check if they are "Static".