K8S AWS Cloud Provider Notes
The AWS Cloud Provider does 2 main things:
Enables mounting/unmounting of EBS volumes
The master drives attaching/detaching the volume to the VM
The kubelet on the VM handles mounting/unmounting and formatting the volume
Creates ELBs and security groups to allow those LBs to connect through
To enable the AWS cloud provider you need to do the following:
Add --cloud-provider=aws to the API Server, Controller Manager and every Kubelet.
If you are using kubeadm, you can use the kubeadm config file to specify the Cloud Provider to configure during kubeadm init. This will add the right flag to the API Server and Controller Manager. To add it to the kubelet, you need to drop in a file as /etc/systemd/system/kubelet.service.d/20-cloud-provider.conf containing:
There is a --cloud-config flag for specifying a cloud provider specific config file. This is generally unneeded.
Set a tag on the following resources with a key in the form of kubernetes.io/cluster/<cluster name>; the value is immaterial.
One and only one SG for each instance should be tagged. This will be modified as necessary to allow ELBs to access the instance
Set up IAM Roles for nodes
For the master, you want a policy like this (CloudFormation snippet):
ec2:* may be overkill here but I haven’t done the work to narrow it down.
Note the * in ec2:Describe*
It is important that the node name (as seen in kubectl get nodes) match the private-dns-name property of the EC2 instance. The cloud provider uses this to look up the instance ID for a node as necessary. This needs to be done by fetching the local hostname from the metadata server. If something is misconfigured you’ll see errors in the API Server of KCM logs.
A one liner to use on the `kubeadm init/join` command line:
--node-name="$(hostname -f 2>/dev/null || curl http://169.254.169.254/latest/meta-data/local-hostname)"
The nodes should read various metadata from the AWS API (such as their zone and instance type) and hoists those to be labels on the node object in the Kubernetes API.
AWS specific Service Annotations
There is a set of annotations that can be set on Kubernetes service objects that impact how the Cloud Provider sets up ELBs. These are really only documented right now in the source code. Inspect it directly for ideas on what can be done.
This is untested but it looks, from code inspection, like the following are needed:
Needed if you let k8s provision EBS volumes for you
Needed if you have the k8s master allocate node cidrs and configure cloud routes