Volumes
K8s volumes tips and tricks.
Create the yml file:
cat > /tmp/storage-class.yml <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF
Deploy it:
kubectl create -f /tmp/storage-class.yml
name: local-storage will be the reference used in the PersistentVolume and PersistentVolumeClaim. You can change the StorageClass name, but remember to update all references below.SSH to your Master node and create the folder which will store the volume:
mkdir /mnt/disks/vol1
Go back to your kubectl workstation.
Create the yml file:
cat > /tmp/persistent-volume.yml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-local-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /mnt/disks/vol1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
EOF
Deploy it:
kubectl create -f /tmp/persistent-volume.yml
Create the yml file:
cat > /tmp/persistent-volume-claim.yml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: example-local-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-storage
resources:
requests:
storage: 5Gi
EOF
Deploy it:
kubectl create -f /tmp/persistent-volume-claim.yml
if by any chance you omitted
storageClassName: local-storage then you need to "patch" the StorageClass first:kubectl patch storageclass local-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'Check if you do not have a default
StorageClass patched already, other wise you might see the following error when creating PersistentVolumeClaim:persistentvolumeclaims "example-local-claim" is forbidden: Internal error occurred: 2 default StorageClasses were foundCreate the yml file:
cat > /tmp/mysql.yml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
labels:
app: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: 123456
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: example-local-claim
EOF
Deploy it:
kubectl create -f /tmp/mysql.yml
export AWS_DEFAULT_REGION=ap-southeast-2
export AWS_REGION=ap-southeast-2
aws ec2 create-volume \
--size=100 \
--volume-type=gp2 \
--availability-zone=ap-southeast-2a
# aws ec2 delete-volume --volume-id vol-05e2481f9b7668e69
# StorageClass
kubectl create -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: ap-southeast-2a
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
EOF
# PersistentVolume
kubectl create -f - <<EOF
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: vol-0d3a5237c9c19eb7e
fsType: ext4
EOF
# PersistentVolumeClaim
kubectl create -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: task-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
# Pod
kubectl create -n default -f - <<EOF
kind: Pod
apiVersion: v1
metadata:
name: task-pod
spec:
volumes:
- name: task-volume
persistentVolumeClaim:
claimName: task-pvc
containers:
- name: task-container
image: mysql:5.6
ports:
- containerPort: 3306
name: "http-server"
volumeMounts:
- mountPath: "/var/lib/mysql"
name: task-volume
EOF
#
#
#
K8S AWS Cloud Provider Notes
Author: Joe Beda, Heptio ([email protected])
Date: 2017-02-14
Updated: 2017-03-25
The AWS Cloud Provider does 2 main things:
Enables mounting/unmounting of EBS volumes
The master drives attaching/detaching the volume to the VM
The kubelet on the VM handles mounting/unmounting and formatting the volume
Creates ELBs and security groups to allow those LBs to connect through
To enable the AWS cloud provider you need to do the following:
Add --cloud-provider=aws to the API Server, Controller Manager and every Kubelet.
If you are using kubeadm, you can use the kubeadm config file to specify the Cloud Provider to configure during kubeadm init. This will add the right flag to the API Server and Controller Manager. To add it to the kubelet, you need to drop in a file as /etc/systemd/system/kubelet.service.d/20-cloud-provider.conf containing:
[Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=aws"
There is a --cloud-config flag for specifying a cloud provider specific config file. This is generally unneeded.
Set a tag on the following resources with a key in the form of kubernetes.io/cluster/<cluster name>; the value is immaterial.
All instances
One and only one SG for each instance should be tagged. This will be modified as necessary to allow ELBs to access the instance
Set up IAM Roles for nodes
For the master, you want a policy like this (CloudFormation snippet):
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ec2:*
- elasticloadbalancing:*
- ecr:GetAuthorizationToken
- ecr:BatchCheckLayerAvailability
- ecr:GetDownloadUrlForLayer
- ecr:GetRepositoryPolicy
- ecr:DescribeRepositories
- ecr:ListImages
- ecr:BatchGetImage
- autoscaling:DescribeAutoScalingGroups
- autoscaling:UpdateAutoScalingGroup
Resource: "*"
ec2:* may be overkill here but I haven’t done the work to narrow it down.
For nodes:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- ec2:Describe*
- ecr:GetAuthorizationToken
- ecr:BatchCheckLayerAvailability
- ecr:GetDownloadUrlForLayer
- ecr:GetRepositoryPolicy
- ecr:DescribeRepositories
- ecr:ListImages
- ecr:BatchGetImage
Resource: "*"
Note the * in ec2:Describe*
It is important that the node name (as seen in kubectl get nodes) match the private-dns-name property of the EC2 instance. The cloud provider uses this to look up the instance ID for a node as necessary. This needs to be done by fetching the local hostname from the metadata server. If something is misconfigured you’ll see errors in the API Server of KCM logs.
A one liner to use on the `kubeadm init/join` command line:
--node-name="$(hostname -f 2>/dev/null || curl http://169.254.169.254/latest/meta-data/local-hostname)"
The nodes should read various metadata from the AWS API (such as their zone and instance type) and hoists those to be labels on the node object in the Kubernetes API.
AWS specific Service Annotations
There is a set of annotations that can be set on Kubernetes service objects that impact how the Cloud Provider sets up ELBs. These are really only documented right now in the source code. Inspect it directly for ideas on what can be done.
Scoping down ec2:*
This is untested but it looks, from code inspection, like the following are needed:
ec2:DescribeInstances
ec2:DescribeSecurityGroups
ec2:AttachVolume
ec2:DetachVolume
ec2:DescribeVolumes
Needed if you let k8s provision EBS volumes for you
ec2:CreateVolume
ec2:DeleteVolume
ec2:DescribeSubnets
ec2:CreateSecurityGroup
ec2:DeleteSecurityGroup
ec2:AuthorizeSecurityGroupIngress
ec2:RevokeSecurityGroupIngress
ec2:CreateTags
ec2:DescribeRouteTables
Needed if you have the k8s master allocate node cidrs and configure cloud routes
ec2:CreateRoute
ec2:DeleteRoute
ec2:ModifyInstanceAttribute
Last modified 4yr ago