DevOps Buzz
Search…
Bash / Shell
Bitbucket
Distros
Elasticsearch
General
Guidelines / Standards
microk8s
Prometheus
RabbitMQ
VirtualBox
Volumes
K8s volumes tips and tricks.

Persistent storage

Create StorageClass

Create the yml file:
1
cat > /tmp/storage-class.yml <<EOF
2
kind: StorageClass
3
apiVersion: storage.k8s.io/v1
4
metadata:
5
name: local-storage
6
provisioner: kubernetes.io/no-provisioner
7
volumeBindingMode: WaitForFirstConsumer
8
9
EOF
Copied!
Deploy it:
1
kubectl create -f /tmp/storage-class.yml
Copied!
name: local-storage will be the reference used in the PersistentVolume and PersistentVolumeClaim. You can change the StorageClass name, but remember to update all references below.

Create PersistentVolume

SSH to your Master node and create the folder which will store the volume:
1
mkdir /mnt/disks/vol1
Copied!
Go back to your kubectl workstation.
Create the yml file:
1
cat > /tmp/persistent-volume.yml <<EOF
2
apiVersion: v1
3
kind: PersistentVolume
4
metadata:
5
name: example-local-pv
6
spec:
7
capacity:
8
storage: 10Gi
9
accessModes:
10
- ReadWriteOnce
11
persistentVolumeReclaimPolicy: Retain
12
storageClassName: local-storage
13
local:
14
path: /mnt/disks/vol1
15
nodeAffinity:
16
required:
17
nodeSelectorTerms:
18
- matchExpressions:
19
- key: kubernetes.io/hostname
20
operator: In
21
values:
22
- minikube
23
24
EOF
Copied!
Deploy it:
1
kubectl create -f /tmp/persistent-volume.yml
Copied!

Create PersistentVolumeClaim

Create the yml file:
1
cat > /tmp/persistent-volume-claim.yml <<EOF
2
kind: PersistentVolumeClaim
3
apiVersion: v1
4
metadata:
5
name: example-local-claim
6
spec:
7
accessModes:
8
- ReadWriteOnce
9
storageClassName: local-storage
10
resources:
11
requests:
12
storage: 5Gi
13
14
EOF
Copied!
Deploy it:
1
kubectl create -f /tmp/persistent-volume-claim.yml
Copied!
if by any chance you omitted storageClassName: local-storage then you need to "patch" the StorageClass first:
kubectl patch storageclass local-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Check if you do not have a default StorageClass patched already, other wise you might see the following error when creating PersistentVolumeClaim:
persistentvolumeclaims "example-local-claim" is forbidden: Internal error occurred: 2 default StorageClasses were found

Deploy a pod using the volume

Create the yml file:
1
cat > /tmp/mysql.yml <<EOF
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
name: mysql
6
labels:
7
app: mysql
8
spec:
9
replicas: 1
10
selector:
11
matchLabels:
12
app: mysql
13
template:
14
metadata:
15
labels:
16
app: mysql
17
spec:
18
containers:
19
- image: mysql:5.6
20
name: mysql
21
env:
22
- name: MYSQL_ROOT_PASSWORD
23
value: 123456
24
ports:
25
- containerPort: 3306
26
name: mysql
27
volumeMounts:
28
- name: mysql-persistent-storage
29
mountPath: /var/lib/mysql
30
volumes:
31
- name: mysql-persistent-storage
32
persistentVolumeClaim:
33
claimName: example-local-claim
34
35
EOF
Copied!
Deploy it:
1
kubectl create -f /tmp/mysql.yml
Copied!

References

Persistent storage (AWS)

1
export AWS_DEFAULT_REGION=ap-southeast-2
2
export AWS_REGION=ap-southeast-2
3
aws ec2 create-volume \
4
--size=100 \
5
--volume-type=gp2 \
6
--availability-zone=ap-southeast-2a
7
8
# aws ec2 delete-volume --volume-id vol-05e2481f9b7668e69
Copied!
1
# StorageClass
2
3
kubectl create -f - <<EOF
4
5
kind: StorageClass
6
apiVersion: storage.k8s.io/v1
7
metadata:
8
name: standard
9
provisioner: kubernetes.io/aws-ebs
10
parameters:
11
type: gp2
12
zone: ap-southeast-2a
13
reclaimPolicy: Retain
14
mountOptions:
15
- debug
16
volumeBindingMode: Immediate
17
18
EOF
Copied!
1
# PersistentVolume
2
3
kubectl create -f - <<EOF
4
5
kind: PersistentVolume
6
apiVersion: v1
7
metadata:
8
name: task-pv
9
spec:
10
capacity:
11
storage: 1Gi
12
accessModes:
13
- ReadWriteOnce
14
awsElasticBlockStore:
15
volumeID: vol-0d3a5237c9c19eb7e
16
fsType: ext4
17
18
EOF
Copied!
1
# PersistentVolumeClaim
2
3
kubectl create -f - <<EOF
4
5
kind: PersistentVolumeClaim
6
apiVersion: v1
7
metadata:
8
name: task-pvc
9
spec:
10
accessModes:
11
- ReadWriteOnce
12
resources:
13
requests:
14
storage: 1Gi
15
16
EOF
Copied!
1
# Pod
2
3
kubectl create -n default -f - <<EOF
4
5
kind: Pod
6
apiVersion: v1
7
metadata:
8
name: task-pod
9
spec:
10
volumes:
11
- name: task-volume
12
persistentVolumeClaim:
13
claimName: task-pvc
14
containers:
15
- name: task-container
16
image: mysql:5.6
17
ports:
18
- containerPort: 3306
19
name: "http-server"
20
volumeMounts:
21
- mountPath: "/var/lib/mysql"
22
name: task-volume
23
24
EOF
Copied!
1
#
Copied!
1
#
Copied!
1
#
Copied!

References

1
K8S AWS Cloud Provider Notes
2
Author: Joe Beda, Heptio ([email protected])
3
Date: 2017-02-14
4
Updated: 2017-03-25
5
6
The AWS Cloud Provider does 2 main things:
7
Enables mounting/unmounting of EBS volumes
8
The master drives attaching/detaching the volume to the VM
9
The kubelet on the VM handles mounting/unmounting and formatting the volume
10
Creates ELBs and security groups to allow those LBs to connect through
11
12
To enable the AWS cloud provider you need to do the following:
13
Add --cloud-provider=aws to the API Server, Controller Manager and every Kubelet.
14
If you are using kubeadm, you can use the kubeadm config file to specify the Cloud Provider to configure during kubeadm init. This will add the right flag to the API Server and Controller Manager. To add it to the kubelet, you need to drop in a file as /etc/systemd/system/kubelet.service.d/20-cloud-provider.conf containing:
15
[Service]
16
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=aws"
17
There is a --cloud-config flag for specifying a cloud provider specific config file. This is generally unneeded.
18
Set a tag on the following resources with a key in the form of kubernetes.io/cluster/<cluster name>; the value is immaterial.
19
All instances
20
One and only one SG for each instance should be tagged. This will be modified as necessary to allow ELBs to access the instance
21
Set up IAM Roles for nodes
22
For the master, you want a policy like this (CloudFormation snippet):
23
Version: '2012-10-17'
24
Statement:
25
- Effect: Allow
26
Action:
27
- ec2:*
28
- elasticloadbalancing:*
29
- ecr:GetAuthorizationToken
30
- ecr:BatchCheckLayerAvailability
31
- ecr:GetDownloadUrlForLayer
32
- ecr:GetRepositoryPolicy
33
- ecr:DescribeRepositories
34
- ecr:ListImages
35
- ecr:BatchGetImage
36
- autoscaling:DescribeAutoScalingGroups
37
- autoscaling:UpdateAutoScalingGroup
38
Resource: "*"
39
ec2:* may be overkill here but I haven’t done the work to narrow it down.
40
For nodes:
41
PolicyDocument:
42
Version: '2012-10-17'
43
Statement:
44
- Effect: Allow
45
Action:
46
- ec2:Describe*
47
- ecr:GetAuthorizationToken
48
- ecr:BatchCheckLayerAvailability
49
- ecr:GetDownloadUrlForLayer
50
- ecr:GetRepositoryPolicy
51
- ecr:DescribeRepositories
52
- ecr:ListImages
53
- ecr:BatchGetImage
54
Resource: "*"
55
Note the * in ec2:Describe*
56
57
It is important that the node name (as seen in kubectl get nodes) match the private-dns-name property of the EC2 instance. The cloud provider uses this to look up the instance ID for a node as necessary. This needs to be done by fetching the local hostname from the metadata server. If something is misconfigured you’ll see errors in the API Server of KCM logs.
58
59
A one liner to use on the `kubeadm init/join` command line:
60
61
--node-name="$(hostname -f 2>/dev/null || curl http://169.254.169.254/latest/meta-data/local-hostname)"
62
63
The nodes should read various metadata from the AWS API (such as their zone and instance type) and hoists those to be labels on the node object in the Kubernetes API.
64
AWS specific Service Annotations
65
There is a set of annotations that can be set on Kubernetes service objects that impact how the Cloud Provider sets up ELBs. These are really only documented right now in the source code. Inspect it directly for ideas on what can be done.
66
Scoping down ec2:*
67
This is untested but it looks, from code inspection, like the following are needed:
68
ec2:DescribeInstances
69
ec2:DescribeSecurityGroups
70
ec2:AttachVolume
71
ec2:DetachVolume
72
ec2:DescribeVolumes
73
Needed if you let k8s provision EBS volumes for you
74
ec2:CreateVolume
75
ec2:DeleteVolume
76
ec2:DescribeSubnets
77
ec2:CreateSecurityGroup
78
ec2:DeleteSecurityGroup
79
ec2:AuthorizeSecurityGroupIngress
80
ec2:RevokeSecurityGroupIngress
81
ec2:CreateTags
82
ec2:DescribeRouteTables
83
Needed if you have the k8s master allocate node cidrs and configure cloud routes
84
ec2:CreateRoute
85
ec2:DeleteRoute
86
ec2:ModifyInstanceAttribute
87
Copied!