DevOps Buzz
Search…
Bash / Shell
Bitbucket
Distros
Elasticsearch
General
Guidelines / Standards
microk8s
Prometheus
RabbitMQ
VirtualBox
Backup
AKS backup tricks.

AKS backup with velero

Install velero

Set env vars.
1
2
# Velero Azure Storace Account Name (must be unique)
3
VELERO_SA_NAME="myaksbackup"
4
5
# Velero Azure Storace Account Resource Group
6
VELERO_RG="rg-aks-backup"
7
8
# Velero Azure Storace Account Region
9
VELERO_REGION="eastus"
10
11
# Velero Azure Storace Account Subscription
12
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"
Copied!
Create Velero Storage Account Resource Group.
1
az group create \
2
--location $VELERO_REGION \
3
--name $VELERO_RG \
4
--subscription $VELERO_SUBSCRIPTION \
5
--tags 'ENV=DEV' 'BU=MARKETING'
Copied!
Create Velero Storage Account.
1
az storage account create \
2
--name $VELERO_SA_NAME \
3
--resource-group $VELERO_RG \
4
--sku Standard_LRS \
5
--encryption-services blob \
6
--https-only true \
7
--kind BlobStorage \
8
--access-tier Hot
Copied!
Create BLOB.
1
BLOB_CONTAINER="AKS-BACKUP"
2
3
az storage container create \
4
-n $BLOB_CONTAINER \
5
--auth-mode login \
6
--public-access off \
7
--account-name $VELERO_SA_NAME
Copied!
Create Velero Service Principal.
1
VELERO_SP_NAME="sp-aks-validate-backup-velero"
2
3
az ad sp create-for-rbac \
4
--skip-assignment \
5
--name $VELERO_SP_NAME
Copied!
Copy and paste in a safe place the Service Principal credentials.
Velero Service Principal must be owner of the AKS Node Resource Group.
1
az role assignment create \
2
--role "Contributor" \
3
--assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
4
--scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"
Copied!
Velero Service Principal must be owner of the Storage Account RG.
1
az role assignment create \
2
--role "Contributor" \
3
--assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
4
--scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"
Copied!
Set env vars.
1
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
2
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
3
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
4
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
5
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"
Copied!
Create a file with Velero credentials.
1
cat << EOF > ./credentials-velero
2
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
3
AZURE_TENANT_ID="$AZURE_TENANT_ID"
4
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
5
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
6
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
7
AZURE_CLOUD_NAME=AzurePublicCloud
8
EOF
Copied!
Install Velero.
1
velero install \
2
--provider azure \
3
--plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
4
--bucket $BLOB_CONTAINER \
5
--secret-file ./credentials-velero \
6
--backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
7
--snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
8
--use-volume-snapshots=true \
9
--velero-pod-cpu-request "500m" \
10
--velero-pod-mem-request "256Mi" \
11
--velero-pod-cpu-limit "2" \
12
--velero-pod-mem-limit "2048Mi" \
13
--use-restic \
14
--restic-pod-cpu-request "500m" \
15
--restic-pod-mem-request "256Mi" \
16
--restic-pod-cpu-limit "2" \
17
--restic-pod-mem-limit "2048Mi"
Copied!

Restic

Requirements

Your pods must have the following annotation.
1
...
2
metadata:
3
...
4
annotations:
5
backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
6
...
Copied!
To backup azure-file, the Storage Classes must have the following mount option:
1
kubectl patch storageclass/azurefile \
2
--type json \
3
--patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'
Copied!
For more details search for "Microsoft Azure" here: https://velero.io/docs/master/restic/

Credentials

1
export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
2
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"
Copied!

List restic repos

1
velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"
Copied!

Get restic password

1
export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
2
velero-restic-credentials \
3
-o jsonpath="{.data.repository-password}" | base64 -d)
4
5
# OR
6
# kubectl get -n velero secrets \
7
# velero-restic-credentials \
8
# -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
9
# export RESTIC_PASSWORD_FILE=~/restic-secret
Copied!

List snapshots

1
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c
Copied!

Mount volume

1
mkdir ~/restore-backup
2
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup
Copied!

Force delete

1
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
2
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
3
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune
Copied!

References

Backup test

Apply the following manifest.
1
---
2
kind: StorageClass
3
apiVersion: storage.k8s.io/v1
4
metadata:
5
name: custom-azurefile
6
provisioner: kubernetes.io/azure-file
7
mountOptions:
8
- nouser_xattr
9
parameters:
10
skuName: Standard_LRS
11
---
12
apiVersion: v1
13
kind: Namespace
14
metadata:
15
name: test-backup
16
---
17
apiVersion: v1
18
kind: PersistentVolumeClaim
19
metadata:
20
name: test-backup-disk
21
namespace: test-backup
22
spec:
23
accessModes:
24
- ReadWriteOnce
25
storageClassName: managed-premium
26
resources:
27
requests:
28
storage: 1Gi
29
---
30
apiVersion: v1
31
kind: PersistentVolumeClaim
32
metadata:
33
name: test-backup-file
34
namespace: test-backup
35
spec:
36
accessModes:
37
- ReadWriteMany
38
volumeMode: Filesystem
39
resources:
40
requests:
41
storage: 1Gi
42
storageClassName: azurefile
43
---
44
apiVersion: v1
45
kind: PersistentVolumeClaim
46
metadata:
47
name: test-backup-custom-file
48
namespace: test-backup
49
spec:
50
accessModes:
51
- ReadWriteMany
52
volumeMode: Filesystem
53
resources:
54
requests:
55
storage: 1Gi
56
storageClassName: custom-azurefile
57
---
58
apiVersion: apps/v1
59
kind: Deployment
60
metadata:
61
name: test-backup
62
namespace: test-backup
63
spec:
64
selector:
65
matchLabels:
66
name: test-backup
67
template:
68
metadata:
69
annotations:
70
backup.velero.io/backup-volumes: disk,file,custom-file
71
labels:
72
name: test-backup
73
spec:
74
containers:
75
- name: test-backup
76
image: busybox
77
command: [ "/bin/sh", "-c", "--" ]
78
args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
79
volumeMounts:
80
- mountPath: "/mnt/disk"
81
name: disk
82
- mountPath: "/mnt/file"
83
name: file
84
- mountPath: "/mnt/custom-file"
85
name: custom-file
86
volumes:
87
- name: disk
88
persistentVolumeClaim:
89
claimName: test-backup-disk
90
- name: file
91
persistentVolumeClaim:
92
claimName: test-backup-file
93
- name: custom-file
94
persistentVolumeClaim:
95
claimName: test-backup-custom-file
Copied!
Create a test backup.
1
velero backup create backup001
Copied!

References

https://velero.io/docs/v1.3.2/restic/ Search for "Microsoft Azure"

Velero (DEPRECATED DOCUMENTATION)

This documentation is outdated. Please visit: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

Setup the storage

Velero stores the backup files on a Storage Account.
Using Azure CLI, login and set your subscription.
1
az login
2
az account set --subscription "YOUR-SUBSCRIPTION"
3
az account list
Copied!
Define variables.
1
AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
2
RG="YOUR-RESOURCE-GROUP"
3
BLOB_CONTAINER=velero
4
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
5
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
6
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
Copied!
Create the Storage Account.
1
az storage account create \
2
--name $AZURE_STORAGE_ACCOUNT_ID \
3
--resource-group $RG \
4
--sku Standard_GRS \
5
--encryption-services blob \
6
--https-only true \
7
--kind BlobStorage \
8
--access-tier Hot
Copied!
Create the Storage Container.
1
az storage container create \
2
-n $BLOB_CONTAINER \
3
--public-access off \
4
--account-name $AZURE_STORAGE_ACCOUNT_ID
Copied!
Create a Service Principal.
1
az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor
Copied!
Get the output and save it in a secure place (it will be displayed only once).
Create the Velero credentials file.
1
cat << EOF > ./credentials-velero
2
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
3
AZURE_TENANT_ID=${AZURE_TENANT_ID}
4
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
5
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
6
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
7
EOF
Copied!

Install Velero

First of all, make sure you have kubectl.
1
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
2
chmod +x ./kubectl
3
sudo mv ./kubectl /usr/local/bin/kubectl
Copied!
Check the latest Velero release: https://github.com/heptio/velero/releases
Install it.
1
wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
2
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
3
chmod +rx velero-v1.0.0-linux-amd64/velero
4
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/
Copied!
Deploy it.
1
velero install \
2
--provider azure \
3
--plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
4
--bucket $BLOB_CONTAINER \
5
--secret-file ./credentials-velero \
6
--backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
7
--snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID
Copied!

Test it

1
velero backup create my-backup-001 --include-namespaces=my-namespace
2
velero backup describe my-backup-001
3
kubectl delete namespace my-namespace
4
velero restore create --from-backup my-backup-001
Copied!

References

Backup schedule

1
velero schedule create entire-cluster \
2
--schedule="0 1 * * *" \
3
--include-resources '*' \
4
--include-namespaces '*' \
5
--include-cluster-resources=true \
6
--ttl 30d \
7
--labels entireCluster=true
Copied!