DevOps Buzz
Search…
⌃K

Backup

AKS backup tricks.

AKS backup with velero

Install velero

Set env vars.
# Velero Azure Storace Account Name (must be unique)
VELERO_SA_NAME="myaksbackup"
# Velero Azure Storace Account Resource Group
VELERO_RG="rg-aks-backup"
# Velero Azure Storace Account Region
VELERO_REGION="eastus"
# Velero Azure Storace Account Subscription
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"
Create Velero Storage Account Resource Group.
az group create \
--location $VELERO_REGION \
--name $VELERO_RG \
--subscription $VELERO_SUBSCRIPTION \
--tags 'ENV=DEV' 'BU=MARKETING'
Create Velero Storage Account.
az storage account create \
--name $VELERO_SA_NAME \
--resource-group $VELERO_RG \
--sku Standard_LRS \
--encryption-services blob \
--https-only true \
--kind BlobStorage \
--access-tier Hot
Create BLOB.
BLOB_CONTAINER="AKS-BACKUP"
az storage container create \
-n $BLOB_CONTAINER \
--auth-mode login \
--public-access off \
--account-name $VELERO_SA_NAME
Create Velero Service Principal.
VELERO_SP_NAME="sp-aks-validate-backup-velero"
az ad sp create-for-rbac \
--skip-assignment \
--name $VELERO_SP_NAME
Copy and paste in a safe place the Service Principal credentials.
Velero Service Principal must be owner of the AKS Node Resource Group.
az role assignment create \
--role "Contributor" \
--assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
--scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"
Velero Service Principal must be owner of the Storage Account RG.
az role assignment create \
--role "Contributor" \
--assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
--scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"
Set env vars.
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"
Create a file with Velero credentials.
cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID="$AZURE_TENANT_ID"
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
Install Velero.
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
--snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
--use-volume-snapshots=true \
--velero-pod-cpu-request "500m" \
--velero-pod-mem-request "256Mi" \
--velero-pod-cpu-limit "2" \
--velero-pod-mem-limit "2048Mi" \
--use-restic \
--restic-pod-cpu-request "500m" \
--restic-pod-mem-request "256Mi" \
--restic-pod-cpu-limit "2" \
--restic-pod-mem-limit "2048Mi"

Restic

Requirements

Your pods must have the following annotation.
...
metadata:
...
annotations:
backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
...
To backup azure-file, the Storage Classes must have the following mount option:
kubectl patch storageclass/azurefile \
--type json \
--patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'
For more details search for "Microsoft Azure" here: https://velero.io/docs/master/restic/

Credentials

export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"

List restic repos

velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"

Get restic password

export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
velero-restic-credentials \
-o jsonpath="{.data.repository-password}" | base64 -d)
# OR
# kubectl get -n velero secrets \
# velero-restic-credentials \
# -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
# export RESTIC_PASSWORD_FILE=~/restic-secret

List snapshots

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c

Mount volume

mkdir ~/restore-backup
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup

Force delete

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune

References

Backup test

Apply the following manifest.
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: custom-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
- nouser_xattr
parameters:
skuName: Standard_LRS
---
apiVersion: v1
kind: Namespace
metadata:
name: test-backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-backup-disk
namespace: test-backup
spec:
accessModes:
- ReadWriteOnce
storageClassName: managed-premium
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-backup-file
namespace: test-backup
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: azurefile
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-backup-custom-file
namespace: test-backup
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: custom-azurefile
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-backup
namespace: test-backup
spec:
selector:
matchLabels:
name: test-backup
template:
metadata:
annotations:
backup.velero.io/backup-volumes: disk,file,custom-file
labels:
name: test-backup
spec:
containers:
- name: test-backup
image: busybox
command: [ "/bin/sh", "-c", "--" ]
args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
volumeMounts:
- mountPath: "/mnt/disk"
name: disk
- mountPath: "/mnt/file"
name: file
- mountPath: "/mnt/custom-file"
name: custom-file
volumes:
- name: disk
persistentVolumeClaim:
claimName: test-backup-disk
- name: file
persistentVolumeClaim:
claimName: test-backup-file
- name: custom-file
persistentVolumeClaim:
claimName: test-backup-custom-file
Create a test backup.
velero backup create backup001

References

https://velero.io/docs/v1.3.2/restic/ Search for "Microsoft Azure"

Velero (DEPRECATED DOCUMENTATION)

This documentation is outdated. Please visit: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

Setup the storage

Velero stores the backup files on a Storage Account.
Using Azure CLI, login and set your subscription.
az login
az account set --subscription "YOUR-SUBSCRIPTION"
az account list
Define variables.
AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
RG="YOUR-RESOURCE-GROUP"
BLOB_CONTAINER=velero
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
Create the Storage Account.
az storage account create \
--name $AZURE_STORAGE_ACCOUNT_ID \
--resource-group $RG \
--sku Standard_GRS \
--encryption-services blob \
--https-only true \
--kind BlobStorage \
--access-tier Hot
Create the Storage Container.
az storage container create \
-n $BLOB_CONTAINER \
--public-access off \
--account-name $AZURE_STORAGE_ACCOUNT_ID
Create a Service Principal.
az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor
Get the output and save it in a secure place (it will be displayed only once).
Create the Velero credentials file.
cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF

Install Velero

First of all, make sure you have kubectl.
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
Check the latest Velero release: https://github.com/heptio/velero/releases
Install it.
wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
chmod +rx velero-v1.0.0-linux-amd64/velero
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/
Deploy it.
velero install \
--provider azure \
--plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
--bucket $BLOB_CONTAINER \
--secret-file ./credentials-velero \
--backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
--snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID

Test it

velero backup create my-backup-001 --include-namespaces=my-namespace
velero backup describe my-backup-001
kubectl delete namespace my-namespace
velero restore create --from-backup my-backup-001

References

Backup schedule

velero schedule create entire-cluster \
--schedule="0 1 * * *" \
--include-resources '*' \
--include-namespaces '*' \
--include-cluster-resources=true \
--ttl 30d \
--labels entireCluster=true