Backup

AKS backup tricks.

AKS backup with velero

Install velero

Set env vars.

# Velero Azure Storace Account Name (must be unique)
VELERO_SA_NAME="myaksbackup"

# Velero Azure Storace Account Resource Group
VELERO_RG="rg-aks-backup"

# Velero Azure Storace Account Region
VELERO_REGION="eastus"

# Velero Azure Storace Account Subscription
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"

Create Velero Storage Account Resource Group.

az group create \
  --location $VELERO_REGION \
  --name $VELERO_RG \
  --subscription $VELERO_SUBSCRIPTION \
  --tags 'ENV=DEV' 'BU=MARKETING'

Create Velero Storage Account.

az storage account create \
  --name $VELERO_SA_NAME \
  --resource-group $VELERO_RG \
  --sku Standard_LRS \
  --encryption-services blob \
  --https-only true \
  --kind BlobStorage \
  --access-tier Hot

Create BLOB.

BLOB_CONTAINER="AKS-BACKUP"

az storage container create \
  -n $BLOB_CONTAINER \
  --auth-mode login \
  --public-access off \
  --account-name $VELERO_SA_NAME

Create Velero Service Principal.

VELERO_SP_NAME="sp-aks-validate-backup-velero"

az ad sp create-for-rbac \
  --skip-assignment \
  --name $VELERO_SP_NAME

Copy and paste in a safe place the Service Principal credentials.

Velero Service Principal must be owner of the AKS Node Resource Group.

az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"

Velero Service Principal must be owner of the Storage Account RG.

az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"

Set env vars.

AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"

Create a file with Velero credentials.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID="$AZURE_TENANT_ID"
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Install Velero.

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
    --snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
    --use-volume-snapshots=true \
    --velero-pod-cpu-request "500m" \
    --velero-pod-mem-request "256Mi" \
    --velero-pod-cpu-limit "2" \
    --velero-pod-mem-limit "2048Mi" \
    --use-restic \
    --restic-pod-cpu-request "500m" \
    --restic-pod-mem-request "256Mi" \
    --restic-pod-cpu-limit "2" \
    --restic-pod-mem-limit "2048Mi"

Restic

Requirements

Your pods must have the following annotation.

...
metadata:
...
  annotations:
    backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
...

To backup azure-file, the Storage Classes must have the following mount option:

kubectl patch storageclass/azurefile \
  --type json \
  --patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'

For more details search for "Microsoft Azure" here: https://velero.io/docs/master/restic/

Credentials

export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"

List restic repos

velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"

Get restic password

export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
  velero-restic-credentials \
  -o jsonpath="{.data.repository-password}" | base64 -d)
  
# OR
# kubectl get -n velero secrets \
#  velero-restic-credentials \
#  -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
# export RESTIC_PASSWORD_FILE=~/restic-secret

List snapshots

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c

Mount volume

mkdir ~/restore-backup
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup

Force delete

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune

References

https://docs.syseleven.de/metakube/en/tutorials/create-backup-and-restore

https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage

https://restic.readthedocs.io/en/latest/060_forget.html

Backup test

Apply the following manifest.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: custom-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
  - nouser_xattr
parameters:
  skuName: Standard_LRS
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-disk
  namespace: test-backup
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: azurefile
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-custom-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: custom-azurefile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-backup
  namespace: test-backup
spec:
  selector:
    matchLabels:
      name: test-backup
  template:
    metadata:
      annotations:
        backup.velero.io/backup-volumes: disk,file,custom-file
      labels:
        name: test-backup
    spec:
      containers:
      - name: test-backup
        image: busybox
        command: [ "/bin/sh", "-c", "--" ]
        args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
        volumeMounts:
        - mountPath: "/mnt/disk"
          name: disk
        - mountPath: "/mnt/file"
          name: file
        - mountPath: "/mnt/custom-file"
          name: custom-file
      volumes:
      - name: disk
        persistentVolumeClaim:
          claimName: test-backup-disk
      - name: file
        persistentVolumeClaim:
          claimName: test-backup-file
      - name: custom-file
        persistentVolumeClaim:
          claimName: test-backup-custom-file

Create a test backup.

velero backup create backup001

References

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

https://github.com/vmware-tanzu/velero/blob/48792ece1f6b10d732b57477059a6b32dd10e25d/site/docs/v1.0.0/restic.md

https://velero.io/docs/v1.3.2/restic/ Search for "Microsoft Azure"

https://stackoverflow.com/questions/61915527/azure-aks-backup-using-velero

https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#recommended-stable-version-for-azure-disk

https://github.com/vmware-tanzu/velero/issues/1404

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/issues/13

Velero (DEPRECATED DOCUMENTATION)

This documentation is outdated. Please visit: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

Setup the storage

Velero stores the backup files on a Storage Account.

Using Azure CLI, login and set your subscription.

az login
az account set --subscription "YOUR-SUBSCRIPTION"
az account list

Define variables.

AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
RG="YOUR-RESOURCE-GROUP"
BLOB_CONTAINER=velero
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`

Create the Storage Account.

az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RG \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot

Create the Storage Container.

az storage container create \
  -n $BLOB_CONTAINER \
  --public-access off \
  --account-name $AZURE_STORAGE_ACCOUNT_ID

Create a Service Principal.

az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor

Get the output and save it in a secure place (it will be displayed only once).

Create the Velero credentials file.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF

Install Velero

First of all, make sure you have kubectl.

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Check the latest Velero release: https://github.com/heptio/velero/releases

Install it.

wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
chmod +rx velero-v1.0.0-linux-amd64/velero
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/

Deploy it.

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
    --snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID

Test it

velero backup create my-backup-001 --include-namespaces=my-namespace
velero backup describe my-backup-001
kubectl delete namespace my-namespace
velero restore create --from-backup my-backup-001

References

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

https://velero.io/docs/v1.0.0/azure-config/

https://blog.getupcloud.com/velero-backup-do-seu-cluster-kubernetes-em-5-minutos-3129a0518541

https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velero-with-restic-and-rook-ceph-as-2e8df15b1487

https://sysadminas.eu/2019/12/part-3-azure-kubernetes-services-aks-backup-restore-your-aks-data-with-velero/

Backup schedule

velero schedule create entire-cluster \
    --schedule="0 1 * * *" \
    --include-resources '*' \
    --include-namespaces '*' \
    --include-cluster-resources=true \
    --ttl 30d \
    --labels entireCluster=true