Backup

AKS backup tricks.

AKS backup with velero

Install velero

Set env vars.


# Velero Azure Storace Account Name (must be unique)
VELERO_SA_NAME="myaksbackup"

# Velero Azure Storace Account Resource Group
VELERO_RG="rg-aks-backup"

# Velero Azure Storace Account Region
VELERO_REGION="eastus"

# Velero Azure Storace Account Subscription
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"

Create Velero Storage Account Resource Group.

az group create \
  --location $VELERO_REGION \
  --name $VELERO_RG \
  --subscription $VELERO_SUBSCRIPTION \
  --tags 'ENV=DEV' 'BU=MARKETING'

Create Velero Storage Account.

az storage account create \
  --name $VELERO_SA_NAME \
  --resource-group $VELERO_RG \
  --sku Standard_LRS \
  --encryption-services blob \
  --https-only true \
  --kind BlobStorage \
  --access-tier Hot

Create BLOB.

BLOB_CONTAINER="AKS-BACKUP"

az storage container create \
  -n $BLOB_CONTAINER \
  --auth-mode login \
  --public-access off \
  --account-name $VELERO_SA_NAME

Create Velero Service Principal.

VELERO_SP_NAME="sp-aks-validate-backup-velero"

az ad sp create-for-rbac \
  --skip-assignment \
  --name $VELERO_SP_NAME

Copy and paste in a safe place the Service Principal credentials.

Velero Service Principal must be owner of the AKS Node Resource Group.

az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"

Velero Service Principal must be owner of the Storage Account RG.

az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"

Set env vars.

AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"

Create a file with Velero credentials.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID="$AZURE_TENANT_ID"
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Install Velero.

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
    --snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
    --use-volume-snapshots=true \
    --velero-pod-cpu-request "500m" \
    --velero-pod-mem-request "256Mi" \
    --velero-pod-cpu-limit "2" \
    --velero-pod-mem-limit "2048Mi" \
    --use-restic \
    --restic-pod-cpu-request "500m" \
    --restic-pod-mem-request "256Mi" \
    --restic-pod-cpu-limit "2" \
    --restic-pod-mem-limit "2048Mi"

Restic

Requirements

Your pods must have the following annotation.

...
metadata:
...
  annotations:
    backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
...

To backup azure-file, the Storage Classes must have the following mount option:

kubectl patch storageclass/azurefile \
  --type json \
  --patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'

For more details search for "Microsoft Azure" here: https://velero.io/docs/master/restic/

Credentials

export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"

List restic repos

velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"

Get restic password

export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
  velero-restic-credentials \
  -o jsonpath="{.data.repository-password}" | base64 -d)
  
# OR
# kubectl get -n velero secrets \
#  velero-restic-credentials \
#  -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
# export RESTIC_PASSWORD_FILE=~/restic-secret

List snapshots

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c

Mount volume

mkdir ~/restore-backup
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup

Force delete

restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune

References

https://docs.syseleven.de/metakube/en/tutorials/create-backup-and-restore

https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage

https://restic.readthedocs.io/en/latest/060_forget.html

Backup test

Apply the following manifest.

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: custom-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
  - nouser_xattr
parameters:
  skuName: Standard_LRS
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-disk
  namespace: test-backup
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: azurefile
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-custom-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: custom-azurefile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-backup
  namespace: test-backup
spec:
  selector:
    matchLabels:
      name: test-backup
  template:
    metadata:
      annotations:
        backup.velero.io/backup-volumes: disk,file,custom-file
      labels:
        name: test-backup
    spec:
      containers:
      - name: test-backup
        image: busybox
        command: [ "/bin/sh", "-c", "--" ]
        args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
        volumeMounts:
        - mountPath: "/mnt/disk"
          name: disk
        - mountPath: "/mnt/file"
          name: file
        - mountPath: "/mnt/custom-file"
          name: custom-file
      volumes:
      - name: disk
        persistentVolumeClaim:
          claimName: test-backup-disk
      - name: file
        persistentVolumeClaim:
          claimName: test-backup-file
      - name: custom-file
        persistentVolumeClaim:
          claimName: test-backup-custom-file

Create a test backup.

velero backup create backup001

References

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

https://github.com/vmware-tanzu/velero/blob/48792ece1f6b10d732b57477059a6b32dd10e25d/site/docs/v1.0.0/restic.md

https://velero.io/docs/v1.3.2/restic/ Search for "Microsoft Azure"

https://stackoverflow.com/questions/61915527/azure-aks-backup-using-velero

https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#recommended-stable-version-for-azure-disk

https://github.com/vmware-tanzu/velero/issues/1404

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/issues/13

Velero (DEPRECATED DOCUMENTATION)

This documentation is outdated. Please visit: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

Setup the storage

Velero stores the backup files on a Storage Account.

Using Azure CLI, login and set your subscription.

az login
az account set --subscription "YOUR-SUBSCRIPTION"
az account list

Define variables.

AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
RG="YOUR-RESOURCE-GROUP"
BLOB_CONTAINER=velero
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`

Create the Storage Account.

az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RG \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot

Create the Storage Container.

az storage container create \
  -n $BLOB_CONTAINER \
  --public-access off \
  --account-name $AZURE_STORAGE_ACCOUNT_ID

Create a Service Principal.

az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor

Get the output and save it in a secure place (it will be displayed only once).

Create the Velero credentials file.

cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF

Install Velero

First of all, make sure you have kubectl.

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Check the latest Velero release: https://github.com/heptio/velero/releases

Install it.

wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
chmod +rx velero-v1.0.0-linux-amd64/velero
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/

Deploy it.

velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
    --snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID

Test it

velero backup create my-backup-001 --include-namespaces=my-namespace
velero backup describe my-backup-001
kubectl delete namespace my-namespace
velero restore create --from-backup my-backup-001

References

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure

https://velero.io/docs/v1.0.0/azure-config/

https://blog.getupcloud.com/velero-backup-do-seu-cluster-kubernetes-em-5-minutos-3129a0518541

https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velero-with-restic-and-rook-ceph-as-2e8df15b1487

https://sysadminas.eu/2019/12/part-3-azure-kubernetes-services-aks-backup-restore-your-aks-data-with-velero/

Backup schedule

velero schedule create entire-cluster \
    --schedule="0 1 * * *" \
    --include-resources '*' \
    --include-namespaces '*' \
    --include-cluster-resources=true \
    --ttl 30d \
    --labels entireCluster=true

PreviousAutoscale NextDashboard

Last updated 5 years ago

hashtagAKS backup with velero

hashtagInstall velero

hashtagRestic

hashtagRequirements

hashtagCredentials

hashtagList restic repos

hashtagGet restic password

hashtagList snapshots

hashtagMount volume

hashtagForce delete

hashtagReferences

hashtagBackup test

hashtagReferences

hashtagVelero (DEPRECATED DOCUMENTATION)

hashtagSetup the storage

hashtagInstall Velero

hashtagTest it

hashtagReferences

hashtagBackup schedule

AKS backup with velero

Install velero

Restic

Requirements

Credentials

List restic repos

Get restic password

List snapshots

Mount volume

Force delete

References

Backup test

References

Velero (DEPRECATED DOCUMENTATION)

Setup the storage

Install Velero

Test it

References

Backup schedule