> For the complete documentation index, see [llms.txt](https://www.devops.buzz/public/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.devops.buzz/public/aks/backup.md).

# Backup

## AKS backup with velero

### Install velero

Set env vars.

```bash

# Velero Azure Storace Account Name (must be unique)
VELERO_SA_NAME="myaksbackup"

# Velero Azure Storace Account Resource Group
VELERO_RG="rg-aks-backup"

# Velero Azure Storace Account Region
VELERO_REGION="eastus"

# Velero Azure Storace Account Subscription
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"
```

Create Velero Storage Account Resource Group.

```bash
az group create \
  --location $VELERO_REGION \
  --name $VELERO_RG \
  --subscription $VELERO_SUBSCRIPTION \
  --tags 'ENV=DEV' 'BU=MARKETING'
```

Create Velero Storage Account.

```bash
az storage account create \
  --name $VELERO_SA_NAME \
  --resource-group $VELERO_RG \
  --sku Standard_LRS \
  --encryption-services blob \
  --https-only true \
  --kind BlobStorage \
  --access-tier Hot
```

Create BLOB.

```bash
BLOB_CONTAINER="AKS-BACKUP"

az storage container create \
  -n $BLOB_CONTAINER \
  --auth-mode login \
  --public-access off \
  --account-name $VELERO_SA_NAME
```

Create Velero Service Principal.

```bash
VELERO_SP_NAME="sp-aks-validate-backup-velero"

az ad sp create-for-rbac \
  --skip-assignment \
  --name $VELERO_SP_NAME
```

Copy and paste in a safe place the Service Principal credentials.

Velero Service Principal must be owner of the AKS Node Resource Group.

```bash
az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"
```

Velero Service Principal must be owner of the Storage Account RG.

```bash
az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"
```

Set env vars.

```bash
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"
```

Create a file with Velero credentials.

```bash
cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID="$AZURE_TENANT_ID"
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
```

Install Velero.

```bash
velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
    --snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
    --use-volume-snapshots=true \
    --velero-pod-cpu-request "500m" \
    --velero-pod-mem-request "256Mi" \
    --velero-pod-cpu-limit "2" \
    --velero-pod-mem-limit "2048Mi" \
    --use-restic \
    --restic-pod-cpu-request "500m" \
    --restic-pod-mem-request "256Mi" \
    --restic-pod-cpu-limit "2" \
    --restic-pod-mem-limit "2048Mi"
```

### Restic

#### Requirements

Your **pods** must have the following annotation.

```bash
...
metadata:
...
  annotations:
    backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
...
```

To backup `azure-file`, the Storage Classes **must** have the following mount option:

```bash
kubectl patch storageclass/azurefile \
  --type json \
  --patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'
```

For more details search for "Microsoft Azure" here: <https://velero.io/docs/master/restic/>

#### Credentials

```bash
export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"
```

#### List restic repos

```bash
velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"
```

#### Get restic password

```bash
export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
  velero-restic-credentials \
  -o jsonpath="{.data.repository-password}" | base64 -d)
  
# OR
# kubectl get -n velero secrets \
#  velero-restic-credentials \
#  -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
# export RESTIC_PASSWORD_FILE=~/restic-secret
```

#### List snapshots

```bash
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c
```

#### Mount volume

```bash
mkdir ~/restore-backup
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup
```

#### Force delete

```bash
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune
```

#### References

<https://docs.syseleven.de/metakube/en/tutorials/create-backup-and-restore>

<https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>

<https://restic.readthedocs.io/en/latest/060_forget.html>

### Backup test

Apply the following manifest.

```bash
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: custom-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
  - nouser_xattr
parameters:
  skuName: Standard_LRS
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-disk
  namespace: test-backup
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: azurefile
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-custom-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: custom-azurefile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-backup
  namespace: test-backup
spec:
  selector:
    matchLabels:
      name: test-backup
  template:
    metadata:
      annotations:
        backup.velero.io/backup-volumes: disk,file,custom-file
      labels:
        name: test-backup
    spec:
      containers:
      - name: test-backup
        image: busybox
        command: [ "/bin/sh", "-c", "--" ]
        args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
        volumeMounts:
        - mountPath: "/mnt/disk"
          name: disk
        - mountPath: "/mnt/file"
          name: file
        - mountPath: "/mnt/custom-file"
          name: custom-file
      volumes:
      - name: disk
        persistentVolumeClaim:
          claimName: test-backup-disk
      - name: file
        persistentVolumeClaim:
          claimName: test-backup-file
      - name: custom-file
        persistentVolumeClaim:
          claimName: test-backup-custom-file
```

Create a test backup.

```bash
velero backup create backup001
```

### References

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>

<https://github.com/vmware-tanzu/velero/blob/48792ece1f6b10d732b57477059a6b32dd10e25d/site/docs/v1.0.0/restic.md>

<https://velero.io/docs/v1.3.2/restic/> Search for "Microsoft Azure"

<https://stackoverflow.com/questions/61915527/azure-aks-backup-using-velero>

<https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#recommended-stable-version-for-azure-disk>

<https://github.com/vmware-tanzu/velero/issues/1404>

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/issues/13>

## Velero (DEPRECATED DOCUMENTATION)

{% hint style="warning" %}
This documentation is outdated. Please visit: <https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>
{% endhint %}

### Setup the storage

Velero stores the backup files on a Storage Account.

Using Azure CLI, login and set your subscription.

```bash
az login
az account set --subscription "YOUR-SUBSCRIPTION"
az account list
```

Define variables.

```bash
AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
RG="YOUR-RESOURCE-GROUP"
BLOB_CONTAINER=velero
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
```

Create the Storage Account.

```bash
az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RG \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot
```

Create the Storage Container.

```bash
az storage container create \
  -n $BLOB_CONTAINER \
  --public-access off \
  --account-name $AZURE_STORAGE_ACCOUNT_ID
```

Create a Service Principal.

```bash
az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor
```

Get the output and save it in a secure place (it will be displayed only once).

Create the Velero credentials file.

```bash
cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF
```

### Install Velero

First of all, make sure you have kubectl.

```bash
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
```

Check the latest Velero release: <https://github.com/heptio/velero/releases>

Install it.

```
wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
chmod +rx velero-v1.0.0-linux-amd64/velero
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/
```

Deploy it.

```bash
velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
    --snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID
```

### Test it

```
velero backup create my-backup-001 --include-namespaces=my-namespace
velero backup describe my-backup-001
kubectl delete namespace my-namespace
velero restore create --from-backup my-backup-001
```

### References

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>

<https://velero.io/docs/v1.0.0/azure-config/>

<https://blog.getupcloud.com/velero-backup-do-seu-cluster-kubernetes-em-5-minutos-3129a0518541>

<https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velero-with-restic-and-rook-ceph-as-2e8df15b1487>

<https://sysadminas.eu/2019/12/part-3-azure-kubernetes-services-aks-backup-restore-your-aks-data-with-velero/>

### Backup schedule

```bash
velero schedule create entire-cluster \
    --schedule="0 1 * * *" \
    --include-resources '*' \
    --include-namespaces '*' \
    --include-cluster-resources=true \
    --ttl 30d \
    --labels entireCluster=true
```