# Backup

## AKS backup with velero

### Install velero

Set env vars.

```bash

# Velero Azure Storace Account Name (must be unique)
VELERO_SA_NAME="myaksbackup"

# Velero Azure Storace Account Resource Group
VELERO_RG="rg-aks-backup"

# Velero Azure Storace Account Region
VELERO_REGION="eastus"

# Velero Azure Storace Account Subscription
VELERO_SUBSCRIPTION="MY-SUBSCRIPTION"
```

Create Velero Storage Account Resource Group.

```bash
az group create \
  --location $VELERO_REGION \
  --name $VELERO_RG \
  --subscription $VELERO_SUBSCRIPTION \
  --tags 'ENV=DEV' 'BU=MARKETING'
```

Create Velero Storage Account.

```bash
az storage account create \
  --name $VELERO_SA_NAME \
  --resource-group $VELERO_RG \
  --sku Standard_LRS \
  --encryption-services blob \
  --https-only true \
  --kind BlobStorage \
  --access-tier Hot
```

Create BLOB.

```bash
BLOB_CONTAINER="AKS-BACKUP"

az storage container create \
  -n $BLOB_CONTAINER \
  --auth-mode login \
  --public-access off \
  --account-name $VELERO_SA_NAME
```

Create Velero Service Principal.

```bash
VELERO_SP_NAME="sp-aks-validate-backup-velero"

az ad sp create-for-rbac \
  --skip-assignment \
  --name $VELERO_SP_NAME
```

Copy and paste in a safe place the Service Principal credentials.

Velero Service Principal must be owner of the AKS Node Resource Group.

```bash
az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-AKS-NODE-RG-ID-HERE"
```

Velero Service Principal must be owner of the Storage Account RG.

```bash
az role assignment create \
  --role "Contributor" \
  --assignee "PASTE-YOUR-VELERO-SP-ID-HERE" \
  --scope "PASTE-YOUR-VELERO-STORAGE-ACCOUNT-RG-ID-HERE"
```

Set env vars.

```bash
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
AZURE_CLIENT_ID="PASTE-VELERO-SP-ID-HERE"
AZURE_CLIENT_SECRET="PASTE-VELERO-SP-PASSWORD-HERE"
AZURE_RESOURCE_GROUP="PASTE-YOUR-AKS-NODE-RG-NAME-HERE"
```

Create a file with Velero credentials.

```bash
cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID"
AZURE_TENANT_ID="$AZURE_TENANT_ID"
AZURE_CLIENT_ID="$AZURE_CLIENT_ID"
AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET"
AZURE_RESOURCE_GROUP="$AZURE_RESOURCE_GROUP"
AZURE_CLOUD_NAME=AzurePublicCloud
EOF
```

Install Velero.

```bash
velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.1.0 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup="$VELERO_RG",storageAccount=aksvalidatebackup \
    --snapshot-location-config apiTimeout="30m",resourceGroup="$VELERO_RG" \
    --use-volume-snapshots=true \
    --velero-pod-cpu-request "500m" \
    --velero-pod-mem-request "256Mi" \
    --velero-pod-cpu-limit "2" \
    --velero-pod-mem-limit "2048Mi" \
    --use-restic \
    --restic-pod-cpu-request "500m" \
    --restic-pod-mem-request "256Mi" \
    --restic-pod-cpu-limit "2" \
    --restic-pod-mem-limit "2048Mi"
```

### Restic

#### Requirements

Your **pods** must have the following annotation.

```bash
...
metadata:
...
  annotations:
    backup.velero.io/backup-volumes: PODS-VOLUME-NAMES-HERE-SEE-EXAMPLE-BELLOW
...
```

To backup `azure-file`, the Storage Classes **must** have the following mount option:

```bash
kubectl patch storageclass/azurefile \
  --type json \
  --patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]'
```

For more details search for "Microsoft Azure" here: <https://velero.io/docs/master/restic/>

#### Credentials

```bash
export AZURE_ACCOUNT_NAME="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-NAME-HERE"
export AZURE_ACCOUNT_KEY="PASTE-YOUR-VELERO-STORAGE-ACCOUNT-KEY-HERE"
```

#### List restic repos

```bash
velero restic repo get -o json | jq -r ".items[].spec.resticIdentifier"
```

#### Get restic password

```bash
export RESTIC_PASSWORD=$(kubectl get -n velero secrets \
  velero-restic-credentials \
  -o jsonpath="{.data.repository-password}" | base64 -d)
  
# OR
# kubectl get -n velero secrets \
#  velero-restic-credentials \
#  -o jsonpath="{.data.repository-password}" | base64 -d > ~/restic-secret
# export RESTIC_PASSWORD_FILE=~/restic-secret
```

#### List snapshots

```bash
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots -c
```

#### Mount volume

```bash
mkdir ~/restore-backup
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" mount ~/restore-backup
```

#### Force delete

```bash
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" snapshots
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" forget PASTE-YOUR-SNAPSHOT-ID-HERE
restic -r "PASTE-YOUR-RESTIC-REPO-HERE" prune
```

#### References

<https://docs.syseleven.de/metakube/en/tutorials/create-backup-and-restore>

<https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>

<https://restic.readthedocs.io/en/latest/060_forget.html>

### Backup test

Apply the following manifest.

```bash
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: custom-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
  - nouser_xattr
parameters:
  skuName: Standard_LRS
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-backup
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-disk
  namespace: test-backup
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: azurefile
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-backup-custom-file
  namespace: test-backup
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: custom-azurefile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-backup
  namespace: test-backup
spec:
  selector:
    matchLabels:
      name: test-backup
  template:
    metadata:
      annotations:
        backup.velero.io/backup-volumes: disk,file,custom-file
      labels:
        name: test-backup
    spec:
      containers:
      - name: test-backup
        image: busybox
        command: [ "/bin/sh", "-c", "--" ]
        args: [ 'while true; do touch "/mnt/disk/file_$(date +%F-%H-%M-%S)"; touch "/mnt/file/file_$(date +%F-%H-%M-%S)"; touch "/mnt/custom-file/file_$(date +%F-%H-%M-%S)"; sleep 60; done' ]
        volumeMounts:
        - mountPath: "/mnt/disk"
          name: disk
        - mountPath: "/mnt/file"
          name: file
        - mountPath: "/mnt/custom-file"
          name: custom-file
      volumes:
      - name: disk
        persistentVolumeClaim:
          claimName: test-backup-disk
      - name: file
        persistentVolumeClaim:
          claimName: test-backup-file
      - name: custom-file
        persistentVolumeClaim:
          claimName: test-backup-custom-file
```

Create a test backup.

```bash
velero backup create backup001
```

### References

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>

<https://github.com/vmware-tanzu/velero/blob/48792ece1f6b10d732b57477059a6b32dd10e25d/site/docs/v1.0.0/restic.md>

<https://velero.io/docs/v1.3.2/restic/> Search for "Microsoft Azure"

<https://stackoverflow.com/questions/61915527/azure-aks-backup-using-velero>

<https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#recommended-stable-version-for-azure-disk>

<https://github.com/vmware-tanzu/velero/issues/1404>

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/issues/13>

## Velero (DEPRECATED DOCUMENTATION)

{% hint style="warning" %}
This documentation is outdated. Please visit: <https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>
{% endhint %}

### Setup the storage

Velero stores the backup files on a Storage Account.

Using Azure CLI, login and set your subscription.

```bash
az login
az account set --subscription "YOUR-SUBSCRIPTION"
az account list
```

Define variables.

```bash
AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"
RG="YOUR-RESOURCE-GROUP"
BLOB_CONTAINER=velero
AZURE_RESOURCE_GROUP="MC_YOUR-AUTO-GENERATED-AKS-RG_brazilsouth"
AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`
AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`
```

Create the Storage Account.

```bash
az storage account create \
    --name $AZURE_STORAGE_ACCOUNT_ID \
    --resource-group $RG \
    --sku Standard_GRS \
    --encryption-services blob \
    --https-only true \
    --kind BlobStorage \
    --access-tier Hot
```

Create the Storage Container.

```bash
az storage container create \
  -n $BLOB_CONTAINER \
  --public-access off \
  --account-name $AZURE_STORAGE_ACCOUNT_ID
```

Create a Service Principal.

```bash
az ad sp create-for-rbac -n $AZURE_STORAGE_ACCOUNT_ID --role contributor
```

Get the output and save it in a secure place (it will be displayed only once).

Create the Velero credentials file.

```bash
cat << EOF  > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=PUT-YOU-PRINCIPAL-SECRET-HERE
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
EOF
```

### Install Velero

First of all, make sure you have kubectl.

```bash
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
```

Check the latest Velero release: <https://github.com/heptio/velero/releases>

Install it.

```
wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xzvf velero-v1.0.0-linux-amd64.tar.gz
chmod +rx velero-v1.0.0-linux-amd64/velero
mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/
```

Deploy it.

```bash
velero install \
    --provider azure \
    --plugins velero/velero-plugin-for-microsoft-azure:v1.0.1 \
    --bucket $BLOB_CONTAINER \
    --secret-file ./credentials-velero \
    --backup-location-config resourceGroup=$YOUR_STORAGE_RG,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_SUBSCRIPTION_ID \
    --snapshot-location-config resourceGroup=$YOUR_STORAGE_RG,subscriptionId=$AZURE_SUBSCRIPTION_ID
```

### Test it

```
velero backup create my-backup-001 --include-namespaces=my-namespace
velero backup describe my-backup-001
kubectl delete namespace my-namespace
velero restore create --from-backup my-backup-001
```

### References

<https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure>

<https://velero.io/docs/v1.0.0/azure-config/>

<https://blog.getupcloud.com/velero-backup-do-seu-cluster-kubernetes-em-5-minutos-3129a0518541>

<https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velero-with-restic-and-rook-ceph-as-2e8df15b1487>

<https://sysadminas.eu/2019/12/part-3-azure-kubernetes-services-aks-backup-restore-your-aks-data-with-velero/>

### Backup schedule

```bash
velero schedule create entire-cluster \
    --schedule="0 1 * * *" \
    --include-resources '*' \
    --include-namespaces '*' \
    --include-cluster-resources=true \
    --ttl 30d \
    --labels entireCluster=true
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://www.devops.buzz/public/aks/backup.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.