Autoscale

AKS autoscale setup and test

Setup the cluster

Enable AZ CLI preview.

az extension add --name aks-preview

Set your subscription.

az account set --subscription "MY-SUBSCRIPTION"

Create a Resource Group.

az group create \
  --location brazilsouth \
  --name my-test-autoscale \
  --subscription "MY-SUBSCRIPTION"

Create the cluster.

az aks create \
  --name my-test-autoscale \
  --resource-group my-test-autoscale \
  --dns-name-prefix my-test-autoscale \
  --dns-service-ip 10.0.0.10 \
  --docker-bridge-address 172.17.0.1/16 \
  --kubernetes-version 1.14.8 \
  --location brazilsouth \
  --network-plugin kubenet \
  --node-count 3 \
  --node-osdisk-size 128 \
  --node-vm-size Standard_D4s_v3 \
  --pod-cidr 10.244.0.0/16 \
  --service-cidr 10.0.0.0/16 \
  --subscription "MY-SUBSCRIPTION" \
  --enable-cluster-autoscaler \
  --max-count 5 \
  --min-count 3

Stress test

Create a few deployments.

for i in {1..3};
do
kubectl run \
  resource-consumer-$i \
  --image=gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4 \
  --expose \
  --service-overrides='{ "spec": { "type": "LoadBalancer" } }' --port 8080
done

Scale the deployments.

for i in {1..3};
do

kubectl scale deployment --replicas=200 resource-consumer-$i

done

Get the public IPs.

for i in {1..3};
do

kubectl get services resource-consumer-$i

done

Trigger resource usage.

for i in {1..100};
do

curl --data "millicores=4000&durationSec=3600" http://IP.OF.SERVICE-01:8080/ConsumeCPU;
curl --data "millicores=4000&durationSec=3600" http://IP.OF.SERVICE-02:8080/ConsumeCPU;
curl --data "millicores=4000&durationSec=3600" http://IP.OF.SERVICE-03:8080/ConsumeCPU;

done

References

https://github.com/kubernetes/kubernetes/tree/master/test/images/resource-consumer

Debug and troubleshoot

Log Analytics

Create a Log Analytics Workspace .

Add Diagnostic Settings on your AKS Resource Group (not the MC_...)

Go to logs and query.

AzureDiagnostics
| where Category == "cluster-autoscaler"

References

https://docs.microsoft.com/en-us/azure/aks/view-master-logs#enable-diagnostics-logs

https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler#retrieve-cluster-autoscaler-logs-and-status

Configmap

Check out autoscaler configmap.

kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml

Troubleshoot

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#ca-doesnt-work-but-it-used-to-work-yesterday-why