Cluster auto-scaling for Azure Kubernetes Service (AKS) is available for quite some time now. I have been using it in several projects so far.This post explains all details about the AKS cluster auto-scaler, shows how to enable it for both - new and existing AKS clusters - and gives you an example of how to use custom auto-scaler profile settings.
- What is the AKS cluster auto-scaler
- Why should you use cluster auto-scaler
- Enable Azure Kubernetes cluster auto-scaler
- Modify auto-scaler boundaries
- Disable cluster auto-scaler
- Verify AKS cluster auto-scaling
- Custom cluster scaling behavior
- Cluster auto-scaler in combination with multiple AKS node-pools
What is the AKS cluster auto-scaler
With cluster auto-scaling, the actual load of your worker-nodes will be monitored actively. By adding and removing worker-nodes from the cluster, it ensures that enough resources are available to keep your application healthy and responsive. In contrast, it removes worker-nodes from the AKS cluster, to optimize resource utilization and be as cost-effective as possible.
The AKS cluster auto-scaler component checks if there are Pods prevented from being deployed to the cluster due to resource limitations. If this happens, the cluster auto-scaler scales out (it adds worker-nodes to the AKS cluster).
If the overall resource utilization drops and worker-nodes are facing a lower load over some time, the cluster will be scaled-in (worker-nodes will be removed from AKS).
Perhaps you are familiar with Kubernetes Horizontal Pod Autoscaling (HPA), which allows us to scale-in and -out_Pods_ based on their actual resource utilization. Conceptually, it is pretty much the same thing, but on another architectural layer. You can also combine HPA and cluster auto-scaler, to create a genuinely elastic Kubernetes cluster in Microsoft Azure.
Why should you use cluster auto-scaler
Creating a rock-solid forecast for cloud-native applications or SaaS projects could be challenging. The unexpected load could happen at any time. Just imagine your solution being featured in one of the most significant online communities, or think about the opposite. Your current product - already running in AKS - helps teams to stay connected while working from home. During weekends, the overall usage of your application drops by 90%.
There are hundreds of scenarios and aspects that affect the load of your infrastructure dramatically within hours, minutes, or even seconds. (Weather, time of the day, public vacation, …).
The AKS cluster auto-scaler allows you to keep your application responsive and healthy, by minimizing the risk of resource shortages. Additionally, the cluster auto-scaler will minimize your operational expenditures (OpEx) by scaling-in the AKS when the cluster has to deal with less load.
Automated scaling behavior of the entire cluster makes the AKS cluster auto-scaler a mandatory and essential feature, which every AKS customer should know about.
Enable Azure Kubernetes cluster auto-scaler
To enable AKS cluster auto-scaling while creating a new AKS cluster with Azure CLI, just append the
--enable-cluster-autoscaler in combination with
--max-count which specify the outer boundaries for the auto-scaler.
AKS_NAME=aks-scaling-demo RESOURCE_GROUP_NAME=aks-scaling-demo #create a Resource Group az group create -n $RESOURCE_GROUP_NAME -l westeurope # Create an AKS cluster with auto-scaler enabled az aks create -g $RESOURCE_GROUP_NAME \ -n $AKS_NAME \ --node-count 1 \ --enable-cluster-autoscaler \ --min-count 1 \ --max-count 4
Azure will provision a new AKS instance and enable cluster auto-scaling. It can take a few minutes to bring everything up and enable cluster auto-scaling. So this may be the right time for a nice cup of coffee ☕️.
If you already have an existing Azure Kubernetes cluster, you can enable the cluster auto-scaler using
az aks nodepool update. I am always defensive when enabling cluster auto-scaler on existing AKS clusters. That said, I check the actual worker-node count before enabling it and use the number of currently assigned worker-nodes as
min-count, to set the lower boundary of the auto-scaler.
AKS_NAME=aks-scaling-demo RESOURCE_GROUP_NAME=aks-scaling-demo # get current node count kubectl get nodes --no-headers | wc -l # 1 # get desired node-pool name az aks nodepool list -g $RESOURCE_GROUP_NAME --cluster-name $AKS_NAME \ -o table --query .name # store node-pool name in temporary variable (AKS_NODE_POOL_NAME) # enable cluster auto-scaler on existing AKS cluster az aks nodepool update -n $AKS_NODE_POOL_NAME --cluster-name $AKS_NAME \ -g $RESOURCE_GROUP_NAME \ --enable-cluster-autoscaler \ --min-count 1 \ --max-count 3
Modify auto-scaler boundaries
Perhaps you have also been a bit defensive while specifying the initial auto-scaler boundaries. Seeing your application growing and becoming more popular, you may want to modify the outer boundaries later on. Again, we use
az aks update to achieve this:
# modify AKS cluster auto-scaler boundaries az aks update -g $RESOURCE_GROUP_NAME \ -n $AKS_NAME \ --update-cluster-autoscaler \ --min-count 1 \ --max-count 10
Disable cluster auto-scaler
Obviously, you can also disable the cluster auto-scaler using
az aks update :
# disable AKS cluster auto-scaler az aks update -g $RESOURCE_GROUP_NAME \ -n $AKS_NAME \ --disable-cluster-autoscaler
Once disabled, we use the good, old
az aks scale command, to manually control the number of worker-nodes in the cluster:
# scale AKS cluster manually az aks scale -g $RESOURCE_GROUP_NAME \ -n $AKS_NAME --node-count 4
Verify AKS cluster auto-scaling
To verify auto-scaling, we will use a sample deployment with explicitly defined resource-requests and -limits. Once deployed to the cluster, we will continuously scale the deployment to increase the overall resource utilization. Once the utilization exceeds the threshold and AKS prevents spinning up new Pods, Azure starts adding new nodes to the AKS cluster.
On the other hand, the AKS cluster will scale-in, as soon as we reduce the number of deployment replicas again. (With some expected delay - more about that later). I have created a new cluster for this sample using the following configuration:
# Create the resource group az group create -n aks-scaling-demo -l westeurope # create the AKS cluster az aks create -g aks-scaling-demo -n aks-scaling-demo \ --node-count 1 \ --node-vm-size Standard_B2s \ --enable-cluster-autoscaler \ --min-count 1 \ --max-count 4 # download credentials and switch kubectl context az aks get-credentials -n aks-scaling-demo -g aks-scaling-demo # verify current kubectl context kubectl config get-contexts
Deploy the sample application
The demonstration application is published on Docker Hub. It’s an API written in .NET Core, that just exposes two simple endpoints:
GET: status/health- Used for Kubernetes Health Probe
GET: status/ready- Used for Kubernetes Readiness Probe
The code for the API is located in this repository on GitHub. It also contains the corresponding
YAML definition-files that we use for the deployment to Kubernetes.
# Create a dedicated namespace kubectl create namespace scaling-demo # Deploy the application to AKS kubectl apply -f https://raw.githubusercontent.com/ThorstenHans/aks-cluster-scaling-demo/master/kubernetes/scaling-demo.yaml -n scaling-demo
Before we start scaling the deployment, let’s check how many worker-nodes our AKS cluster currently consists of:
# check AKS cluster nodes count kubectl get nodes --no-headers | wc -l 1
Initially, the deployment
demo in the
scaling-demo namespace will create
2 replicas. We can verify this using
kubectl get pods -n scaling-demo or using
kubectl get deploy -n scaling-demo.
Let’s scale the deployment to
40 replications using:
# scale demo deployment to 40 replicas kubectl scale deploy/demo -n scaling-demo --replicas 40 # verify Pods being created kubectl get po -n scaling-demo -w
Fire up a new terminal instance, and start watching your cluster worker-nodes:
# watch AKS worker-nodes kubectl get nodes -w NAME STATUS ROLES AGE. VERSION aks-nodepool1-11111111-vmss000000 Ready agent 35m v1.15.10
After a couple of seconds or minutes, you will see new worker-nodes appearing. To see the resource utilization per worker-node, use the
kubectl top nodes command.
# get AKS worker-nodes kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-11111111-vmss000000 Ready agent 7h46m v1.15.10 aks-nodepool1-11111111-vmss000001 Ready agent 2m22s v1.15.10 # get AKS worker-node utilization kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% aks-nodepool1-11111111-vmss000000 257m 13% 1796Mi 83% aks-nodepool1-11111111-vmss000001 98m 5% 883Mi 41%
We can scale-in the Kubernetes deployment again using
kubectl scale deploy/demo -n scaling-demo --replicas 2. Kubernetes will start terminating Pods immediately. However, scaling in worker-nodes takes some time. It takes some time, due to the default cluster auto-scaling profile and its settings, which we will look at in a few.
Retract the sample application
To retract the sample application, we can simply delete the entire namespace using the
kubectl delete namespace command.
#delete the entire sample namespace kubectl delete ns aks-scaling-demo
Custom cluster scaling behavior
The previously demonstrated scaling behavior is based on a default configuration provided by Microsoft. This behavior is excellent and works for common scenarios. However, from time to time, you may want more control. I prefer the auto-scaler to scale-in more aggressively. Or, perhaps you want to scale-out earlier. You achieve this, by providing a custom AKS cluster auto-scaler profile settings.
Currently, Microsoft exposes nine (9) different settings that can be customized to tailor the auto-scaler experience. To make scale-in behavior more aggressive, we have to alter the values of two settings.
scale-down-after-add have a default value of
10 minutes, which we will decrease, to get our cluster scaled-in earlier.
Check out the official documentation to get the full list of available settings to customize the auto-scaler experience.
Custom auto-scaler profiles affect all node-pools in an AKS cluster.
There is no chance to provide individual auto-scaler settings per node-pool at this point in time. To customize the auto-scaler profile settings, we have to install the AKS Preview extension for Azure CLI.
# Install the extension az extension add -n aks-preview # Update the extension to ensure the latest version is installed az extension update -n aks-preview
Specify custom sclaing profile settings
Once the AKS preview extension is installed, we can use
az aks update to modify the auto-scaler profile.
# set desiered profile settings to 3 minutes az aks update -g $RESOURCE_GROUP_NAME \ -n $AKS_NAME \ --cluster-autoscaler-profile scale-down-after-add=3m scale-down-unneeded-time=3m
With those customizations being applied to the cluster, we can cycle through the demo process once again and see a more aggressive scale-in behavior.
Cluster auto-scaler in combination with multiple AKS node-pools
If your AKS cluster is running with different node pools, you can configure cluster auto-scaler independently for every node pool using the
az aks nodepool commands.
For example, you can enable cluster auto-scaling for a given node pool using the following command:
# enable auto-scaler for node-pool az aks nodepool update -g $RESOURCE_GROUP_NAME --cluster-name aks-scaling-demo \ -n gpunodepool \ --enable-cluster-autoscaler \ --min-count 1 \ --max-count 5
You can update the boundaries or disable cluster auto-scaling correspondingly:
# update auto-scaler boundaries on node pool az aks nodepool update -g $RESOURCE_GROUP_NAME --cluster-name aks-scaling-demo \ -n gpunodepool \ --update-cluster-autoscaler \ --min-count 1 \ --max-count 5 # disable cluster auto-scaler on node pool az aks nodepool update -g $RESOURCE_GROUP_NAME --cluster-name aks-scaling-demo \ -n gpunodepool \ --disable-cluster-autoscaler
From my point of view, cluster auto-scaler in AKS is mission-critical and one of the features I was looking for since AKS was released. Worker-nodes will finally scale-in and -out depending on actual load. The cluster auto-scaler plays well in combination with Horizontal Pod Auto Scaling (HPA).
By using both, you can make your applications run efficiently, no matter if you are facing unexpected high or low usage. Additionally, the optimization of hardware utilization will have a direct impact on your monthly Azure bill.
Are you using Azure Kubernetes Service and facing problems while implementing proper cluster auto-scaler behavior? Reach out and let me know.