Add krustlet node pools to Azure Kubernetes Service

If we had traveled three years back in time, running WebAssembly (Wasm) workloads in Kubernetes would have sounded like fiction. Today, it’s real! With krustlet, developers can run Wasm workloads directly in Kubernetes. This article will look at what krustlet is and how we can bring it to our Azure Kubernetes Service (AKS) clusters.

What is Krustlet

Krustlet is a Kubernetes kubelet written in Rust to run Wasm workloads in Kubernetes. Nodes running krustlet have special tolerations assigned, which we can use within Kubernetes manifests, to ensure Wasm workloads will be scheduled to krustlet nodes. Under the covers, krustlet uses wasmtime as the runtime for our Wasm workloads using the wasm32-wasi architecture.

On top of that, we got WebAssembly Gateway Interface (WAGI), which allows us to create HTTP workloads with ease. When writing this article, WAGI is under active development and should be considered experimental.

Now that we know what krustlet is at a higher level, we can move on and create a Kubernetes cluster with a dedicated krustlet node pool. In the end, we will have an AKS cluster with two dedicated node pools. The default node pool, which is a Linux-based node pool. We can use this one to run regular workloads (speaking of Linux-based, containerized workloads). The 2nd node pool will consist of nodes running krustlet, which we can use to run our Wasm workloads.

AKS - krustlet node pools along with regular node pool

Provision an AKS cluster

First, let’s provision an AKS cluster along with an ACR instance using Azure CLI:

# variables
rgName="rg-blog-sample"
location="germanywestcentral"
aksName="aks-blog-sample"
acrName="blogsample"

# Login
az login

# set desired subscription
az account set --subscription <SUB_NAME or SUB_ID>

# create an Azure Resource Group
az group create -n $rgName -l $location

# create an Azure Container Registry
az acr create -n $acrName -g $rgName \
  -l $location \
  --sku Basic \
  --admin-enabled false

# grab ACR Id
acrId=$(az acr show -n $acrName \
  -g $rgName \
  -o tsv --query "id")

# create an AKS cluster
az aks create -n $aksName -g $rgName \
  -l $location \
  --enable-managed-identity \
  --attach-acr $acrId \
  --node-count 2

# grab AKS credentials
az aks get-credentials -n $aksName -g $rgName

# verify current kubectl context points to new cluster
kubectl config get-contexts

Before we extend our shiny new AKS cluster with krustlet, let’s inspect the currently provisioned cluster and take a look at the nodes using kubectl:

# Get nodes (custom columns used to increase readability)
kubectl get node -o custom-columns=Name:metadata.name,Runtime:status.nodeInfo.containerRuntimeVersion,OsImage:status.nodeInfo.osImage,Kernel:status.nodeInfo.kernelVersion

Name               Runtime                    OsImage              Kernel
aks-np1-vmss0000   containerd://1.4.9+azure   Ubuntu 18.04.6 LTS   5.4.0-1059-azure
aks-np1-vmss0001   containerd://1.4.9+azure   Ubuntu 18.04.6 LTS   5.4.0-1059-azure

Create a krustlet node-pool in AKS

The Azure team announced the public preview for running Wasm workloads using WASI in AKS in mid-October 2021. To use this preview feature, we must enable it on our Azure subscription before using it.

Register Wasm workload preview feature

We can easily register preview features in Azure using Azure CLI. Keep in mind that registering the preview feature may take some minutes (On average, it took around ten minutes -> registered it on four different subscriptions so far. So a perfect point in time to refill the ☕️). Once the preview feature is registered, you must refresh the underlying provider registration for Microsoft.ContainerService. You can always check the state of a feature registration via az feature list and correspondingly az provider list as shown in the following snippet:

# register WasmNodePoolPreview feature
az feature register --namespace "Microsoft.ContainerService" \
  -n "WasmNodePoolPreview"
# Grab a coffee ☕️ this took ~10 mins in average for my subs....

# check registration state for WasmNodePoolPreview
az feature list -o table \
  --query "[?contains(name, 'Microsoft.ContainerService/WasmNodePoolPreview')].{Name:name,State:properties.state}"

# Refresh provider registration
az provider register --namespace Microsoft.ContainerService

# verify provider registration
az provider list | grep Microsoft.ContainerService

Install the aks-preview extension in Azure CLI

We also have to install or update the aks-preview extension for Azure CLI itself. Adding aks-preview to Azure CLI will allow us to provision node-pools with krustlet:

# Install aks-preview extension for Azure CLI
az extension add -n aks-preview

# Update the aks-preview extension to latest the version
az extension update -n aks-preview

Provision Krustlet node-pool

Finally, we have everything in place to extend our AKS cluster and add a dedicated node-pool which we can use to run our Wasm workloads:

# variables
rgName="rg-blog-sample"
location="germanywestcentral"
aksName="aks-blog-sample"

# add a dedicated node-pool to our AKS cluster
az aks nodepool add --cluster-name $aksName \
    -g $rgName \
    -n wasmwasipool \
    --node-count 2 \
    --workload-runtime wasmwasi

# again, time to refill ☕️

Verify Krustlet nodes being added to AKS

Once the node pool has been added to the AKS cluster, we can use kubectl to inspect actual nodes available in the cluster. At this point, we should find four nodes compared to the two nodes we saw previously.

 # Get nodes (custom columns used to increase readability)
kubectl get node -o custom-columns=Name:metadata.name,Runtime:status.nodeInfo.containerRuntimeVersion,OsImage:status.nodeInfo.osImage,Kernel:status.nodeInfo.kernelVersion
Name                          Runtime                    OsImage              Kernel
aks-np1-vmss0000              containerd://1.4.9+azure   Ubuntu 18.04.6 LTS   5.4.0-1059-azure
aks-np1-vmss0001              containerd://1.4.9+azure   Ubuntu 18.04.6 LTS   5.4.0-1059-azure
aks-wasmwasipool-2-vmss0000   mvp
aks-wasmwasipool-2-vmss0001   mvp

We can already see the different container runtime (Runtime column) specified on our wasmwasi nodes. To spot individual labels, and taints being set on the actual node, we use kubectl describe no aks-wasmwasipool-2-vmss0000. The CLI will print all details about the desired node.

Name:               aks-wasmwasipool-2-vmss0000
Labels:             agentpool=wasmwasipool
                    kubernetes.io/arch=wasm32-wagi
                    kubernetes.io/os=wasm32-wagi
                    type=krustlet
Taints:             kubernetes.io/arch=wasm32-wagi:NoExecute
                    kubernetes.io/arch=wasm32-wagi:NoSchedule

Take the kubernetes.io/arch label for example; we can use this one as part of a nodeSelector to instruct Kubernetes deploying Wasm workloads to one of the nodes having that label assigned.

Conclusion

I’ve been following krustlet for quite some while now. Compiling existing applications against Wasm and leveraging WASI and wasmtime to bring applications to Kubernetes without leveraging a container will change the way we can utilize computing resources in the cloud. Although it is in preview, adoption is easy, and it is definitely worth giving it a try. I’m more than eager to see the broader adoption of Wasm in Azure. Having AKS as a first-class provider for Wasm workloads is bold.

I would love to see a serverless runtime for Wasm modules too. This would drive Wasm adoption (beyond the browser) even more because we would be able to offload workloads from Wasm nodes in Kubernetes to a serverless computing model 🚀. It’s time to start virtual-krustlet 💪🏼

Thorsten Hans