Hot-Reload .NET Configuration in Kubernetes with ConfigMaps

Leveraging ConfigMaps, we can inject configuration data into applications running in Kubernetes. In this article, we’ll look at how to use a Kubernetes ConfigMap to configure a .NET application from the outside and automatically reload changes applied to the ConfigMap.

Automatically reloading configuration data without redeploying the application and without doing a code change is super handy - not just for cloud-native applications. Being able to modify certain aspects of the application to hunt and pinpoint bugs is valuable and something you should consider adding to your applications too. For demonstration purposes, we will create a ConfigMap that allows us to modify the logging behavior of a simple HTTP API acting as the sample application.

The sample application
Kubernetes ConfigMap refresher
- Setting Environment Variables from ConfigMaps
- Mount ConfigMaps into Container filesystem
ConfigMap to control ILogger
Feeding .NET IConfiguration
Testing Configuration Hot-Reload in Kubernetes
Set the configuration folder using an Environment Variable
Conclusion

The sample application

The sample application - a fairly basic .NET HTTP API - exposes just a single GET endpoint at /verify. You can either download the entire sample code from GitHub or use the container images already published and available at ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:3632688430 and ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:latest.

Because we will modify the configuration of the actual application throughout this article, you will notice that we will use both tags (3632688430 and latest).

As mentioned at the beginning of the article, we’re going to control which logs will be sent to STDOUT from outside of our application. The /verify endpoint splits out a bunch of log messages (each using a different log level):

[HttpGet("verify")]
public IActionResult Verify()
{
  _logger.LogTrace("Trace from {HostName}", Environment.MachineName);
  _logger.LogDebug("Debug from {HostName}", Environment.MachineName);
  _logger.LogInformation("Info from {HostName}", Environment.MachineName);
  _logger.LogWarning("Warning from {HostName}", Environment.MachineName);
  _logger.LogCritical("Critical message from {HostName}", Environment.MachineName);
  _logger.LogError("Error from {HostName}", Environment.MachineName);
  return Ok();
}

The sample application comes with a default configuration, appsettings.json (and the same values for appsettings.Development.json), that defines the following logging configuration:

{
 "Logging": {
  "LogLevel": {
   "Default": "Information",
   "ThorstenHans.SampleApi": "Warning",
   "Microsoft.AspNetCore": "Warning"
  }
 }
}

When we run the app with the default configuration and send a GET request to /verify, we will see the following output written to STDOUT:

warn: ThorstenHans.SampleApi.Controllers.VerificationController[0]
   Warning from d5c83f9c8f02
crit: ThorstenHans.SampleApi.Controllers.VerificationController[0]
   Critical message from d5c83f9c8f02
fail: ThorstenHans.SampleApi.Controllers.VerificationController[0]
   Error from d5c83f9c8f02

The entire source code of the sample application is available on GitHub.

Kubernetes ConfigMap refresher

Haven’t you used ConfigMaps in Kubernetes yet? No problem. Before implementing hot-reload, let’s do a quick refresher and dive into ConfigMaps.

In Kubernetes, we use ConfigMaps group and organize our non-sensitive configuration data. Containers (Pods) running in the same Kubernetes-Namespace can use that configuration data.

As part of the Pod specification, we can use configuration data from ConfigMaps for two everyday things:

Set Environment Variables
Mount configuration data into the container filesystem

Setting Environment Variables from ConfigMaps

Each container of a Pod can have multiple Environment Variables. Again, we have two options here. We can either link a single item of the ConfigMap to a particular Environment Variable or create Environment Variables from all items specified in a specific ConfigMap. Before we dive into linking Environment Variables, let’s create a simple ConfigMap for demonstration purposes:

apiVersion: v1
kind: ConfigMap
metadata:
 name: my-sample-config
data:
 SortOrder: "asc"
 PageSize: "50"

Let’s take a look at linking a single item (called SortOrder from the my-sample-config ConfigMap):

apiVersion: v1
kind: Pod
metadata:
 name: api
spec:
 containers:
 - name: main
  image: ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:3632688430
  env:
   - name: API_SortOrder
    valueFrom:
     configMapKeyRef:
      key: "SortOrder"
      name: "my-sample-config"
  resources:
   limits:
    cpu: 50m
    memory: 128Mi
  ports:
   - containerPort: 5000

You can also mark the configMapKeyRef as required by setting optional to false.

As an alternative approach, we can link all items from the ConfigMap using envFrom and configMapRef. In this case, we can use the prefix property to decorate all Environment Variables linked by the desired ConfigMap with an individual prefix. Again we use optional to control if the configuration data is required or not:

apiVersion: v1
kind: Pod
metadata:
 name: api
spec:
 containers:
 - name: main
  image: ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:3632688430
  envFrom:
   - prefix: API_
    configMapRef:
     name: my-sample-config
     optional: false
  resources:
   limits:
    cpu: 50m
    memory: 128Mi
  ports:
   - containerPort: 5000

Although both approaches are handy and commonly used in Kubernetes, you should remember that Environment Variables are only passed to processes upon startup! Environment Variables are not reloaded once the process has started. That said, we should use Environment Variables for configuration data that must not be reloaded while the application runs.

Although using ConfigMaps to set Environment Variables does not address the actual requirement in the first place, we’ll use this mechanism later in the article to connect the dots 😁.

Mount ConfigMaps into Container filesystem

In addition to setting Environment Variables from a ConfigMap, we can mount the items of a given ConfigMap into a folder of the container’s filesystem. When using this approach, Kubernetes will create a symlink for every item of your ConfigMap in the specified folder. The content of every symlinked ConfigMap-item consists of the actual configuration data.

To achieve this, we must first define a dedicated volume and update our Pod specification by adding a volumeMount block to link the volume to a particular folder in the container filesystem:

apiVersion: v1
kind: Pod
metadata:
 name: api
spec:
 containers:
 - name: main
  image: ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:3632688430
  volumeMounts:
   - name: sampleconfig
    mountPath: /etc/sampleapi
    readOnly: true
  resources:
   limits:
    cpu: 50m
    memory: 128Mi
  ports:
   - containerPort: 80
 volumes:
  - name: sampleconfig
   configMap:
    name: my-sample-config
    optional: false
    items:
     - key: SortOrder
      path: SortOrder
     - key: PageSize
      path: ItemsPerPage
    defaultMode: 0644

As you can see, we have some additional options when defining the volume. We can pick particular keys from the ConfigMap, control the filename they’ll get using the items array, and control the default file permission using defaultMode. In this example, all selected items will be linked into the /etc/myapp folder, which is explicitly marked as readOnly.

ConfigMap to control ILogger

Now that everybody knows what ConfigMaps in Kubernetes is and what we can do with them. It’s time to revisit our actual requirement: We want to control which logs are written to STDOUT without modifying or redeploying our application.

To do so, let’s now build the necessary ConfigMap and specify which logs should be written to STDOUT. In contrast to the default behavior, we want our application to include logs of level Debug and higher when logs are produced from within the C# namespace ThorstenHans.SampleApi:

apiVersion: v1
kind: ConfigMap
metadata:
 name: api-config
data:
 Logging__LogLevel__Default: "Information"
 Logging__LogLevel__ThorstenHans.SampleApi: "Debug"
 Logging__LogLevel__Microsoft.AspNetCore: "Warning"

Feeding .NET IConfiguration

In .NET, we use IConfiguration to pull configuration data from different sources. Luckily, there is an existing extension method provided by Microsoft which addresses our needs: We will now use AddKeyPerFile to load configuration data which is linked from the Kubernetes ConfigMap to the configuration folder:

builder.Configuration.AddKeyPerFile("/etc/sampleapi", false, true);

Several overloads are available for AddKeyPerFile. In this case, we use the second parameter to mark this configuration source as required, and the third parameter is used to make .NET reload configuration data upon changes.

Unfortunately, reloading on changes has yet to work!

Every file in /etc/sampleapi is as symlink, and .NET is not able to pick up changes applied to the original file by just knowing about the symlink. Fortunately, we can instruct .NET to use a polling mechanism instead. It will periodically read all files in /etc/sampleapi and make changes available to our application.

Let’s update our Pod specification and set the DOTNET_USE_POLLING_FILE_WATCHER Environment Variable:

apiVersion: v1
kind: Pod
metadata:
 name: api
spec:
 containers:
 - name: main
  image: ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:3632688430
  env:
   - name: DOTNET_USE_POLLING_FILE_WATCHER
    value: "true"
  volumeMounts:
   - name: apiconfig
    mountPath: /etc/sampleapi
    readOnly: true
  resources:
   limits:
    cpu: 50m
    memory: 128Mi
  ports:
   - containerPort: 5000
 volumes:
  - name: apiconfig
   configMap:
    name: api-config
    optional: false
    defaultMode: 0644

Testing Configuration Hot-Reload in Kubernetes

Now, we can deploy both (the ConfigMap and the Pod) to any Kubernetes cluster and see the configuration modifications being applied without having to restart the application:

kubectl apply -f ./kubernetes/config-map.yml
kubectl apply -f ./kubernetes/pod.yaml

kubectl port-forward pod/api 5000:5000

You can use cURL or any other HTTP client of your choice to interact with the API:

curl http://localhost:5000/verify

Consult the logs and see logs with level Debug being written to STDOUT as shown here:

Go ahead, and modify the api-config ConfigMap by executing kubectl edit cm api-config and set Logging__LogLevel__ThorstenHans.SampleApi to Warning.

Keep in mind that it will take some time to update the actual configuration inside the container (about two minutes on my AKS cluster here). See the corresponding section of the Kubernetes documentation to understand why reloading the configuration may take some time. Once the value has reloaded, you will see only messages with level Warning and higher being logged to STDOUT.

Set the configuration folder using an Environment Variable

Although our implementation already works as expected, we should add another improvement. We now access the configuration folder (/etc/sampleapi) by using a magic string inside our application. We should refactor this to specify the folder name from the outside. To do so, we’ll set the ConfigurationFolder Environment Variable on our container and use it in combination with the AddKeyPerFile method we used in the previous section.

First, let’s revisit the final Pod specification (also note that we moved from tag 3632688430 to latest:

apiVersion: v1
kind: Pod
metadata:
 name: api
spec:
 containers:
 - name: main
  image: ghcr.io/thorstenhans/hotreloading-configuration-data-using-kubernetes-configmaps:latest
  env:
   - name: DOTNET_USE_POLLING_FILE_WATCHER
    value: "true"
   - name: ConfigurationFolder
    value: /etc/sampleapi
  volumeMounts:
   - name: apiconfig
    mountPath: /etc/sampleapi
    readOnly: true
  resources:
   limits:
    cpu: 50m
    memory: 128Mi
  ports:
   - containerPort: 5000
 volumes:
  - name: apiconfig
   configMap:
    name: api-config
    optional: false
    defaultMode: 0644

Having the ConfigurationFolder Environment Variable specified, let’s review the corresponding part of the .NET application that deals with it:

const string varName = "DOTNET_RUNNING_IN_CONTAINER";
var runningInContainer = bool.TryParse(Environment.GetEnvironmentVariable(varName),
  out var isRunningInContainer)
  && isRunningInContainer;

var configFolder = builder.Configuration.GetValue<string>("ConfigurationFolder");

if (runningInContainer &&
  !string.IsNullOrWhiteSpace(configFolder) &&
  Directory.Exists(configFolder))
{
  builder.Configuration.AddKeyPerFile(configFolder, false, true);
}
else
{
  Console.WriteLine($"ConfigurationFolder set to: '{configFolder}'.");
  Console.WriteLine($"ConfigurationFolder exists: {Directory.Exists(configFolder)}");
  Console.WriteLine($"Running in Container: '{runningInContainer}'.");
   
  Console.WriteLine("Relying on default configuration");
}

With this modification, we can configure everything from the outside without restarting or redeploying the container.

Conclusion

Especially when running applications in the (private-)cloud, you should consider implementing hot-reload for your configuration data. Flipping some switches while the app remains running is priceless and a real timesaver.

Kubernetes comes with batteries that make consuming configuration data from different sources easy and straightforward.

No matter what you do, think twice before you hammer a vendor-specific SDK into your application! Try to stay as cloud- and cloud-vendor-agnostic as possible. It’ll payout in the long run.

Thorsten Hans