In my recent article “How to encrypt Kubernetes secrets with Mozilla SOPS”, I demonstrated how to encrypt regular Kubernetes secret manifests (yaml) using SOPS CLI in conjunction with Azure Key Vault as the backend for the encryption-key.

This post will take it one step further; it guides you through the process of creating a custom Helm chart that will reference encrypted secrets. In the end, you will be able to add your encrypted secrets to source control without leaking sensitive information.

We will again deploy a simple Web API to expose all environment variables via HTTP for demonstration purposes.

Prerequirements

This article assumes that you’ve installed Helm CLI and access to an instance of Azure Key Vault as described in the “How to encrypt Kubernetes secrets with Mozilla SOPS” article.

Install the helm-secrets plugin

To use encrypted, sensitive data in Helm, we have to install the helm-secrets plugin locally.

helm plugin install https://github.com/jkroepke/helm-secrets

# alternatively you can install a specific version
# see releases at https://github.com/jkroepke/helm-secrets/releases
helm plugin install https://github.com/jkroepke/helm-secrets \
    --version v3.8.3

Generate a secrets file

You can create new encrypted secrets using sops CLI. At this point, we will use an encryption key stored in Azure Key Vault.

sops --azure-kv $ENCRYPTION_KEY_ID secrets.yml

This will open your default editor; for demonstration purposes, let’s create two sensitive values:

username: thns
password: secret_value

Save the file and close the editor again. Take a look at the new secrets.yml file (cat secrets.yml); it contains the encrypted representation of your configuration data. If you want to get the decrypted secrets, you can use the helm secrets view command like this:

# decrypt & print all secrets 
helm secrets view secrets.yml

# username: thns
# password: secret_value

The secrets plugin also allows editing previously created and already encrypted secrets using helm secrets edit secrets.yml. Use helm secerts to get a list of all available sub-commands.

Create a custom Helm chart

You can create new Helm charts by using the helm create <chart_name> command. Ensure that the chart is generated as a sibling to secrets.yml.

ls
# total 1
# -rw-r--r--  1 thorsten  staff     1.3K Aug 11 11:24 secrets.yml

# create a new Helm chart called app
# helm CLI will place the chart in ./app
helm create app

At this point, Helm CLI will generate an example chart, which we will use as the starting point and apply some simple modifications before deploying it to Kubernetes.

Reference encrypted secrets

Now that the Helm chart has been created, add a new template to the app/templates folder with the name secret.yaml and provide the following content:

apiVersion: v1
kind: Secret
metadata:
  name: samplesecret
type: Opaque
data:
  username: {{ .Values.username | b64enc | quote }}
  password: {{ .Values.password | b64enc | quote }}

You can reference encrypted secrets like any other value in Go template language using the {{ .Values.NAME }} expression. Remember, references are case-sensitive. Next, update the app/templates/deployment.yaml and populate environment variables by adding a envFrom.secretRef to the sample deployment:

# omitted
containers:
  - name: {{ .Chart.Name }}
    image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
    imagePullPolicy: {{ .Values.image.pullPolicy }}
     envFrom:
      - secretRef:
          name: samplesecret
# omitted

Also in app/templates/deployment.yaml update ports.containerPort from 80 to 5000

# omitted
ports:
  - name: http
    containerPort: 5000
    protocol: TCP
# omitted

Last, you have set the desired container image and tag in app/values.yaml look for the image block and update it to match the following:

image:
  repository: thorstenhans/env-via-http
  pullPolicy: IfNotPresent
  tag: "0.0.1"

Deploy the application to Kubernetes

To deploy the Helm chart to Kubernetes, I have provisioned an Azure Kubernetes Service (AKS) instance. To isolate this sample from other applications running in the cluster, create a new Kubernetes namespace using kubectl:

# create a new Kubernetes namespace
kubectl create ns encrypted-secrets
# namespace/encrypted-secrets created

The previously installed plugin helm-secrets provides required commands to install a Helm chart that relies on encrypted secrets. Use helm secrets install and provide the desired Kubernetes namespace and the encrypted secrets file besides the reference to the chart itself as shown below:

helm secrets install ./app -n encrypted-secrets -f secrets.yml

Test the deployment

To test the deployment, create a port forwarding with kubectl and curl the root route as shown here:

# get all pods in desired namespace
kubectl get po -n encrypted-secrets

# NAME                   READY   STATUS    RESTARTS   AGE
# app-74cf6fdcbf-zbxx5   1/1     Running   0          99s

# create port-forwarding
kubectl port-forwarding -n encrypted-secrets \
    app-74cf6fdcbf-zbxx5 8080:5000

# Forwarding from 127.0.0.1:8080 -> 5000
# Forwarding from [::1]:8080 -> 5000

Open an additional terminal session and curl the root URL. The sample below pipes the standard output to jq to increase readability:

curl http://127.0.0.1:8080 | jq

{
  "hostName": "app-74cf6fdcbf-zbxx5",
  "envVars": [
    // omitted
    "username=thns",
    "password=secret_value",
    // omitted
  ]
}

Conclusion

Distributing containerized applications with Helm has become the default since the release of Helm 3. Using the Helm secrets plugin, you can also use encrypted secrets in your custom Helm charts. Storing the encrypted representation of your secrets in source control is no longer an issue, and you are not in danger of leaking sensitive information. Again, Mozilla SOPS builds the flexible yet robust foundation for all of this.