One of the most common questions I get when teaching developers in Kubernetes is

“How can I debug my application once it is deployed to Kubernetes?”

Although there are several ways how to achieve that, this post will explain and demonstrate how to debug applications running in Kubernetes using Bridge to Kubernetes (Bridge).

What is Bridge to Kubernetes

Bridge to Kubernetes (Bridge), the successor of “local process” is an IDE / editor extension. It allows developers to debug and test application counterparts (or microservices) on their local machine. While debugging locally, downstream and related components are consumed directly from within the Kubernetes cluster. Existing configuration objects - such as Kubernetes ConfigMaps, or Secrets -, and volumes mounted to the Pod are also applied to the local debugging instance seamlessly.

Bridge authenticates with your Kubernetes cluster by re-using existing kubectl authentication context. Basically, Bridge allows you to:

  • debug and test locally
  • debug in isolation when targeting a shared-cluster environment
  • debug using your preferred and familiar tools

Install Bridge

Bridge is currently available for Visual Studio and Visual Studio Code (VSCode). To install Bridge for Visual Studio, download and install the Bridge extension from Visual Studio Marketplace. In VSCode, you can browse for the Bridge to Kubernetes extension using the Extension view (Cmd+Shift+X or Ctrl+Shift+X). Alternatively, you can install the extension via command line using code --install-extension mindaro.mindaro.

Debugging a simple API

For demonstration purposes, we will debug an API written in Go. You can find the source of the API in this repository and the corresponding container image on Docker Hub.

Once you have cloned the repo, you find the manifests to deploy the application to Kubernetes in the kubernetes folder. The application consists of two things:

  1. a Kubernetes service of type LoadBalancer that routes traffic from the internet to the API
  2. a Kubernetes Deployment which is responsible for running the API

The following script shows how to deploy the application to a dedicated namespace in a Kubernetes cluster:

# set proper context for kubectl
kubectl config use-context aks-dev

# deploy the namespace
kubectl apply -f ./kubernetes/namespace.yml

# deploy the application
kubectl apply -f ./kubernetes/app

Once the deployment is finished, you should check the status of the Pod. It should be in state Running:

# get pods (namespaced)
kubectl get po -n bridge-demo

NAME                   READY   STATUS    RESTARTS   AGE
api-7cdf645b95-gv6x9   1/1     Running   0          8s

You should also check the service and its endpoints to get the public IP address, and to verify that the service points to the Pod:

# get services and endpoints (namespaced)
kubectl get svc,ep -n bridge-demo

NAME          TYPE           CLUSTER-IP   EXTERNAL-IP    PORT(S)          AGE
service/api   LoadBalancer   10.0.62.31   20.79.93.212   8080:30110/TCP   29s

NAME            ENDPOINTS          AGE
endpoints/api   10.244.1.10:8080   29s

Last, modify your kubectl context, to use the bridge-demonamespace by default.

# set bridge-demo as default namespace
kubectl config set-context --current --namespace bridge-demo

Open the repository in VSCode. Open the cmd/api/main.go file and fire up the Bridge configuration using the VSCode Command Palette (CMD+SHIFT+P). Look for Bridge: Configure. Bridge configuration is used to generate a debug configuration (launch.json) which we can use to quickly start debugging from now on. Bridge configuration will ask you a couple of questions to create a proper configuration. These are questions like:

  1. the desired Kubernetes service you want to intercept
  2. the local port you want to use for debugging
  3. the debugging configuration bridge should use for your application
  4. weather you want to debug in isolation from other developers

Let’s go through these questions

  1. Bridge will list all services from the currently selected Kubernetes namespace. If you want to change the namespace your kubectl is referring to by default, use kubectl config set-context --current --namespace ABC. If you followed the snippets above, you should see the api service. Select it.
  2. Bridge will spin up our API locally and forward traffic from the Kubernetes cluster to it. To do so, it must know on which port the API will run. Please provide 8080 as port. That’s the port used for running the API according to the getPort function in cmd/api/main.go:44.
  3. The repository comes with a predefined debugging configuration to debug the API locally. Select Launch Sample API to answer this question.
  4. Bridge supports debugging in shared cluster environments. We will discuss that in more detail later. For now assume that you are not running in a shared cluster environment. Answer the question with No.

Bridge will add a dedicated launch configuration called Launch Sample API with Kubernetes in .vscode/launch.json and it will add a new task to .vscode/tasks.json containing all necessary configuration to establish the connection with your Kubernetes environment. Now, having everything configured, ensure to start the previously generated Launch Sample API with Kubernetes debugging configuration by pressing F5.

Bridge will spin up the API locally and ensure traffic is routed from Kubernetes to your local machine. It will also ask for administrative privileges (more details on this later).

Debugging experience

Let’s add a new breakpoint to verify debugging works as expected. Open pkg/handlers/products.go and add a breakpoint in line 18 as shown in the following figure.

Set a breakpoint in VSCode

Grab the external IP address from our service using kubectl get svc. You can either use a browser or a tool like curl to issue a request as shown below:

# issue an request to list all products
# assuming the external ip address of the API service being set to 20.79.93.212

curl -iX GET http://20.79.93.212:8080/api/products

At this point, your VSCode instance should already be waiting at the breakpoint and you can debug the request issues to the Kubernetes cluster locally on your machine as shown in the following picture.

Local debugging experience in VSCode

Bridge behind the scenes

Bridge seems like magic, doesn’t it? However, if you break down the process of starting a debug session with Bridge, you will quickly recognize that the team utilizes common patterns and practices to get everything up and running:

  1. You are promoted to select the Kubernetes Service that should be replaced, and the local port that should be used for the debugging session
  2. Bridge replaces the containers in the pods on the cluster with an “remote agent” to forward route traffic to your local machine
  3. Bridge starts kubectl port-forward to allow network traffic flowing between the port - you specified in step 1 - on your local machine and the “remote agent”
  4. Bridge attaches variables (perhaps based on ConfigMap or Secret objects within Kubernetes) to your IDE / editor
  5. The local host file gets updated to map cluster services to your local IP addresses
  6. Bridge starts your application (or microservice) locally and attaches the debugger (potentially running local processes, that allocate specified ports for traffic routing, will be terminated by Bridge)

Currently, Bridge requires administrative permissions on the local machine to make changes to the host file, and to potentially terminate processes that allocate specified ports for traffic routing.

Debugging in a shared-cluster environment

Bridge also works in shared cluster environments. You can debug a certain parts of a bigger application, without interfering other cluster users. To achieve this, Bridge duplicates the selected services and associates a generated sub-domain. Every incoming request targeting the newly generated sub-domain will be modified. The custom HTTP header kubernetes-route-as will be added to requests. Provisioned Envoy proxies are responsible to route only requests with corresponding HTTP Header to your local debugging-instance.

Requests issued to the application that are not using the generated subdomain, won’t be modified and will be routed through the cluster as before.

Limitations

Bridge is under active development. At the point of writing this article, the following limitations are known:

  • Only single container pods are supported
  • Only Linux containers are supported
  • Bridge requires elevated privileges on your local machine to modify your hosts file
  • Bridge can’t be used in clusters running Azure Dev Spaces

The team shares the project roadmap on GitHub. Check the status of the project and its limitations from time to time.

Conclusion

Bridge dramatically simplifies the process of debugging distributed applications running in Kubernetes. The seamless integration with VisualStudio Code allows developers using their familiar development environment to debug and troubleshoot. Being able to debug in isolation allows using Bridge also in shared-cluster environments. Personally, I think Bridge is a must have for every developer using Kubernetes as application platform.