A Sysdig + Kubernetes adventure, Part 1: How Kubernetes services work

By on December 2, 2015

We recently released Kubernetes support for our open source visibility and troubleshooting tool, sysdig, and the feedback from the community has been overwhelmingly positive – so today I want to continue the technical discussion on Kubernetes by going through a few examples of advanced Kubernetes troubleshooting. In particular, I’m going to focus on Kubernetes services.

To refresh your memory, a service is a Kubernetes abstraction that works by providing a convenient and single entry point to access a group of Kubernetes pods. In other words, a service can be thought of as a dynamic load balancer for a set of pods (and the containers living inside them), automatically managed by the framework itself.

I absolutely love services because they represent a very clean and consistent pattern that allows developers to write applications more efficiently, without having to worry about hand-crafted and bug-prone service discovery mechanisms. I am also surprised by how well they work – so much so that I decided to study them with sysdig!

In this two-part series, we’ll first explore how services work from a technical point of view, to see how all the magic is implemented under the hood. In the second part, we’ll do a similar kind of exploration, but in faulty environments, trying to find the cause of various issues.

Setting up a Kubernetes service

I am a firm believer that in order to be effective at troubleshooting when anomalous conditions happen, one needs to be well aware of how the application works in a healthy environment: this way, by understanding the normal behavior of the system, spotting differences is much easier. Let’s start by studying how a service behaves in a fully working environment.

I’m going to make the setup as simple as possible: it consists of two nginx pods, serving the standard index.html page that comes with nginx, and a single service that automatically balances the incoming traffic between them.

Creating the pods is as simple as running:

gianluca@sid:~$ kubectl run my-nginx --image=nginx --replicas=2 --port=80
my-nginx     my-nginx       nginx      run=my-nginx   2

And for the service:

gianluca@sid:~$ kubectl expose rc my-nginx --port=8080 --target-port=80
NAME       LABELS         SELECTOR       IP(S)     PORT(S)
my-nginx   run=my-nginx   run=my-nginx             8080/TCP

This last command creates a service, named my-nginx, that exposes a virtual IP address through which other pods in the cluster are transparently able to access the two backend nginx pods, using port 8080. This IP never changes for the entire lifespan of the service, which makes it incredibly useful for service discovery, because even if all my pods die and are rescheduled somewhere else, the entry point never changes and will not break the other parts of the application. I can get the virtual IP address with:

gianluca@sid:~$ kubectl get services
NAME         LABELS                                    SELECTOR       IP(S)       PORT(S)
my-nginx     run=my-nginx                              run=my-nginx   8080/TCP

But, since IP addresses are notably hard to remember, I prefer to use nice DNS names for my services, and Kubernetes has a very cool DNS extension that, once installed, runs a local DNS server inside the cluster and automatically translates DNS requests with the service name in it to the proper virtual IP address. Absolutely magic!

Now that the infrastructure is up and running, before jumping into the low level details let’s run a pod with a simple Ubuntu container that doesn’t run anything other than a shell. I am going use this pod as a main access point to interact with my application from inside the cluster:

gianluca@sid:~$ kubectl run -i --tty client --image=tutum/curl
Waiting for pod to be scheduled
Waiting for pod default/client-wd32o to be running, status is Pending, pod ready: false
Waiting for pod default/client-wd32o to be running, status is Running, pod ready: false

And once I’m in the shell I can finally interact with my service, using the DNS extension:

root@ubuntu:/# curl my-nginx.default.cluster.local:8080

<title>Welcome to nginx!</title>

Digging into services

Now that the environment is set up, I’m going to take a sysdig capture while curl runs:

gianluca@sid:~$ sudo sysdig -w trace.scap

Once the capture has been taken, I am going to use sysdig to read it, visualizing only the network-related events:

gianluca@sid:~$ sysdig -r trace.scap -A fd.type=ipv4 or fd.type=ipv6

To start, I can see curl connecting to the DNS server, because obviously it needs to resolve the IP address for my-nginx.default.cluster.local:

43340 15:40:58.352366394 3 curl (51152) > connect fd=3(<4>)
43341 15:40:58.352375784 3 curl (51152) < connect res=0 tuple=>

The DNS server that serves this request is, as predicted, the skydns process, which is part of the Kubernetes DNS extension and runs on my local machine (IP We can see the binary data that is sent by curl, corresponding to a typical DNS request:

43373 15:40:58.352568546 2 skydns (40700) > recvmsg fd=6(<3t>:::53)
43374 15:40:58.352570915 2 skydns (40700) < recvmsg res=48 size=48 data=
	0x0000: 0942 0100 0001 0000 0000 0000 086d 792d  .B...........my-
	0x0010: 6e67 696e 7807 6465 6661 756c 7407 636c  nginx.default.cl
	0x0020: 7573 7465 7205 6c6f 6361 6c00 001c 0001  uster.local.....

And this is where the magic starts happening: the skydns process converts the DNS request to an HTTP request to etcd (port 4001) for the key “skydns/local/cluster/default/my-nginx”, to which etcd replies with the key value (in JSON), containing, among other things, the IP address of the service,, which is then returned to curl under the form of a DNS response:

43405 15:40:58.353069087 2 skydns (40714) > write fd=3(<4t>> size=182
43406 15:40:58.353099313 2 skydns (40714) < write res=182 data=
GET /v2/keys/skydns/local/cluster/default/my-nginx?quorum=false&recursive=true&sorted=false HTTP/1.1
User-Agent: Go 1.1 package http
Accept-Encoding: gzip

43468 15:40:58.353384452 1 skydns (48700) > read fd=3(<4t>> size=4096
43469 15:40:58.353388313 1 skydns (48700) < read res=406 data=
HTTP/1.1 200 OK
Content-Type: application/json
X-Etcd-Cluster-Id: f68652439e3f8f2a
X-Etcd-Index: 166
X-Raft-Index: 1205
X-Raft-Term: 2
Date: Thu, 05 Nov 2015 23:40:58 GMT
Content-Length: 205


Now that curl got the IP address of the service from skydns, it can connect to it on port 8080 and send the HTTP request:

43645 15:40:58.356720087 2 curl (51151) < connect res=-115(EINPROGRESS) tuple=>
43763 15:40:58.357571155 2 curl (51151) > sendto fd=3(<4t>> size=99 tuple=NULL
43772 15:40:58.357612691 2 curl (51151) < sendto res=99 data=
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: my-nginx.default.cluster.local:8080
Accept: */*

However, this is where things get a bit odd. As you can see next, the hyperkube process seems to intercept the connection that curl sent, and read the original HTTP request from it. In particular, notice how the source port seems to be in both cases 50950, but the destination IP address and port seem to have magically changed from to

43795 15:40:58.357730454 2 hyperkube (39579) > read fd=10(<4t>> size=32768
43796 15:40:58.357733201 2 hyperkube (39579) < read res=99 data=
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: my-nginx.default.cluster.local:8080
Accept: */*

The trick used by Kubernetes here is to leverage the Linux kernel Netfilter capabilities to create a virtual IP that never leaves the host. As you can read from my iptables configuration below, every time someone tries to connect to, it will be redirected to, where the Kubernetes proxy server (shown as hyperkube in my sysdig trace file) will get the request and will properly route it to one of the backend nginx pods:

gianluca@sid:~$ sudo iptables -t nat -L
Chain KUBE-PORTALS-CONTAINER (1 references)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere              /* default/my-nginx: */ tcp dpt:http-alt redir ports 39551

Chain KUBE-PORTALS-HOST (1 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere              /* default/my-nginx: */ tcp dpt:http-alt to:

Once the proxy is done reading the HTTP request, it can normally forward it to one of the nginx pods ( in this case), where the nginx process can serve the transaction:

43798 15:40:58.357735649 2 hyperkube (39579) > write fd=11(<4t>> size=99
43799 15:40:58.357755452 2 hyperkube (39579) < write res=99 data=
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: my-nginx.default.cluster.local:8080
Accept: */*

43907 15:40:58.358733815 2 nginx (40473) > recvfrom fd=3(<4t>> size=1024
43908 15:40:58.358735877 2 nginx (40473) < recvfrom res=99 data=
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: my-nginx.default.cluster.local:8080
Accept: */*

And this is how Kubernetes services are implemented! Notice that there might be different behaviors depending on the specific Kubernetes configuration and version, but this represents a very common behavior.


Today we explored a pretty advanced use case, and we went a long way understanding one of the trickiest parts of Kubernetes, all with sysdig. If we pair this with the Kubernetes support described in our previous blog post, we can immediately see how sysdig can be helpful in a variety of situations, from basic Kubernetes exploration with csysdig’s views to advanced process troubleshooting with sysdig’s system and Kubernetes filters. And if you want to keep an eye on your whole multi-node Kubernetes cluster, we even have Sysdig Cloud!

In part two, we’ll leverage all the knowledge gained here to troubleshoot faulty environments where Kubernetes services are misbehaving. Stay tuned!

Eager to learn more? Check out our online session: Building an Open Source Container Security Stack

On this session Sysdig and Anchore are presenting how using Falco and Anchore Engine you can build a complete open source container security stack for Docker and Kubernetes.

This online session will live demo:

  • Using Falco, NATS and Kubeless to build a Kubernetes response engine and implement real-time attack remediation with security playbooks using FaaS.
  • How Anchore Engine can detect software vulnerabilities in your images, and how can be integrated with Jenkins, Kubernetes and Falco.

Stay up to date!

Get new articles from this blog (weekly)
Or container ecosystem updates (monthly)

Thanks so much for signing up!
Please check your inbox for a confirmation email.