A Sysdig + Kubernetes adventure, Part 2: Troubleshooting Kubernetes services

By on December 3, 2015

Welcome to the second half of this two-part series focusing on troubleshooting Kubernetes services with sysdig. In the first part, we studied how a Kubernetes service works: in particular, we explored an HTTP transaction across a Kubernetes service, starting from the “magic” DNS resolution done by SkyDNS and followed by the Kubernetes proxy intercepting requests leveraging the Linux Netfilter capabilities and forwarding them to the backend pods.

After studying the behavior under normal conditions, we are now ready to troubleshoot some real application failures. The key here is that we now know what ought to be expected from a properly behaving system, because we’ve thoroughly analyzed it already. As we’ll see, this workflow of “troubleshooting by visually spotting differences” will be very powerful.

Scenario #1: Wrong port configuration

Here’s the setup: I’ve now deployed a new Kubernetes cluster with pretty much the exact same application as before in Part 1: two nginx pods with a single Kubernetes service to load balance them. But this is what I get now:

root@ubuntu:/# curl my-nginx.default.cluster.local:8080
curl: (52) Empty reply from server

Weird. And the curl error doesn’t help me. Through a quick inspection via kubectl, everything seems to be up and running just fine:

gianluca@sid:~$ kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
my-nginx-tsyps         1/1       Running   0          2m
my-nginx-wu92j         1/1       Running   0          2m
ubuntu                 1/1       Running   0          2m

gianluca@sid:~$ kubectl get services
NAME         LABELS                                    SELECTOR       IP(S)        PORT(S)
my-nginx     run=my-nginx                              run=my-nginx   10.0.0.159   8080/TCP

Let’s explore with sysdig what happens under the hood. Once again, I’m going to use the same command lines as before for capturing the system events and reading them during the analysis. For brevity, I’m going to omit the steps of the troubleshooting that match the behavior of the previous use case, jumping right to the core of the issue.

Here we can see curl, after having correctly done the DNS resolution, trying to connect to the service virtual IP address (which matches the output of kubectl):

41582 13:17:57.400711569 3 curl (18949) > connect fd=3(<4>)
41817 13:17:57.401695929 3 curl (18949) < connect res=-115(EINPROGRESS) tuple=172.17.0.7:50476->10.0.0.159:8080

While curl is connecting to the service, we can see the proxy process accepting the connection (correctly translated by iptables, like before), and the proxy process immediately tries to connect to one of the nginx pods (172.17.0.6), in order to forward traffic to it:

41590 13:17:57.400854541 3 hyperkube (7654) < accept fd=9(<4t>172.17.0.7:50476->172.17.42.1:35039) tuple=172.17.0.7:50476->172.17.42.1:35039 queuepct=0 queuelen=0 queuemax=128
41622 13:17:57.401068116 3 hyperkube (7654) > connect fd=10(<4>)
41623 13:17:57.401113733 3 hyperkube (7654) < connect res=-115(EINPROGRESS) tuple=172.17.42.1:58682->172.17.0.6:8080

However, the connection is closed shortly right after:

41642 13:17:57.401180860 3 hyperkube (7654) > close fd=10(<4t>172.17.42.1:58682->172.17.0.6:8080)

And we can even see the proxy logging on its standard error (file descriptor 2) a message:

41644 13:17:57.401205751 3 hyperkube (7654) > write fd=2(<p>pipe:[35815]) size=107
41663 13:17:57.401257404 3 hyperkube (7654) < write res=107 data=
E1111 21:17:57.401198       1 proxysocket.go:99] Dial failed: dial tcp 172.17.0.6:8080: connection refused

Do you see why it didn’t work? The reason is that the proxy is trying to reach my pods on port 8080, not port 80, where nginx is listening! In this case, I forgot to specify --target-port=80 when declaring my service, so by default Kubernetes used as a target port the same one exposed by the service itself. When things like these happen, troubleshooting can be very painful!

Scenario #2: Wrong labels

Let’s try another mystery issue. In this example, curl is giving me the same error as before:

root@ubuntu:/# curl my-nginx.default.cluster.local:8080
curl: (52) Empty reply from server

However, what happens under the hood is substantially different. Here we can see the usual step of curl trying to connect to the service virtual IP address:

26690 13:40:49.770057107 2 curl (39926) > connect fd=3(<4>)
26759 13:40:49.770500704 2 curl (39926) < connect res=-115(EINPROGRESS) tuple=172.17.0.11:43268->10.0.0.215:8080

But right after accepting the translated connection, the proxy immediately closes the connection, without even attempting to connect to one of my backend pods (like in the previous case):

26698 13:40:49.770175853 2 hyperkube (35468) < accept fd=9(<4t>172.17.0.11:43268->172.17.42.1:33919) tuple=172.17.0.11:43268->172.17.42.1:33919 queuepct=0 queuelen=0 queuemax=128
26746 13:40:49.770455105 2 hyperkube (35468) > close fd=9(<4t>172.17.0.11:43268->172.17.42.1:33919)
26747 13:40:49.770455717 2 hyperkube (35468) < close res=0

Why did this happen? This clearly points out a misconfiguration in the Kubernetes service, because the proxy decided that there are no available pods, even though they are clearly here:

gianluca@sid:~$ kubectl get pods
NAME                   READY     STATUS    RESTARTS   AGE
my-nginx-llbe1         1/1       Running   0          24s
my-nginx-p7ooc         1/1       Running   0          13m
ubuntu                 1/1       Running   0          13m

It turns out that during the pod creation I made a typo when setting their labels, as I can see by running csysdig:

gianluca@sid:~$ sudo csysdig -v kubernetes_pods

Notice how the selector is looking for “my-nginx” pods, whereas the pod label is “my-nginxx”.

Scenario #3: Firewall misconfiguration

This time the error curl gives me is different:

root@ubuntu:/# curl my-nginx.default.cluster.local:8080
…
curl: (7) Failed to connect to my-nginx.default.cluster.local port 8080: Connection timed out

A timeout after a couple minutes. It’s worth digging in with sysdig to see what happens, the usual way. Here I see curl correctly trying to connect to the service:

56039 15:48:56.154725448 2 curl (69746) > connect fd=3(<4>)
56040 15:48:56.154883625 2 curl (69746) < connect res=-115(EINPROGRESS) tuple=172.17.0.19:46320->10.0.0.226:8080

But this is pretty much the only thing that happens. The proxy doesn’t seem to receive any incoming connection or do any network traffic like we’ve seen in the previous cases, so it definitely can’t forward any request to the backend pods. It seems almost as if there was a system misconfiguration in iptables and the routing logic for the virtual IP address wasn’t there. Let’s check:

# iptables -t nat -L
Chain KUBE-NODEPORT-CONTAINER (0 references)
target     prot opt source               destination

Chain KUBE-NODEPORT-HOST (0 references)
target     prot opt source               destination

Chain KUBE-PORTALS-CONTAINER (0 references)
target     prot opt source               destination

Chain KUBE-PORTALS-HOST (0 references)
target     prot opt source               destination

Completely empty! No wonder curl gives me a timeout error, since it is actually trying to reach a real host with IP address 10.0.0.226 somewhere according to my network routing table, which obviously doesn’t exist. The culprit here is a custom script that I ran that flushes all the iptables rules and, although Kubernetes recreates them periodically to ensure long term consistency, when I ran curl I was still in this vulnerable time window and the traffic wasn’t intercepted by the proxy. The solution here is to tweak my script to not be so disruptive and avoid flushing iptables rules for the Kubernetes chains!

Scenario #4: Accessing a service from the host

So far I’ve been accessing the service using a client pod running inside Kubernetes itself. Sometimes it might be convenient to access the service directly from the host. This is what happens if I run the same curl command from my host:

gianluca@sid:~/infrastructure/test-infrastructures/kubernetes-demo$ curl my-nginx.default.cluster.local:8080
..
curl: (7) Failed to connect to my-nginx.default.cluster.local port 8080: Connection refused

After a couple minutes, curl returns this error. Using sysdig, here we have curl connecting to the DNS server, and receiving a reply from it:

22611 16:15:46.985067704 0 curl (91366) < connect res=0 tuple=172.16.189.136:47319->172.16.189.2:53
...
22623 16:15:46.987606830 1 curl (91366) < recvfrom res=103 data=
	0x0000: 584d 8180 0001 0001 0001 0001 086d 792d  XM...........my-
	0x0010: 6e67 696e 7807 6465 6661 756c 7407 636c  nginx.default.cl
	0x0020: 7573 7465 7205 6c6f 6361 6c00 0001 0001  uster.local.....
	0x0030: c00c 0001 0001 0000 0005 0004 ac1e 0001  ................
	0x0040: 0000 0200 0100 0000 0500 0c03 6e73 3102  ............ns1.
	0x0050: 6b61 036c 7476 00c0 4b00 0100 0100 0000  ka.ltv..K.......
	0x0060: 0500 04ac 1b01 01                      .......
 tuple=172.16.189.2:53->172.16.189.136:47319

But this is clearly wrong, notice how there isn’t any activity from the skydns process, as we’ve seen before, and yet curl received a DNS response. Obviously curl is contacting another DNS server, 172.16.189.2, which is the one offered by my provider. In fact, curl then tries to connect to a totally random host on another network:

22676 16:15:46.989700389 1 curl (91366) > connect fd=4(<4>)
22677 16:15:46.989707967 1 curl (91366) < connect res=0 tuple=172.16.189.136:40960->172.30.0.1:8080

This will eventually fail because that host either does not exist or is just letting all connections on port 8080 timeout. Correlating this with the previous normal baseline behavior, I can see that skydns listens to requests on localhost (:::53 in the initial trace), and so the fix is as simple as adding “nameserver 127.0.0.1” to my /etc/resolv.conf file, and trying again:

gianluca@sid:~$ curl my-nginx.default.cluster.local:8080

<html>
<head>
<title>Welcome to nginx!</title>
<style>

With this little change, now I can contact my Kubernetes service, using its user-friendly DNS name, right from my host.

Conclusions

As we’ve seen, Kubernetes brings a lot of components that distributed application developers can leverage to be more efficient, and one of those is definitely the concept of Kubernetes services. We have also seen how these components, while working in an almost magical way, can face significant troubleshooting challenges when something doesn’t work as expected. Hopefully though, container-native troubleshooting tools like sysdig can help you overcome these difficulties and deploy Kubernetes in production with confidence.

Sysdig is available open source at sysdig.org, or sign up for a free trial of Sysdig Cloud today and monitor your entire distributed Kubernetes cluster in a single pane of glass!

Happy digging!




Eager to learn more? Check out our online session: Sysdig + Grafana, Dashboards or Metric Exploration, Which One Should You Use?

Join us for a discussion of what each of these solutions provides to help simplify monitoring and troubleshooting across a wide scope of technologies including Prometheus, Docker, and Kubernetes


Stay up to date!

Get new articles from this blog (weekly)
Or container ecosystem updates (monthly)

Thanks so much for signing up!
Please check your inbox for a confirmation email.