Here we are again. Launch day. Having spent a good part of the past decade working with my team on Wireshark and WinPcap, I know how satisfying it is to pour your heart into a piece of free software and watch a community grow up around it. But there’s something uniquely exciting about the day you put something new out there. Today we are launching our third ambitious Open Source project, and this time, we’re focusing on system-level monitoring and troubleshooting. We are delighted to present to you: sysdig (website / github repo).
Sysdig is, for us, the natural evolution of our work with network packets. It’s the consequence of many things that, over the years, have continued to frustrate us about existing approaches.
One prime example is the missing link between the network and the host. This kills me. There are tools that let you see what’s happening on the wire, and different tools that let you look at activity inside your machines. But even though the two worlds are deeply related, the picture you get today is disconnected at best. So disconnected, in fact, that the simple task of understanding which process is sending data to the network is something that has eluded a clean solution.
Until now.
With sysdig you can do this:
> sysdig fd.type=ipv4
14:45:31.562700885 0 wget (3294) < write res=113 data=GET / HTTP/1.1..User-Agent: Wget/1.14 (linux-gnu)
14:45:31.563811184 0 wget (3294) > read fd=3(<4t>192.168.232.136:36489->192.168.232.139:80) size=313
14:45:31.563815235 0 wget (3294) < read res=313 data=HTTP/1.1 302 Found..Date: Sun, 30 Mar 2014 18:45:31 GMT
14:45:31.573750284 2 dropbox (1901) > recvfrom fd=22(<4u>192.168.74.136:17500->192.168.74.126:17500) size=65536
14:45:31.573769824 2 dropbox (1901) < recvfrom res=209 data={"host_int": 386459914, "version": [1, 8] ""
Or, if you like a simpler view, this:
> sysdig -ctopprocs_net Bytes Process ------------------------------ 3.59KB wget 2.50KB httpd 1.94KB dropbox
Another issue we found deeply frustrating is the huge drop off in quality of experience for network troubleshooting versus system-level troubleshooting. Looking at networks is done through elegant workflows that include saving the information using standard formats, exploring it with well-known filtering languages, displaying it through a de-facto standard user interface, Wireshark. Digging into system activity, on the other hand, still largely involves logging into the machine with SSH and using a plethora of dated tools with very inconsistent interfaces. And since most tools don’t offer any kind of history, you’re left struggling to reproduce the problem. Or even worse, just staring at the screen hoping that it happens again.
Not anymore.
With sysdig, you can easily create a trace file that you can export to a different machine:
sysdig -w savefile.scap
Then you can slice your trace data with a filter:
sysdig -r savefile.scap proc.name=mysqlor extract CPU usage:
sysdig -r savefile.scap -ctopprocs_cpu
or check where disk I/O is happening:
sysdig -r savefile.scap -ctopfiles
or follow the data exchange on a network connection:
sysdig -r savefile.scap -X -cecho_fds fd.cip=192.168.0.1
The information you need is all there in that file. So, before restarting a machine and hoping that the problem goes away, now you can take a trace. Just in case.
We also give you a bunch of cool ways to help you find what you’re looking for in the data – like Lua scripts, which we call chisels (to carve up the data you unearthed.. get it?). And we intentionally designed sysdig so that more cool functionality can be added super easily, by anyone in the community – like you!
This is just the first little piece of our grand plan (so stay tuned!), but we think it offers a pretty revolutionary approach. We’ve been working really hard on this, and we are very excited to bring it to you and to hear what you think. I hope it makes your day a little better.
A final plug: if you do enjoy using sysdig, we need your help spreading the word. Vote us up on Hacker News or Reddit, star our github repo, tweet at us, or write a blog post. Get involved by joining the official mailing list. And use the link below to comment. We’d love to hear from you. Thanks!
Sincerely,
Loris Degioanni Founder and CEO