fdbytes_by is one of my favorite chisels.
Quick aside: sysdig’s chisels are embedded scripts that analyze sysdig trace files or the live event stream to perform useful actions. Chisels are written in Lua, a well known, powerful, and extremely efficient scripting language.
fdbytes_by is relatively simple to use, but expressive enough to cover several interesting use cases, especially if combined with filtering. It lets you display disk, network, or any other type of I/O according to a criterium that you specify. It will update once per second if used on a live system, while it will print the full summary when applied to a file.
In this post I want to share the workflow that I use to explore disk I/O usage and understand what happens in a file system. I’ll walk through an example of how to use fdbytes_by to (1) identify the amount of file I/O in a system, (2) trace that I/O to a specific directory and specific files, (3) identify the process responsible for the I/O on each file, and (4) dive into the detailed I/O activity on each file.
The fdbytes_by Chisel
First, a basic example to show how the chisel works – here’s how to display I/O activity by FD type:
> sysdig –r trace.scap –c fdbytes_by fd.type Bytes fd.type ------------------------------ 485.00M file 4.63M unix 509.92KB pipe 99.83KB ipv4 53.79KB event 3.08KB inotify
We can see that this trace contains a lot of disk activity, and a bit of unix socket, pipe and network activity.
Using fdbytes_by requires a basic understanding of the sysdig type system. Sysdig encodes properties (e.g. the process name) and parameters (e.g. the return value) of the events it captures using a generic type system, so that any of those properties and parameters can be used to create filters or format the output. Chisels have access to the same type system, so they can leverage it too. The –l command line flag can be used to list the available event properties:
> sysdig –l ---------------------- Field Class: fd fd.num the unique number identifying the file descriptor. fd.type type of FD. Can be 'file', 'ipv4', 'ipv6', 'unix','pipe', 'event', 'signalfd', 'eventpoll', 'inotify' or 'signalfd'. fd.typechar type of FD as a single character. Can be 'f' for file, 4 for IPv4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pipe, 'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i' for inotify, 'o' for uknown. fd.name FD full name. If the fd is a file, this field contains the full path. If the FD is a socket, this field contain the connection tuple. fd.directory If the fd is a file, the directory that contains it. fd.filename If the fd is a file, the filename without the path. fd.ip matches the ip address (client or server) of the fd. fd.cip client IP address. fd.sip server IP address. …
Any of these fields can be used as an argument for fdbytes_by.
Visualizing file I/O activity by directory
Let’s see the places in the file system where most file I/O happened. We do that by using the fd.type=file filter to limit our analysis to file I/O, and using fd.directory as the grouping criterion.
> sysdig –r trace.scap –c fdbytes_by fd.directory "fd.type=file" Bytes fd.directory ------------------------------ 101.89M /tmp/ 90.26M /root/.ccache/tmp/ 85.75M /usr/include/c++/4.7.2/bits/ 24.74M /usr/include/c++/4.7.2/ 23.79M /usr/include/ 15.17M /root/agent/build/release/userspace/libsanalyzer/CMakeFiles/sanalyzer.dir/ 10.05M /usr/include/bits/ 9.82M /root/agent/build/release/userspace/libsanalyzer/CMakeFiles/sanalyzer.dir/root/sysdig/userspace/libsinsp/ 7.90M /root/.ccache/d/e/ 7.54M /root/agent/userspace/libsanalyzer/ 7.44M /usr/include/c++/4.7.2/x86_64-redhat-linux/bits/
Looks like there’s some serious compiling going on on this machine! A lot of the activity, in fact, happens in directories containing C or C++ include files. But the top directory, with 101MB of I/O, is /tmp. What’s happening in there?
Visualizing file I/O activity for each file in a directory
To find out which files have been accessed in /tmp, we use a filter that keeps just the activity in /tmp, and we group by file name:
> sysdig –r trace.scap –c fdbytes_by fd.filename "fd.directory=/tmp/" Bytes fd.filename ------------------------------ 13.64M ccJvb4Mi.s 9.33M cc4pbJkV.s 6.72M ccx9zir6.s 5.95M cc9ZG6af.s 4.05M ccTSZOV9.s 4.04M cc6IZVan.s 3.88M ccd7wm9E.s 3.81M ccG0IloP.s 3.73M ccxV9PLa.s 3.61M ccRxYpyU.s
A little trick here: fd.filename shows just the name of the file, with no path. If you are interested in the full path, use fd.name.
Based on their extension, those definitely look like compiler intermediate files, so they are probably related to the compiling happening on the machine. But let’s make sure.
Identifying file I/O activity by process name
Here’s how I can find out which processes accessed these files:
> sysdig –r trace.scap –c fdbytes_by proc.name "fd.directory=/tmp/ and fd.filename contains .s" Bytes proc.name ------------------------------ 50.57M as 49.03M cc1plus 1.54M cc1 1B httpd
Notice how this time the filter accepts only files in /tmp whose name contains “.s”. And, as expected, the result shows that the C++ compiler and assembler are the culprits.
Exploring detailed file I/O activity for a file
Need more details? We can find out exactly what was going on with those files by displaying the full I/O activity that they were involved with. Let’s pick for example the file on top of the list, ccJvb4Mi.s:
> sysdig -A –r trace.scap –c echo_fds "fd.filename=ccJvb4Mi.s" ------ Write 4.00KB to /tmp/ccJvb4Mi.s .file"chisel.cpp" .text .Ltext0: .section.text._ZNSt9exceptionC2Ev,"axG",@p ------ Write 4.00KB to /tmp/ccJvb4Mi.s _ZdlPvS_: .LFB382: .loc 3 117 0 .cfi_startproc pushq%rbp .cfi_def_cfa_off ------ Write 4.00KB to /tmp/ccJvb4Mi.s x), %rax testq%rax, %rax je.L26 .loc 5 250 0 movq-8(%rbp), %rax movq411 …
Notice how the first write on the file includes the name of a file, chisel.cpp, which is part of the sysdig source code repository! We can safely claim that the machine disk I/O, including the one in /tmp, was caused by somebody compiling sysdig. I guess that would be me!
This is of course just an example, but the takeaway here is the workflow around fdbytes_by. Once you get used to it, it will come pretty naturally and you can apply it to identifying disk bottlenecks for your servers, your databases and a bunch of other stuff.
My next post will cover using fdbytes_by to explore network usage, so stay tuned!