In our recent Fishing for Hackers blog post, we explored a sysdig trace of an actual system breach from an actual malicious attacker. Based on the interest in that post, and the great feedback that we’ve received since publishing it, we decided to spend some time making the forensics features of sysdig even better. Updates in the latest release include: * A new chisel, list_login_shells * Improvements to the spy_users chisel * Additional useful filter fields In this post, I’d like to show you how to use these new features to investigate your trace files with even more ease, and quickly identify that prime catch.
Sorting Through the Fishing Net
Other than in trivial situations, the commands executed on a unix system are launched from multiple login sessions, each of which can either be strictly interactive, or run scripts, or a mix of the two. From the forensics point of view, it’s important to quickly uncover the “interesting” sessions, separating them from noisy scripts – this is usually nontrivial and time intensive.
The revamped spy_users chisel provides better support for this kind of investigation. Let’s apply it to the trace file captured for the original Fishing for Hackers blog post.
You can notice a couple of things:
- Each line begins with a number: that’s the pid of the first shell of the session, in other words a unique ID for the session. That way, it’s possible to follow a specific session even when several of them are active at the same time.
- The executed commands are indented, based on the following rule: when a new shell is executed, the indentation increases. This is useful because, typically, a shell that executes another shell is running a script. Thus, the indentation clearly separates real users’ interactions from the scripts they execute.
Since interesting activity usually happens at higher levels in the tree (ie. less indentation), spy_users now includes an optional argument to filter the output based on the indentation level. For example
sysdig -r trace.scap.gz –c spy_users 0
will only show activity in an original starting shell, while
sysdig -r trace.scap.gz –c spy_users 1
will show activity in that shell and its children.
Finding the Prime Catch
Still, even with this improved readability, finding the needle in the haystack can be challenging, especially on busy machines. Fortunately, the new list_login_shells chisel comes to our rescue. In the simplest case, this chisel shows the ids of all the login sessions detected in the trace file. Let’s try it!
Quite a lot of activity! 38 sessions, to be precise. We know one of them contains a successful hacker penetration, but which one is it?
list_login_shells has two optional parameters. The first one makes it possible to search for sessions that contain the specified command name, while the second one does the same but with the command arguments.
In the context of detecting attacks, interesting sessions tend to contain commands like ssh and wget, and arguments that include /etc or /bin. In this case, I’m going to search for the execution of tar, assuming it’s a good indication that someone is trying to install something on my system:
Much better, this time I get only one session! Now I can use the proc.loginshellid sysdig filter field to restrict the output of spy_users.
Look at how clean this trace is. A hacker catch in all its splendor. Including the download of multiple malicious files and the execution of scripts with names such as whathefuckyouwant. And it keeps going on and on.
Conclusion
Hopefully these new features make it even easier for you to explore users’ activity within your sysdig traces (whether malicious or not!). In order to leverage the features shown in this post, make sure to update to the latest version of sysdig (instructions here). And as always, if you have comments, questions, or proposals for improvements, please let us know in the comments on Hacker News using the link below.
Thanks for reading our “Fishing for Hackers” blog post! We’re actually looking to create something security-specific with Sysdig and were looking for companies to brainstorm with. Do you have time to meet me next week to discuss? Contact us here and we’ll set up a meeting with Loris, the creator of sysdig.
