Fargate offers a great value proposition to AWS users: forget about virtual machines and just provision containers. Amazon will take care of the underlying hosts, so you will be able to focus on writing software instead of maintaining and upgrading a fleet of Linux instances. Fargate brings many benefits to the table, including small maintenance overhead, lower attack surface, and granular pricing.
However, as any cloud asset, leaving your AWS Fargate tasks unattended can lead to nasty surprises. If they are compromised, you may get a huge bill after someone uses your infrastructure for crypto mining. Or an attacker could perform cloud lateral movement to gain full access to your data.
Detecting issues like these requires deep visibility into the activity of your fargate workloads, exactly the same visibility you would get when running applications in traditional infrastructures.
Unfortunately, lack of access to the Operating system makes it very challenging to reach such a level of visibility in Fargate. Fortunately, Sysdig came up with the perfect solution to the problem!
But before talking about the solution, let’s describe the problem.
Current deep instrumentation approaches
When you run a runtime security product, commercial or Open Source, with sufficient granularity and support for containers, it’s almost guaranteed to employ one of these three main instrumentation techniques: ptrace, LD_PRELOAD, and kernel instrumentation.
These techniques make it possible to observe the system activity (file I/O, network, etc.) with a high degree of granularity (e.g., single access to disk, network payload content) and are necessary for true security.
When comparing these techniques, there are two important factors to consider: accuracy and overhead. Let’s learn more about the three techniques and how they compare with regards to these factors.
1. LD_PRELOAD
This technique consists of using an environment variable, LD_PRELOAD, to direct the operating system to load a different version of the libc dynamic library. Libc is used by the majority of the programs to invoke kernel functions like open() or connect(). The version loaded with LD_PRELOAD may contain additional instrumentation to record every call to these functions and their parameters.
LD_PRELOAD is relatively efficient, but it’s not accurate. That’s because many programs, such as anything written in go, don’t use the libc dynamic library and call the kernel directly. LD_PRELOAD will be completely blind to these types of programs. Additionally, since LD_PRELOAD is a user-level technique, it can be fooled by a motivated attacker, like calling system calls directly instead of using libc.
Finally, replacing system libraries is fairly risky and can cause serious stability issues in the instrumented application.
2. ptrace & friends
ptrace is one of the most advanced (and complicated!) pieces of functionality exported by the Linux kernel. It makes it possible for a process with the right privileges to pause, introspect, and control another process. Also, it should be noted that in this context, when we use the term ptrace we include other user level introspection techniques, like inotify or fanotify. They have the same privilege requirements and overhead profile of ptrace. Several tools you use on a regular basis are based on ptrace. One of them, for example, is gdb, the GNU debugger.
ptrace and the other similar techniques are very accurate, but they are not efficient. They are accurate because they are language and stack independent (i.e., they can see the activity of Go programs). Also, the information is produced by the operating system kernel and the kernel can’t be fooled, at least not easily. However, all of these techniques involve multiple context switches to hand the data to the collection process, which makes them very slow.
When something is too slow, the solution is to collect less data to reduce overhead, but this makes it less accurate. Thus, ptrace is either too slow or not very accurate.
3. Kernel instrumentation
This technique consists of going down to the kernel with traditional (and more invasive) methods based on kernel modules. Or use more modern, less invasive methods like eBPF, which is based on a virtual machine that can run code safely inside the kernel.
Kernel instrumentation is the most efficient and accurate way to collect system activity. It’s efficient because collecting the actions in the kernel guarantees the lowest overhead. It’s accurate because, as we know, “the kernel never lies.”
For this reason, Falco, Sysdig Secure, and Sysdig Monitor have been based on this technique. It is dramatically superior, both in terms of accuracy and performance to any other approach.
The problem with Fargate
The problem with Fargate instrumentation is easy to summarize: with no access to the host, kernel instrumentation is not available, so we have to choose between the LD_PRELOAD and ptrace-based approach. As a consequence, we have to sacrifice either accuracy, performance, or both. Or do we?
Introducing Sysdig’s advanced Fargate instrumentation
With the release of Fargate support in Sysdig Secure and Sysdig Monitor, Sysdig is coming to the rescue with a new patented technology that builds on top of the Open Source pdig framework and adds powerful optimizations that focus on the nature of serverless applications. The focus during the development of this solution was zero compromises: we wanted something that could offer the same data precision as kernel capture, full support for any type of executable (including Go programs!), and an overhead comparable to what you would experience in traditional containerized infrastructures.
The result? Let’s have the numbers tell us the story! The chart below shows the time it takes to run an I/O intensive workload (over 100k IOPS) in a Fargate container, with full accuracy (every I/O operation is recorded).
We can observe that, even in this demanding scenario, kernel instrumentation causes a minimal slowdown in the instrumented app. On the other hand, at least at full accuracy, ptrace’s impact is very substantial, with a slowdown of over 4X.
The Sysdig advanced Fargate instrumentation manages to contain the instrumentation overhead close to the one of kernel instrumentation, while simultaneously capturing every I/O operation and supporting languages like Go.
Conclusions
If you are serious about Fargate and its security, Sysdig’s advanced Fargate instrumentation brings you the deep visibility you need to confidently run and secure your workloads without having to accept compromises. Sysdig just announced AWS Fargate runtime security based on this technology. Check it out by requesting a 30-day free trial. You’ll be set in just minutes!