Detecting malicious processes is already complicated in cloud-native environments, as without the proper tools they are black boxes. It becomes even more complicated if those malicious processes are hidden.
A malware using open source tools to evade detection has been reported. The open source project used by the malware is libprocesshider, a tool created by Sysdig’s former chief architect Gianluca.
The libprocesshider tool can hide a process exploiting Linux’s library preloading technique. Through overwriting the key function readdir() provided in libc via a malicious shared library, tools commonly used to monitor processes, like ps or top, will be blinded.
In this article, we’ll dig into how this hiding processes scenario can be applied to containers and Kubernetes clusters. We’ll also cover how Falco can detect and mitigate these attacks.
Hiding a process inside a container
For this example, we’ll use libprocesshider to hide a non-malicious process, sleep.
In order to do that, we first edit processhidder.c and set the process_to_filter constant:
/* * Every process with this name will be excluded */ static const char* process_to_filter = "sleep";
Then, we compile the malicious C code as a shared library.
Next, we add it to /etc/ld.so.preload.
This config file defines a list of additional, user-specified, ELF-shared libraries to be loaded before all others. It is a powerful tool that can be used to selectively override functions in other shared libraries.
This is how the sleep process usually looks when listed by ps:
We can see the sleep process running with the PID 1182.
This is how it looks after the malicious library is added to /etc/ld.so.preload:
The sleep process is gone.
The same thing happens when using top:
However, the process is still alive. It is just not visible in the readdir() function which monitoring tools like ps or top depend on.
We could find the sleep process in the /proc file system, which contains the container’s system runtime information:
Another thing to note is that tampering with /etc/ld.so.preload within the container doesn’t impact the process view from the host:
Calling docker top from the host still shows the process running inside the container.
The PID that we are seeing is the one assigned in the host level process namespace. (i.e., tail had PID 1 within the container).
The conclusion we can draw here is that tampering with a container file system doesn’t impact the host (assume there is no host path mounting).
What about Kubernetes?
Hiding a process in a Kubernetes Pod
When using a sidecar to monitor an application container, it is common to enable the shareProcessNamespace feature in a Kubernetes pod. By sharing the process namespace, the application processes are visible from the sidecar container.
Here is a simplified example of how you would set up such a Pod:
apiVersion: v1
kind: Pod
metadata:
name: shared
labels:
app: nginx
spec:
shareProcessNamespace: true
containers:
- name: ubuntu
image: ubuntu
command: ["sh"]
args: ["-c", "--", "while true; do top -n 1 -b; sleep 1; done"]
env:
- name: CONAINER_NAME
value: ubuntu
- name: nginx
image: kaizheh/nginx
env:
- name: CONAINER_NAME
value: nginx
The ubuntu Pod is used to monitor the nginx Pod via top:
In a regular scenario, after executing sleep in the nginx pod, we can find it in the ubuntu container’s log:
Now, if an attacker gains access to the nginx Pod and hides a malicious process:
# echo /usr/local/lib/libprocesshider.so >> /etc/ld.so.preload
The sleep process is invisible when running ps in the nginx container. However, it is still visible from the ubuntu container’s perspective:
This is because the ubuntu container has its own libraries, which are different from the ones in the nginx container.
How can we hide the sleep from the ubuntu container?
Due to the containers sharing processes, an attacker can get a lot of information of the ubuntu container by executing ps inside the nginx container. In particular:
USER PID … COMMAND root 6 … sh -c -- while true; do top -n 1 -b; sleep 1; done
This looks like the entry process for the ubuntu container, with PID 6.
Now, an attacker can infer that ubuntu’s root filesystem is located at /proc/6/root. Then they only need to copy the malicious shared library object to the right place:
# echo /usr/local/lib/libprocesshider.so > /proc/6/root/etc/ld.so.preload
Then the sleep process is invisible from the ubuntu container as well:
The sidecar container, which is used to monitor the application container, is now useless.
Similarly, monitoring tools that use the host level process namespace to monitor other containers can be fooled if the attacker gains access to the monitoring pod.
Unveiling the malicious process with Falco
Now we know how this attack works. But how could we detect it?
As Falco taps into the system events, it can detect the /etc/ld.so.preload file being modified and flag it as a suspicious event.
Though there is a “write below etc“ rule included out-of-the-box in Falco, we can create a new rule to address this exact scenario:
- rule: Modify ld.so.preload desc: Detect an attempt to write to file /etc/ld.so.preload condition: open_write and fd.name=/etc/ld.so.preload output: "File below /etc/ld.so/preload opened for writing (user=%user.name command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)" priority: WARNING tags: [filesystem, mitre_persistence, mitre_defense_evasion]
And once there is an attempt to modify /etc/ld.so.preload, it will trigger the following Falco alert:
00:44:41.298026834: Warning File below /etc/ld.so/preload opened for writing (user=root command=bash parent=containerd-shim pcmdline=containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/43fecfe06f03b0bd833a2ea0e28d59008d7d10ab0643503480685d04817471be -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc file=/etc/ld.so.preload program=bash gparent=containerd ggparent=systemd gggparent=<NA> container_id=43fecfe06f03 image=kaizheh/ubuntu)
As also pointed out in Gianluca’s article, even though the malicious process is concealed, the activities from the malicious process can’t be hidden. Syscalls never lie.
Other Falco rules can be used to detect typical malware (including crypto miner behaviors):
- Container Drift Detected (chmod): Detect if a malware landed and prepared to execute.
- Outbound Connection to C2 Servers: Detect network connection to command and control servers.
- Detect outbound connections to common miner pool ports: Detect outbound connections to crypto mining pools.
- Write below etc: Most malware establishes persistence through cronjob.
Other mitigation strategies
Besides using Falco as a detection mechanism, there are a few other ways to mitigate tamperings with the preload file. Most are considered container image security best practices.
Enabling read only root filesystem: Once the read only root file system is enabled, any write to the root filesystem, like /etc, is forbidden.
Running as non-root user: If an attacker gains access to the monitoring pod through application vulnerability, the attacker will only get non-root privileges. They won’t be able to modify the /etc/ld.so.preload file.
Using a minimal base image: In order to support dynamic loading, the container file system requires libraries like libdl.so to load shared libraries. That said, for microservices, the required libraries can be known at the development stage, so it’s a good practice to restrict the image from supporting dynamic loading shared libraries.
Apply an AppArmor profile: AppArmor profiles, with their proper fine-grained control, are perfect to prevent file tampering. The sample profile below can be applied to protect the /etc folder from being tampered.
profile k8s-apparmor-deny-write-below-etc flags=(attach_disconnected, mediate_deleted) {
# allow to be terminated by runc
signal (receive) peer=unconfined,
file,
deny /etc/** w,
}
Any file tampering under the /etc folder will be blocked:
Conclusion
In order to discover defense evasion techniques like libprocesshider, you’ll need a good File Integrity Monitoring solution.
However, containerized environments make FIM more challenging.
Each container in the cluster has its own filesystem. If the read only root file system is not enabled, important files can be tampered. Also, containers come and go and are usually short-lived. Detecting files changes in such an ephemeral environment becomes even harder.
Sysdig is able to provide deep insight into your cloud native environment through system calls at the kernel level that are enriched with cloud native context. Please check out Falco and Sysdig Secure for more information.
If you would like to find out more about Falco:
- Get started in Falco.org.
- Check out the Falco project in GitHub.
- Get involved with the Falco community.
- Meet the maintainers on the Falco Slack.
- Follow @falco_org on Twitter.