A critical vulnerability, CVE-2021-44228 known as “log4shell,” in Apache’s log4j was revealed on December 10th, 2021, and has already seen wide exploitation around the Internet. Previously, we discussed the vulnerability and how to find it in your images using Sysdig Scanning reports. In a perfect world, patching would be quick, easy, and completed without any issues.
In reality, there are often hurdles, both technological and bureaucratic, that make patching slower than we’d all like. Then there are third party devices and software which we simply may not have insight into to properly remediate a vulnerability. This is where a mitigation strategy comes into play. In this post, we will discuss how the log4j issue can be mitigated, in particular by leveraging Sysdig’s runtime generated Kubernetes network policies.
Traditional Mitigation
Web Application Firewalls (WAF) are often the first option for mitigating vulnerabilities that require HTTP transport. Google Cloud discusses using their Cloud Armor WAF to block requests with some of the JNDI strings known to be used in the exploitation of log4j.
Other WAF vendors have been releasing similar posts. While initially effective and easy to apply, these solutions are based on pattern matches, which can often be bypassed. This Github repo shows some of the many different ways the string can be changed to bypass signature detection.
Kubernetes Mitigation
If your environment is running Kubernetes, there is another option for mitigating log4j.
Network policies dictate how pods are able to communicate with network entities in and outside of your cluster. One of the key aspects of the log4j vulnerability is that it forces the server to reach out to another server in order to get its payload. This egress can be prevented with a Kubernetes network policy.
The tricky part of creating a Kubernetes network policy is making one that doesn’t interfere with the normal operations of your workloads but still blocks the threat. This is where Sysdig Secure’s runtime analysis of workloads is very valuable.
Sysdig will monitor your workloads as they run and start to understand how the application operates and what is normal. For example, below is what Sysdig understands about a simple Java-based application.
A more complex example would show common networks the application needs to communicate with and other relevant resources, but for this example, a simple application will work fine.
The key feature is that you can generate a Kubernetes network policy directly from this information by simply clicking on the Generated Policy tab. In this case, the generated policy for the example application would look like this:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: generated-network-policy
namespace: example-java-app
spec:
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
app: raw
chart: raw-0.2.5
heritage: Helm
release: namespaces
podSelector:
matchLabels:
app.kubernetes.io/instance: example-java-app
app.kubernetes.io/name: example-java-app-javaapp
ports:
- port: 8080
protocol: TCP
- to:
- namespaceSelector: {}
ports:
- port: 53
protocol: UDP
podSelector:
matchLabels:
app.kubernetes.io/instance: example-java-app
app.kubernetes.io/name: example-java-app-jclient
policyTypes:
- Ingress
- Egress
As seen in the runtime generated policy above, egress activity has been limited to resources installed using Helm and initiating DNS activity over UDP port 53. All other traffic would be disallowed, including the stage of the log4j exploit that downloads the malicious payload because it uses TCP.
An advanced attacker could try to exfiltrate data over DNS which would be allowed by the generated policy. If that is a concern, DNS monitoring may be required.
If egress is expected but not listed, individual IP Addresses or CIDR blocks that have been seen can be added under the Egress tab. The same applies to ingress traffic. Any additional changes should be made on the Ingress tab. The generated policy should be reviewed for accuracy before implementation.
NOTE: The example policy listed above will deny all ingress traffic as well.
To change the ingress settings, there is a handy dropdown box on the Ingress tab which will let you change the policy to some default settings such as allow all ingress.
With the network policy created, it can be applied using any of the tools employed by your DevOps teams, such as kubectl.
NOTE: Any previously applied network policies should be considered prior to applying a new one.
Conclusion
Ideally, the log4j vulnerability would be quickly and completely remediated. This is often not immediately feasible, so a mitigation strategy must be implemented in the meantime. WAFs are one option, but may have trouble keeping up with constantly mutating attack strings.
Sysdig’s runtime analysis and Kubernetes network policies provide another layer of mitigation, preventing the exploit from retrieving its malicious payload while still allowing your environment to operate normally.