Digital Forensics and Incident Response (DFIR) is a fundamental part of the container security process in any organization.
Container forensics and postmortem analysis improves your security posture, helping to understand and contain the impact of any security breach, where your company revenue and reputation, may both, be at stake. Allows to respond faster to security incidents and recover effectively.
Container forensics it’s often a requirement on many data protection laws and security compliance policies.
Forensics answers questions after a security incident, like: where, what, when, who, why? When using containers, Docker and Kubernetes these questions become very specific:
- Which pod or external request initiated the attack?
- What happened inside? File access and modification, backdoor installation, cryptojacking?
- Did the attack spread to other systems?
- Was any sensitive information exposed and leaked?
- When did the incident cease?
- What if the attacker or target container do not exist anymore?
- Did Kubernetes kill the target pod?
- What were the last events and logs for that container?
- What was the actual content of all network connections or file activity?
- Who did it? How the attacker gain remote access?
- SQL injection or command injection?
- Was any malware, webshell or backdoor already installed?
- Did they exploit unsafe configuration?
- Used any leaked or weak credentials?
Why container forensics is difficult?
Forensic analysis and postmortem are not easy on distributed, dynamic environments, especially with the ephemeral nature of containers. You might have SSH’d into your machine to perform further analysis, but that doesn’t work with containers anymore:
- Containers are created and destroyed often, services are scaled up and down. Your affected container might have been destroyed already, or rescheduled into a different node.
- Container images are immutable, but their execution runtime is not. Their internal file systems and logs are ephemeral. When the container dies, any change in that container is also gone.
- Containers communicate over ad-hoc virtual software-defined networks, their internal IP addresses change frequently.
- Container security context like resource access and execution privileges are defined dynamically by the orchestration tool, like Kubernetes.
Reconstructing a meaningful timeline of events and system activity like executed commands, running processes, network connections, file system operations and system calls is almost impossible if your forensics toolkit doesn’t support the microservices and containers paradigm.
Then, are containers going to be an obstacle to provide security and forensic analysis throughout my organization? Only if you are using the wrong tools!
How to solve the challenges of container forensics and postmortem analysis?
Sysdig records pre and post attack activity through system calls, allowing us to analyze everything that happened – not only after the break-in, but also before so you can understand how they did it! To do so, Sysdig leverages its unique kernel level instrumentation: ContainerVision, to gain observability inside containers and running processes.
Sysdig tightly integrates with all the container orchestration tools like Kubernetes, Openshift, DC/OS Mesos or Docker Swarm, tagging all the collected information with the container metadata so you can filter events coming from an specific Kubernetes resource, for example. We call this ServiceVision.
The Sysdig container forensics toolset comes in different packages and licenses:
Container forensics and postmortem analysis in Kubernetes
This is an exciting use case for Sysdig Secure. Sysdig provides unrivaled container forensics capabilities that allow you inspect data outside of production, even if your containers are long gone. So let’s dig into how it works.
Sysdig Secure automatically audits every single command and process executed across your entire infrastructure and doesn’t require any changes in your apps or containers. You can browse through all the commands, filter or in this case – correlate them to specific incidents.
When a security incident is fired, Sysdig Secure can respond triggering a capture to collect even more information. A Sysdig capture is a file that contains all the system calls executed by the system, like a time machine, so you can explore what happened step by step. Actually, captures taken by Sysdig Secure can include pre-attack activity, potentially including precious information about how the security breach happened. This way you can analyze an endless list of different scenarios:
- Interactive terminals activity (docker exec or kubectl exec/attach)
- Access or modification of sensitive, configuration or binary files
- Application protocols to analyze SQL or command injections
- Commands executed, malicious scripts, malware, backdoor and rootkits
- Cryptomining / Cryptojacking
- Performance indicators, logs and Docker / Kubernetes events
- Low level information like raw I/O buffers, systems calls, page faults
Basically everything that happened in your system.
Sysdig Inspect, an interface built specifically for container forensics, inspired by popular security tools like Wireshark is integrated with Sysdig Secure. This way you can open capture files and conduct deep forensics of your containers directly from your browser.
So that’s just one example of how we can analyze an incident. Including looking into executed commands, finding the files that were accessed or modified, and in this case identifying malware that was downloaded into that container – even if the container it’s long gone.
DevSecOps teams and security analysts can now run containers in production with Sysdig Secure, without worrying how they might perform forensic analysis and postmortems after a security incident and recovering from security breaches effectively.