In Kubernetes, managing and analyzing network traffic poses unique challenges due to the ephemeral nature of containers and the layered abstraction of Kubernetes structures like pods, deployments, and services. Traditional tools like Wireshark, although powerful, struggle to adapt to these complexities, often capturing excessive, irrelevant data – what we call “noise.”
The Challenge with Traditional Packet Capturing
The ephemerality of containers is one of the most obvious issues. By the time a security incident is detected and analyzed, the container involved may no longer exist. When a pod dies in Kubernetes, it’s designed to instantly recreate itself again. When this happens, it has new context, such as a new IP address and pod name. As a starting point, we need to look past the static context of legacy systems and try to do forensics based on Kubernetes abstractions such as network namespaces and service names.
It’s worth highlighting that there are some clear contextual limitations of Wireshark in cloud native. Tools like Wireshark are not inherently aware of Kubernetes abstractions. This disconnect makes it hard to relate network traffic directly back to specific pods or services without significant manual configuration and contextual stitching. Thankfully, we know Falco has the context of Kubernetes in the Falco rule detection. Wireshark with Falco bridges the gap between raw network data and the intelligence provided by the Kubernetes audit logs. We now have some associated metadata from the Falco alert for the network capture.
Finally, there’s the challenge of data overload associated with PCAP files. Traditional packet capture strategies, such as those employed by AWS VPC Traffic Mirroring or GCP Traffic Mirroring, often result in vast amounts of data, most of which is irrelevant to the actual security concern, making it harder to isolate important information quickly and efficiently. Comparatively, options like AWS VPC Flow Logs or Azure’s attempt at Virtual network tap, although less complex, still incur significant costs in data transfer/storage.
When’s the appropriate time to start a capture? How do you know when to end it? Should it be pre-filtered to reduce the file size, or should we capture everything and then filter out noise in the Wireshark GUI? We might have a solution to these concerns that bypasses the complexities and costs of cloud services.
The /555 Guide For Security Practitioners
Meet The Only Benchmark For Cloud Security!
Introducing a New Approach with Falco Talon
Organizations have long dealt with security blindspots related to Kubernetes alerts. Falco and Falco Talon address these shortcomings through a novel approach that integrates Falco, a cloud-native detection engine, with tshark, the terminal version of Wireshark, for more effective and targeted network traffic analysis in Kubernetes environments.
Falco Talon’s event-driven, API approach to threat response is the best way to deal with initiating captures in real time. It’s also the most stable approach we can see with the existing state-of-the-art in cloud-native security – notably, Falco.
Step-by-Step Workflow:
- Detection: Falco, designed specifically for cloud-native environments like Kubernetes, monitors the environment for suspicious activity and potential threats. It is finely tuned to understand Kubernetes context, making it adept at spotting Indicators of Compromise (IoCs). Let’s say, for example, it triggers a detection for specific anomalous network traffic to a Command and Control (C2) server or botnet endpoints.
- Automating Tshark: Upon detection of an IoC, Falco sends a webhook to the Falco Talon backend. Talon has many no-code response actions, but one of these actions allows users to trigger arbitrary scripts. This trigger can be context-aware from the metadata associated with the Falco alert, allowing for a tshark command to be automatically initiated with metadata context specific to the incident.
- Contextual Packet Capturing: Finally, a PCAP file is generated for a few seconds with more tailored context. In the event of a suspicious TCP traffic alert from Falco, we can filter a tshark command for just TCP activity. In the case of a suspicious botnet endpoint, let’s see all traffic to that botnet endpoint. Falco Talon, in each of these scenarios, initiates a tshark capture tailored to the exact network context of the alert. This means capturing traffic only from the relevant pod, service, or deployment implicated in the security alert.
- Improved Analysis: Finally, the captured data is immediately available for deeper analysis, providing security teams with the precise information needed to respond effectively to the incident. This is valuable for Digital Forensics & Incident Response (DFIR) efforts, but also in maintaining regulatory compliance by logging context specific to security incidents in production.
This targeted approach not only reduces the volume of captured data, making analysis faster and more efficient, but also ensures that captures are immediately relevant to the security incidents detected, enhancing response times and effectiveness.
Collaboration and Contribution
We believe this integrated approach marks a significant advancement in Kubernetes security management. If you are interested in contributing to this innovative project or have insights to share, feel free to contribute to the Github project today.
This method aligns with the needs of modern Kubernetes environments, leveraging the strengths of both Falco and Wireshark to provide a nuanced, powerful tool for network security. By adapting packet capture strategies to the specific demands of cloud-native architectures, we can significantly improve our ability to secure and manage dynamic containerized applications.
Open source software (OSS) is the only approach with the agility and broad reach to set up the conditions to meet modern security concerns, well-demonstrated by Wireshark over its 25 years of development. Sysdig believes that collaboration brings together expertise and scrutiny, and a broader range of use cases, which ultimately drives more secure software.
This proof-of-concept involves three OSS technologies (Falco, Falco Talon, and Wireshark). While the scenario was specific to Kubernetes, there is no reason why it cannot be adapted to standalone Linux systems, Information of Things (IoT) devices, and Edge computing in the future.