Vulnerabilities are software bugs or weaknesses that could be used by an attacker. They could be present in the operating system, application code, and third-party code dependencies, such as libraries, frameworks, programming scripts, and so on. By taking a secure DevOps approach and identifying vulnerabilities early in development, you avoid frustrating developers with delays when an application is ready for production. So, preventing vulnerable workloads from entering production is paramount, but keep in mind that new vulnerabilities and exploits can be discovered for software already in production. Scanning for vulnerabilities must be done throughout the entire workload life cycle, including at runtime.
Discover how to identify vulnerabilities in container images and hosts, integrate vulnerability assessment and controls in your software development life cycle, and unify container and host vulnerability management.
Understanding vulnerabilities in containers
Modern cloud-native workloads run as microservices built on containers and deployed as Kubernetes clusters in the cloud. A vulnerability in a container can be exploited to grant attackers a foothold into your Kubernetes cluster, and from there, they can find ways to move laterally in your cloud environment.
According to the Sysdig 2022 Cloud-Native Security and Usage Report, 75 percent of images running in production contain patchable vulnerabilities of high or greater severity. The risk of exposing your organization through vulnerabilities present in containers is very real.
When securing containers it’s important to understand where vulnerabilities could be and how they could be introduced in your environment. Containers are built from read-only files called container images. Container images can be built totally from scratch or, more commonly, by using other available images, creating a layered structure. A base image (for example, Ubuntu, Debian, BusyBox, Alpine, and so on) provides the operating system (OS) dependencies and serves as a foundation for all other layers. The next layer could be a language or framework image, such as Ruby Gem, Python PiP, Node.js npm, and more. By using those readily available images, developers speed up development with a jumpstart. But, they could also be inadvertently introducing vulnerabilities into their containers.
Many developers aren’t aware that even official images stored in popular registries aren’t safe. Unfortunately, these repositories have been used as a channel to disseminate vulnerable software by malicious actors, like in supply chain attacks. For instance, DockerHub repository is known for having high-severity or critical vulnerabilities in a significant portion of its publicly available images. Some repositories offer vulnerability scanning; however, the scanning is often done just on OS images.
According to the Sysdig 2022 Cloud-Native Security and Usage Report, a high rate of high-severity or critical vulnerabilities exist in non-OS image layers. Both OS and non-OS images must be scanned.
Image scanning is a must-have in container environments. The application code and all image layers’ dependencies (OS and non-OS) in the container could have software vulnerabilities. As new vulnerabilities in existing software packages can be discovered at any time, scanning and auditing container images isn’t a one-time deal. The process requires continuous monitoring that starts with your code development and extends to production.
Integrating vulnerability scanning into DevOps life cycle
Integrating vulnerability scanning into the software development life cycle (SDLC), as for Infrastructure as Code (IaC) security, starts with a shift-left approach by embedding scanning early in the continuous integration and continuous delivery (CI/CD) process. Checking container images earlier ensures that development can run fast without costly fixes to the final build, and security isn’t a blocker for delivery.
Automate vulnerability scanning in the CI/CD pipeline
Integration with developers’ existing tools is key to successfully introducing security into DevOps processes. So, you should embed image scanning into the CI/CD pipeline to catch vulnerabilities (in all image layers, dependencies, and artifacts) early in the development life cycle. By doing so, you will also be saving engineering cycles required in more complex late fixes. Integration with CI/CD tools, such as GitHub, Google Cloud Build, and Amazon Web Services (AWS) Codebuild, can be done in two main forms, by pulling out the images for scanning or via inline scanning.
Inline scanning offers a more secure process. With inline scanning, your application code doesn’t leave your environment because you don’t need to send a copy to an intermediary repository for scanning. Only the metadata of the scan results are sent out for policy evaluation. Scan results are passed back to the pipeline, resulting in a pass or fail event.
Take a look into remediation cases that could be automated or facilitated, such as vulnerabilities with a fix requiring a low-risk version update of a package. And for those vulnerabilities without a fix or with a fix that would require more complex remediations, consider compensating controls with a risk assessment.
Automate registry image scanning
Keeping your private image repositories and registries free from vulnerable container images is another security best practice. Organizations commonly use registry services like DockerHub, Amazon ECR, Google Container Registry, Quay, and JFrog, among others. Even though these registries offer vulnerability scanning features, make sure that they’re comprehensive. Make sure that your registry covers each and every image layer and OS and non-OS dependencies. Most only offer OS image scanning, so your scanning solution should check for vulnerabilities in non-OS packages as well. Developers may be unknowingly pulling in vulnerabilities from non-OS open-source packages such as Python PIP, Ruby Gem, and more, which introduces security risks.
Protection against image tampering is also important. Make sure to digitally sign images and check signatures before using them. Verifying image signatures can assure that the specific image digest has been signed by the publisher.
As in the case of CI/CD pipeline integration, image scanning integration with registries can be done by pulling out the images to execute the scan or by scanning them inline. Organizations usually prefer inline scans that are done without pulling their images to a third-party repository for scanning.
Enforce vulnerability security with Kubernetes admission controller
Kubernetes admission controllers are the last line of defense before exposing the organization to container vulnerabilities in Kubernetes clusters. The admission controller evaluates requests to the Kubernetes application programming interface (API) server, then determines whether to allow the request. As an example, a request for a pod deployment could be denied if the admission controller receives a failed status on a check.
Admission controllers offer an extensible solution to enforce security policies using webhook integrations. Therefore, image scanners can be integrated with the admission controller to provide vulnerability check status prior to a container deployment in production. Kubernetes admission controllers also provide flexible rules to accept or reject requests based on the environment specifications. So, it is important that the vulnerability scanning solution also supports rich context within scan results so the environment context such as cloud, container, Kubernetes, and application criticality, among others, can be used to assess risk and define the appropriate policies.
Prevent vulnerabilities in serverless workloads
In serverless computing, cloud providers dynamically assign resources to run your workloads, but in a shared responsibility model, you’re responsible for protecting these workloads. Vulnerabilities in your code and dependencies can still provide attackers with a way to compromise your environment. You can and should implement vulnerability scanning on serverless workloads as well.
For instance, before launching a task on AWS Fargate — a serverless computing platform service to run containers as a service (CaaS) without the need to manage the underlying infrastructure — you could automatically trigger the scanning of the container image in the AWS ECR repository. And again, scanning can be implemented inline or by pulling out the image. Make sure that all layers are verified for vulnerabilities and that you can block deployment if high-risk ones are found.
Manage vulnerabilities at runtime
New vulnerabilities and exploits are discovered every day on new and old versions of software packages and OS distributions. If you’re only scanning at the build stage, you’re leaving yourself exposed to recently discovered vulnerabilities.
Managing vulnerabilities in your runtime environment is necessary. Your vulnerability management solution should keep track of images deployed in production and alert when an image is affected by new vulnerabilities in feed updates from vulnerability data sources. Security tools use different strategies to achieve this. One way is to rescan all images periodically, but that’s not an efficient use of your resources. Ideally, the image metadata of deployed containers is stored and new vulnerabilities can be detected without the need of a full rescan.
Let us just add this additional observation point: Alerting to vulnerabilities at runtime without context isn’t really effective. Environment information, such as cloud, Kubernetes, and application context, is necessary for risk assessment, remediation prioritization, and guidance regarding containment and compensating measures.
Implement host scanning
Hosts are the computing instances where your containers run. In Kubernetes environments, hosts are the nodes of a cluster where Pods are deployed, and an estimated quarter of organizations in the cloud have unpatched hosts with high severity and critical vulnerabilities. Although not as numerous as containers, vulnerable computing instances in public clouds expose the organization to serious risks.
Securing host virtual machines (VMs) is just as important as securing the containers running on them. Make sure that you have a tool in place that can scan hosts. For example, cloud VMs (computing instances such as EC2s) are hosts that need scanning. Host scans must also be done regularly, so your teams are given actionable information to prioritize and expedite remediation.
Adopting a single vulnerability management workflow for containers and hosts
Siloed solutions create security gaps and inefficiencies. Adopting a unified approach to vulnerability management for containers and hosts speeds the time to detect and remediate vulnerabilities, while generating fewer alerts. Organizations are often faced with thousands of vulnerabilities detected in their environments while their teams can only timely handle a small fraction of them. Prioritization and efficient remediation become fundamental to vulnerability management.
Risk comes in different forms. Not only do vulnerabilities in the code expose your environment to threats, but also it’s important to verify if security best practices are followed as well as all compliance standards and regulations that your organization adheres to.
Implement vulnerability prioritization
How to prioritize which vulnerabilities to fix is a constant challenge for resource-constrained security teams. Evaluating the severity of a vulnerability may only lead to cycles spent on vulnerabilities that don’t represent practical risk. When you’re doing prioritization, the crucial decision to make is about what can’t wait. So make sure to also include in your analysis contextual risk that reveals the true relevance of a critical vulnerability to your environments.
Think about the following questions:
- Is the vulnerability located in packages loaded when the container is running, and therefore with real chances of being exploited, or in a package that isn’t used?
- Is the vulnerability exposing critical applications?
- Is the vulnerability present in environments that aren’t externally accessible?
- Are there available exploits being used, and if so, what’s the reported threat activity?
As you answer these questions, it will become clear which vulnerabilities in your environment context are truly exploitable and incur high risk to the organization. Those areas are the ones where you need to focus your remediation efforts.
Managing vulnerabilities requires understanding how to effectively reduce risk in your environment. Make sure that your vulnerability scanning solution provides detection of vulnerabilities on containers and hosts. Check the quality of the threat data and context provided with the vulnerability alert for prioritization and remediation, tracking improvements in reducing exposure risk.
Check for compliance requirements
Unified host and container scans can be used beyond vulnerabilities checks. The scans can also validate regulatory compliance (for example, PCI DSS, NIST regulations, SOC 2) and security best practices, such as the Center for Internet Security (CIS) Benchmarks. Scanners should detect security risks, such as port misconfigurations, unprotected secrets, open-source software (OSS) licenses, and file integrity, among security controls.
Want to learn more about cloud security?
Check out our “Securing Containers & Cloud” ebook, and discover topics like Infrastructure as Code (IaC), responding to threats, keeping your containers and cloud compliant, and more!