In the face of persistent data breaches and escalating cyber threats, organizations are compelled to prioritize cloud defense in depth. These measures are indispensable for protecting critical assets and upholding the integrity of cloud-based systems. By establishing a comprehensive security plan, organizations can effectively convey their commitment to security and lay a solid foundation for a resilient and secure cloud environment.
In this blog post, we will delve into the profound strength and versatility offered by open source, cloud-native tools, which play a pivotal role in mitigating the lateral movement of malware, like Kinsing, within Kubernetes. This form of malware poses a significant threat to databases operating in cloud environments. Although we focus on Kinsing as a recent attack pattern, the principles discussed here can be extended to other types of malware that target cloud-native applications.
What is Cloud Defense in Depth and Why Should it be in Every Cloud Security Plan?
To strengthen cloud security further, organizations must embrace the concept of defense-in-depth. Cloud defense in depth extends beyond the generic constraints of supply chain security and host/workload runtime security. It encompasses the proactive implementation of multiple layers of security controls throughout an organization’s cloud infrastructure.
The shift-left and shield-right methodologies have emerged as powerful practices that organizations can adopt to enhance cloud security. Shift-left emphasizes integrating security considerations early in the development process, enabling developers to identify and address vulnerabilities at their root. By incorporating security tools and practices such as static code analysis, vulnerability scanning, and secure coding guidelines, organizations can proactively eliminate potential risks before they propagate throughout the application.
On the other hand, Shield-Right focuses on implementing security controls and protections at runtime and in the operational phase of the application lifecycle. It ensures that robust security measures are in place to shield the application from attacks and malicious activities. Kubernetes, a popular container orchestration platform, plays a crucial role in the Shield-Right methodology. It enables organizations to secure their containerized applications by leveraging features such as Role-Based Access Controls (RBAC), Kubernetes Network Policies (KNP), and runtime monitoring through Falco.
How an attacker moves from a Database to Cloud:
Let’s discuss an attack scenario that justifies the need for end-to-end detections to secure cloud-native workloads. The incident involves the Kinsing malware, which exploits vulnerabilities in container images and when misconfigured, exposed PostgreSQL containers to breach Kubernetes clusters. Kinsing, a Linux malware with a history of targeting containerized environments for cryptomining, utilizes compromised server resources to generate illicit profits for the threat actors.
If you’re unfamiliar with Kinsing malware, we provide dedicated resources to help you understand these types of attacks. The operators behind Kinsing are notorious for exploiting well-known vulnerabilities like Log4Shell.
Their objective is to gain initial access to Linux servers, regardless of whether they are operating on-premises or in the cloud, by exploiting the two standard options. The third point outlines potential techniques for lateral movement towards the cloud environment.
- Mitigating risk for misconfigured PostgreSQL databases
2. Mitigating risk in vulnerable container images
3. Mitigating lateral movements to the cloud
Mitigating risk for misconfigured PostgreSQL databases
When exploiting image vulnerabilities, the threat actors hunt for remote code execution flaws that enable them to push their payloads. As noted in the previous diagram, there are several mitigation strategies that could be applied, such as vulnerability scanning for potentially vulnerable images, as well as hardening your network security – each of which we will discuss in the context of open source, cloud-native technologies.
Use known registries for container’s images
Using known registries for container images is crucial to avoid database compromise because it helps ensure the integrity and security of the images used in your environment. When pulling container images from trusted and reputable registries, you can have more confidence in the authenticity and quality of the images.
Known registries often have established security measures in place, such as image scanning, vulnerability detection, and access controls, which help mitigate the risk of deploying compromised or malicious images. By leveraging open source tools like Trivy, you can enforce the use of known registries and perform image scanning to identify vulnerabilities and security issues.
In the below example, it can scan a Docker image and enforce known registries:
trivy image --only-fixed-versions --clear-cache --exit-code 1 docker.io/postgresql:latest
Code language: Perl (perl)Alternatively, tools like Docker Content Trust (DCT) provide image signing and verification mechanisms to ensure the integrity and authenticity of container images. You can set up a policy to enforce the use of signed images from known registries. This can be achieved by creating a notary configuration file (notary-config.json) with the list of trusted repositories and their associated keys.
{
"trust_dir": "~/.docker/trust",
"remote_server": {
"url": "https://notary.example.com",
"root_ca": "/path/to/root-ca.crt"
},
"repositories": {
"docker.io/library": {
"default": {
"signing_keys": [
{
"key_id": "<your-key-id>",
"key_path": "~/.docker/trust/private/<keyname>.key"
}
]
...
Code language: Perl (perl)Harden network access to the server
Assuming your organization has failed to identify the misconfigured database server, or assuming the database is not patched in time before being pushed into a production environment that does not enforce “least privilege” networking controls, it’s important to be able to detect the payload deployment from a running database workload.
- rule: DB program spawned process
desc: >
a database-server related program spawned a new process other than itself.
This shouldn\'t occur and is a follow on from some SQL injection attacks.
condition: >
proc.pname in (db_server_binaries)
and spawned_process
and not proc.name in (db_server_binaries)
and not postgres_running_wal_e
and not user_known_db_spawned_processes
output: >
Database-related program spawned process other than itself (user=%user.name user_loginuid=%user.loginuid
program=%proc.cmdline pid=%proc.pid parent=%proc.pname container_id=%container.id image=%container.image.repository)
priority: NOTICE
tags: [host, container, process, database, mitre_execution, T1190]
Code language: Perl (perl)The provided Falco detection rule reveals that we can identify instances where the compromised PostgreSQL database spawns a process other than itself. This is a clear indication of compromise associated with the Kinsing malware or potential SQL injection attacks on databases.
Scan images for vulnerabilities
Scanning images for vulnerabilities in the CI/CD pipeline and ensuring their origin from known registries are two crucial practices that should not be overlooked. However, one often neglected aspect is the runtime scanning of in-use containers to identify vulnerabilities.
To assess the vulnerability status of your PostgreSQL database, the open source tool Anchore Engine is highly recommended. Anchore Engine offers extensive image scanning capabilities and vulnerability analysis specifically designed for containers during runtime, providing valuable insights into the security posture of your PostgreSQL database.
Pull the container image you want to scan using Docker:docker pull postgresql:latest
You can then scan the pulled image using Anchore Engine. The ‘add‘ action is used to add a container image to Anchore Engine for analysis. On the other hand, the ‘wait‘ action quite literally waits for the analysis of a specific image to complete. Finally, the ‘content‘ command retrieves the detailed information about the content of an image.
docker run -e ANCHORE_CLI_URL=http://<anchore-engine-host>:8228/v1 --rm anchore/anchore-cli image add postgresql:latest
docker run -e ANCHORE_CLI_URL=http://<anchore-engine-host>:8228/v1 --rm anchore/anchore-cli image wait postgresql:latest
docker run -e ANCHORE_CLI_URL=http://<anchore-engine-host>:8228/v1 --rm anchore/anchore-cli image content postgresql:latest
Code language: Perl (perl)Of course, you’ll need to replace <anchore-engine-host> with the hostname or IP address of your Anchore Engine instance. Once you have done this, Anchore Engine will analyze the image and provide a detailed vulnerability report, including information about any vulnerabilities found in the image’s packages and dependencies. This confirms if your running containerized database can be compromised by the Kinsing malware.
Patch on time
Having a robust patch management strategy for databases in Kubernetes remains crucial despite the rollout process for containers. While containers provide isolation and encapsulation, vulnerabilities can still exist within container images, including the database software. Therefore, it’s vital to regularly update and patch the databases to address security vulnerabilities and stay protected against potential exploits.
However, due to the dynamic and automated nature of container deployment in Kubernetes, relying solely on manual patching may not be sufficient. This is where a tool like Gatekeeper comes into play. By leveraging Gatekeeper, you can enforce policies that reject containers with failed Common Vulnerabilities and Exposures (CVE) scores, ensuring that only containers with acceptable security levels are deployed.
This proactive approach complements the patch management strategy, providing an additional layer of defense against potential security risks in containerized databases. To reject known CVEs at runtime using OPA Gatekeeper, you can define policies that check for specific vulnerabilities and enforce restrictions on the deployment of resources that have those vulnerabilities. These policies can be written using the Rego language, which is the policy language used by OPA.
package kubernetes.cve_rejection
deny[msg] {
input.kind == "Deployment"
input.apiVersion == "apps/v1"
input.metadata.labels.app == "postgresql"
input.spec.template.spec.containers[_].image == "vulnerable-image:latest"
msg = "Deployment of my-app with vulnerable image is not allowed."
}
Code language: Perl (perl)In this example, the policy checks if a deployment resource with the label app: postgresql that is using the image vulnerable-image:latest is being created. If such a deployment is detected, the policy triggers and rejects it with a corresponding error message.
Gatekeeper creates a ConstraintTemplate manifest that defines the policy and can be used to create Constraints that are applied to specific resources. By utilizing OPA Gatekeeper in this manner, you can enforce runtime rejection of known CVEs by defining and applying custom policies that match your specific vulnerability criteria.
Preventing Exploitation of Container Images
To prevent exploitation of container images, it is essential to implement key security measures. These include removing trust authentication, securing network access, removing default users, and enforcing tight RBAC controls. By taking these steps, you can enhance the security of your containerized databases and reduce the risk of unauthorized access and potential breaches.
Remove trust authentication
One of the most common misconfigurations the attackers leverage is the ‘trust authentication’ setting, which instructs PostgreSQL to assume that “anyone who can connect to the server is authorized to access the database.” Where possible, it’s strongly recommended to disable this setting.
An open source tool that can help enforce authentication settings and security policies in PostgreSQL is pgAudit. This tool provides detailed logging and monitoring capabilities for PostgreSQL, including the ability to log and analyze authentication attempts and database activity.
By configuring pgAudit, you can gain insights into authentication patterns and identify any unauthorized access attempts. However, Falco works to detect suspicious process activity from the Postgres database. By working together, they address the authentication behavior and the process behavior.
Harden the network access to the database
Another mistake is assigning an IP address range that is far too wide, including any IP address the attacker may be using to give them access to the server. This means that Kubernetes Network Policies, and network visibility in general, are heavily required for both the vulnerable image and the misconfigured PostgreSQL database.
Attacks start with scanning of a wide range of IP addresses, looking for an open port that matches the default port of specific, popular web applications like WordPress. The general best practice in these cases would be to minimize access to exposed containers by using IP allow lists and following least privilege principles.
By default, all pods within a Kubernetes cluster can communicate with each other without any restrictions. Kubernetes Network Policies help you isolate the microservice applications from each other to limit the blast radius and improve the overall security posture.
Thankfully, Kubernetes Network Policies allow users to generate “least-privilege” policies to protect your workloads. You need to understand what port and IP traffic you wish to allow for your PostgreSQL workload. That way, we only allow what we are expecting, regardless of whether the workload is compromised or not.
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
Metadata:
name: postgresql-policy
Spec:
Selector:
matchLabels:
app: postgresql
Ingress:
- action: Allow
protocol: tcp
Source:
selector: app=app1
Destination:
Ports:
- 5432
Egress:
- action: Allow
protocol: tcp
Destination:
selector: app=frontend
Source:
Ports:
- 5432
Code language: Perl (perl)
The above policy targets pods labeled with app: postgresql. The policy only allows ingress (incoming) traffic on port 5432 (the default port for PostgreSQL) from pods labeled with app: frontend. It also allows egress (outgoing) traffic to pods labeled with app: frontend on port 5432.
This network policy also assumes that you have already deployed and labeled your PostgreSQL and frontend pods accordingly. You will need to adjust the policy based on your deployment configuration.
Native Kubernetes Network Policies do not require any additional networking requirements other than the (Container Networking Interface) CNIs already supported. The example we provided was for Calico Network Policies. You can use either Calico, Cilium, or the default Network Policy implementation to achieve this security goal.
A second “Default-Deny” policy is required to ensure all traffic that wasn’t already allowed in the packet pipeline should be dropped. This is a global default deny rule for a cluster that excludes CoreDNS (UDP port 53) traffic from being blocked. If this is too broad, you can create a default-deny on a per network namespace-level.
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
Metadata:
name: deny-app-policy
Spec:
namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system"}
Types:
- Ingress
- Egress
Egress:
- action: Allow
protocol: UDP
Destination:
selector: 'k8s-app == "kube-dns"'
Ports:
- 53
Code language: Perl (perl)Network Policies certainly narrowed the blast radius of the attack, but they do not address the initial compromise. That’s where we need a defense-in-depth strategy powered by deep intrusion detection capabilities with Falco. If a packet is being dropped, we need to know why. Is it a suspicious network connection? IPTables won’t give us this kind of context on its own:
- rule: Outbound or Inbound Traffic not to Authorized Server Process and Port
desc: Detects traffic that is not to an authorized server process and port.
condition: >
inbound_outbound and
container and
container.image.repository in (allowed_image) and
not proc.name in (authorized_server_binary) and
not fd.sport in (authorized_server_port)
enabled: false
output: >
Network connection outside authorized port and binary
(command=%proc.cmdline pid=%proc.pid connection=%fd.name user=%user.name user_loginuid=%user.loginuid container_id=%container.id
image=%container.image.repository)
priority: WARNING
tags: [container, network, mitre_discovery, TA0011]
Code language: Perl (perl)Remove default users, and extensive permissions
Default users play a crucial role in enhancing the security of a PostgreSQL database. It is important to eliminate default users to minimize the risk of unauthorized access or potential security breaches. Default users often have broad permissions and known credentials, making them attractive targets for attackers.
However, it is equally, if not more, important to enforce granular RBAC controls in Kubernetes to limit the blast radius. By implementing RBAC, you can assign specific roles and permissions to individual users or service accounts, ensuring they have only the necessary privileges required to perform their tasks.
Another open source tool that could help enforce these granular RBAC controls is the Kubernetes RBAC Manager. It allows you to define and manage RBAC policies declaratively using custom resources.
apiVersion: rbacmanager.reactiveops.io/v1beta1
kind: RBACDefinition
Metadata:
name: database-access
Spec:
Roles:
- name: database-reader
Rules:
- apiGroups: ["postgres.databases.io"]
resources: ["database"]
verbs: ["get", "list"]
- name: database-writer
Rules:
- apiGroups: ["postgres.databases.io"]
resources: ["database"]
verbs: ["get", "list", "create", "update", "delete"]
roleBindings:
- name: read-access-binding
Subjects:
- kind: User
name: nigel
roleName: database-reader
- name: write-access-binding
Subjects:
- kind: User
name: daniel
roleName: database-writer
Code language: Perl (perl)RBAC is defined to create two roles:
- database-reader with read-only access
- database-writer with read and write access to the database resource
The RoleBindings then associate the roles with specific users (Nigel and Daniel in this case). With Kubernetes RBAC Manager, you can ensure that only authorized users have the necessary permissions within Kubernetes, limiting the blast radius and maintaining a more secure environment.
By directly capturing system call events from the host in real-time, the Falco agent enables prompt alerting. If your PostgreSQL database has fallen victim to the Kinsing malware, it is important to note that this malware primarily targets Linux-based systems and Docker containers. Typically, the objective of the Kinsing malware is to do harm within Kubernetes – not to expand into the cloud.
However, if your Kubernetes environment has been compromised, how can you prevent adversaries from advancing from cloud-native workloads into the cloud? This becomes particularly relevant when your Kubernetes cluster is a managed service in the cloud, such as Elastic Kubernetes Service (EKS) on AWS. These considerations are integral to an end-to-end security plan, as it emphasizes the need to secure not only the image pipeline and container runtime, but also the cloud services hosting your cloud-native workloads.
Preventing Lateral Movement to the Cloud
It’s worth noting that movement from a compromised PostgreSQL database to the cloud would involve leveraging additional techniques and exploiting vulnerabilities in the cloud infrastructure. Here’s a generalized scenario that an adversary could usually follow to gain access to the cloud account that hosts the Kubernetes clusters and PostgreSQL workload:
- The initial compromise has already been discussed.
The adversary has gained access to the PostgreSQL database through various means, such as exploiting vulnerabilities, weak passwords, or insecure configurations. - Now, the adversary needs to escalate privileges within the compromised database to gain broader access and control over the system.
This can involve exploiting privilege escalation vulnerabilities, but is usually achieved by leveraging weak database configurations. - Assuming they have successfully identified the weaknesses in the database configuration or exploited a known vulnerability, they can perform reconnaissance to gather information about the targeted cloud environment.
This includes identifying the cloud provider, understanding the network architecture, and mapping out potential entry points.
Remember, they are doing all of this on the host server that is hosting the PostgreSQL DB. Falco is therefore able to detect instances where the attacker is trying to search for private keys or sensitive credentials on those systems.
Detect attempts to access sensitive credentials in Kubernetes
In an attempt to steal private keys or passwords from a Kubernetes cluster, an adversary might utilize the grep command to search through various files, logs, or configuration data within the cluster. By leveraging regular expressions, they can identify patterns associated with private keys or passwords, extracting sensitive information that could grant them unauthorized access to the cluster’s resources and compromise the security of the entire environment.
- rule: Search Private Keys or Passwords
desc: Detects grep private keys or passwords activity.
condition: >
(spawned_process and
((grep_commands and private_key_or_password) or
(proc.name = "find" and (proc.args contains "id_rsa" or proc.args contains "id_dsa")))
)
output: >
Grep private keys or passwords activities found
(user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline pid=%proc.pid container_id=%container.id container_name=%container.name
image=%container.image.repository:%container.image.tag)
Priority: WARNING
tags: [host, container, process, filesystem, mitre_credential_access, T1552.001]
Code language: Perl (perl)Assuming you don’t have real-time detection capabilities for this sort of behavior, the adversary could exploit cloud infrastructure vulnerabilities undetected, such as weak access controls, exposed management interfaces, or unpatched software. Kinsing may attempt to exploit these weaknesses to gain unauthorized access to the cloud infrastructure. This further reinforces the need for real-time detections.
Extend detection capabilities to cloud services
Extending detection capabilities to the cloud is essential to enhance overall security in Kubernetes environments. By correlating exfiltration attempts in Kubernetes with suspicious activities in the cloud, such as unauthorized deletion of S3 bucket encryption, organizations can gain a comprehensive view of potential security incidents and detect sophisticated attack patterns.
- rule: Delete Bucket Encryption
desc: Detects the deletion of configurations used to encrypt bucket storage.
Condition:
ct.name="DeleteBucketEncryption" and not ct.error exists
Output:
A encryption configuration for a bucket has been deleted
(requesting user=%ct.user,
requesting IP=%ct.srcip,
AWS region=%ct.region,
bucket=%s3.bucket)
priority: CRITICAL
source: aws_cloudtrailCode language: Perl (perl)By extending detection capabilities to the cloud, organizations can establish a holistic, cloud defense in depth security approach that covers both Kubernetes and cloud environments, ensuring a stronger defense against emerging threats and reducing the likelihood of data exfiltration and unauthorized access.
Conclusion
To mitigate the risks associated with Kinsing malware attacks, organizations can adopt a comprehensive, open source approach that combines shift-left security practices and robust defensive measures. This involves implementing image scanning for vulnerabilities during the pipeline phase and continuously monitoring running containers for potential exploits.
It is crucial to acknowledge the potential for attacks originating in cloud-native, containerized workloads, such as PostgreSQL, to propagate within Kubernetes and potentially extend into the cloud. While attacker techniques may evolve over time, adhering to these best practices provides a solid foundation for maintaining a robust security plan.
By following these guidelines, organizations can have greater confidence in the effectiveness of their security measures. For further insights, the Sysdig webinar on the value of combining shift-left and shield-right methodologies can provide valuable information: https://go.sysdig.com/WebShiftCloudSecurityEMEA.html.