Elastic Compute Cloud (EC2) is arguably one of the most popular AWS services, and really needs no introduction but here is one anyway. With Sysdig, you can secure EC2 by detecting threats and vulnerabilities, controlling configuration and permission risks, and meeting compliance requirements.
When it comes to EC2 and Hosts themselves, Sysdig Secure alerts us in multiple ways:
- Vulnerabilities in OS and software packages on the EC2 Hosts
- Runtime threat detection on workloads running on EC2 Hosts
- Periodic CSPM/compliance checks of EC2
- Continuous checking of changes / security monitoring of EC2
Vulnerabilities on EC2 Hosts
When it comes to OS or software package vulnerabilities, these can manifest in containers or Hosts themselves. Hosts can refer to Nodes in a Kubernetes cluster, or can simply be EC2 hosts that run workloads without any container use cases.
It is important to understand these protections apply to container environments just as they apply to EC2 Hosts.
Sysdig Secure can check for vulnerabilities in Fargate as well as in the EC2 host.
No matter whether the vulnerabilities are in the OS (e.g. rpm, dpkg) or non-OS (e.g. Java packages or Ruby gems), Sysdig Secure makes it easy to see our EC2 hosts and the Scan status for each:
Drilling down onto one of these hosts, we can see first of all it is an Amazon Linux 2 based host.
We are given a list of Vulnerabilities, and to prioritize the most urgent vulnerabilities first, Sysdig allows us to choose to view only those that are Critical or High and that have a Fix available:
Focusing on the first Vulnerability, ALAS2-2021-1722 because this is in an Amazon Linux 2 host, Sysdig Secure is using the Amazon Security Tracker database for this vulnerability.
Sysdig Secure shows us details of the issue:
…and a link takes us to the official database for more information:
Runtime protection of Workloads on EC2 Hosts
Runtime security is the only thing standing between your EC2 Hosts and threats that have managed to elude the other defenses. By continuously auditing your runtime environments, you can minimize the potential impact of runtime threats.
Sysdig has rich runtime workload protection for containers, and Sysdig policies act at the host level the same way they do at the container level.
The same engine and rules and Falco syntax can detect malicious activities and threats at the host level, in this case, for our EC2 Hosts.
Here, we can do a test with the “Delete Bash History” out-of-the-box rule.
We add this rule to our Policy, and we scope our policy to apply only to Hosts.
Now a simple test on our EC2 Host…
And Voila, Sysdig shows us the Event …
And drilling down into the event we see all the Who/What/When/Where details we need, allowing us to take action. This is what runtime workload protection looks like for your EC2 hosts.
CSPM validation of our EC2 environment
Workload protection on our EC2 hosts is one thing. But our EC2 Hosts need to reside in a secure overall cloud environment. We have to ensure the AWS environment is secure.
Cloud Security Posture Management (CSPM) unifies the different use cases aimed to protect the entire AWS environment. CSPM audits and tracks cloud resources to evaluate the static configuration of the cloud against a framework.
Also, one of the main use cases of CSPM is to check that cloud settings are following best practices defined in Industry compliance frameworks.
CSPM can alert you to:
- Data storage exposed directly to the internet
- Lack of encryption on databases
- Lack of multi-factor authentication enabled on critical system accounts
Here we see a list of CSPM Compliance reports we have run in our AWS Environment:
Here we see the details of one our CSPM scans showing passing controls, and some failing controls we need to fix.
The framework used here was AWS WAF COMPLIANCE which is a custom framework.
Continuous detection of EC2 changes
CSPM provides a picture of your security posture at a single point in time. On the other hand, continuous detection based on watching CloudTrail gives us real-time alerting. Unified CSPM and Continuous Cloudtrail detection together give us maximum visibility.
Even if CSPM reports are all clear, if something changes, we need to be alerted as close to real-time as possible. Today, we will uncover how changes in your AWS EC2 configuration can create major security holes in real-time.
We will then hunt for these high-risk configuration events using native tools, and then we will compare this to how Sysdig Secure helps in these situations.
Which Amazon EC2 Configuration events create a Threat?
Sysdig has identified 20 risky configuration events, and this list is extensible.
Several of these events map to one or more well-known MITRE ATT&CK TTPs, https://attack.mitre.org/matrices/enterprise/cloud/iaas/ that are essential for cloud infrastructure threat mapping.
| Security Event | Impact | Description |
| Allocate New Elastic IP Address to AWS Account | High | Detects that a public IP address has been allocated to the account. |
| Run Instances with Non-standard Image | High | Detects launching of a specified number of instances with a non-standard image. |
| Run Instances in Non-approved Region | Medium | Detects launching of a specified number of instances in a non-approved region. |
| Run Instances | Medium | Detect launching of a specified number of instances. |
| Revoke Security Group Ingress | High | Detect removal of the specified ingress rules from aws security group |
| Revoke Security Group Egress | High | Detect removal of the specified egress rules from AWS EC2 securitygroup. |
| Replace Route | Medium | Detecting replacing an existing route within a route table in a VPC. |
| Modify Snapshot Attribute | Medium | Detects addition or removal of permission settings for the specified EC2 snapshot. |
| Modify Image Attribute | Medium | Detects modification of the specified attribute of the specified AMI. |
| Make EBS Snapshot Public | High | Detect making public an EBS snapshot. |
| Get Password Data | High | Detects retrieval of the encrypted administrator password for a running Windows instance. |
| EC2 Serial Console Access Enabled | High | Detects AWS EC2 serial Console Access enabled in the account for a specific region. |
| Disable EBS Encryption by Default | Medium | Detect disabling EBS encryption by default for an account in the current region. |
| Describe Instances | Medium | Detect description of the specified AWS EC2 instances or all EC2 instances. |
| Delete Subnet | High | Detects deletion of the specified subnet. |
| Delete Cluster | High | Detect deletion of the specified cluster |
| Create Snapshot | Medium | Detects creation of an EBS volume snapshot and stores it in Amazon S3. |
| Authorize AWS Security Group Ingress | High | Detects addition of the specified ingress rules to AWS EC2 securitygroup. |
| Authorize AWS Security Group Egress | High | Detects addition of the specified egress rules to AWS EC2 securitygroup. |
| Associate Elastic IP Address to AWS Network Interface | High | Detects that a public IP address has been associated with a network interface. |
We will focus on three risky moves from this list that you will probably agree are very interesting from a security perspective:
- Run Instances with Non-standard Image
- Authorize AWS Security Group Ingress
- Disable EBS Encryption by Default
Non-Standard AMIs are Risky
According to Amazon, an Amazon Machine Image (AMI) is a template that contains a software configuration (for example, an operating system, an application server, and applications). From an AMI, you launch an instance, which is a copy of the AMI running as a virtual server in the cloud.
In your AWS account there are probably several different types of AMIs in use, and you need to know what is in each one, including the OS and the software packages. These versions of OS and software may contain vulnerabilities and you need a run-time solution for that.
But just as importantly, the first step is awareness. In other words, you can’t secure what you can’t see, so you need to be alerted instantly when a custom AMI is instantiated. Only then can you triage the situation and take corrective action.
Sysdig finds all occurrences of this event for you with the built-in Run Instances with Non-standard Image rule.
Wide-Open ports in Amazon EC2, you say?
There are certain ports that arguably SHOULD be open to the internet, such as those used for Web Servers such as port 443.
But there are other ports that really should never be open like this, and if they are, it’s either a big mistake or an intrusion situation. We need an AWS EC2 Check Open Ports process.
The following table shows some ports which should / should not be open to the public:
| Port | Purpose | Should this be open to the Internet? |
| 80,8080,8008 | HTTP (cleartext) | Probably not, but unfortunately a lot of websites are cleartext |
| 443,4434,4443 | HTTPS (encrypted TLS) | Yes, probably should |
| 25,587 | SMTP | Maybe, although cloud providers may block this |
| 22,3389 | SSH and RDP | No! |
Out of these ports, let’s pick port 22, for SSH because there really is no good reason why port 22 should be open to the Internet, making this a good candidate for our example.
It's time to take your best guess: How many open instances of port 22 do you think exist right now in the AWS IP Space?
Let’s use Shodan public search to find out. Did you guess 3,000,000? If so, you indeed win the prize (a free trial ) because there are roughly 3 million instances listening on port 22 within the global AWS IP Space.
The distribution amongst Amazon major IP blocks (according to Shodan) is as follows:
source: Shodan public search for Port 22, by AWS Organization, as of 5/5/2022
The Scene of the Crime
Can you manually find all AWS security groups that are wide open to the world within your AWS account? Yes, and here’s a simplified example of using the AWS CLI for this:
The obvious downsides to this approach is it’s pretty manual and unsophisticated so let’s ramp it up a notch.
We found a fun project that finds exposed IP addresses in our AWS account, makes a Shodan API call to get a rating for them, and stores everything in a DynamoDB database.
To get started we just clone the repo. Running the Lambda, we get nice results as shown here in our AWS DynamoDB table:
Ignoring for a moment that every additional chunk of code (no matter how easily obtained) increases our management burden, do you see the fundamental problem here? If you guessed this suffers from the same TOCTOU problem, you are correct.
With this AWS Lambda, we only find out about public IP addresses when we run it. If someone in our AWS account makes an IP public after we run this, we won’t know until the next time we run.
And if they reverted their changes before we run the Lambda again, we will never know it happened.
Recreating the Crime Scene
So how do we create one of these wide-open AWS Security Group rules? Here we use the AWS CLI to create and revoke a AWS security group that allows all access from 0.0.0.0/0. It’s actually doing three things:
- Finding the InstanceID based on name tag
- Finding the corresponding EC2 SecurityGroup ID for this InstanceID
- Adding 0.0.0.0/32:22 to this AWS Security Group ID
In between these two AWS EC2 security group events, the attacker could have exploited an SSH session while the Amazon EC2 security group was wide open to the Internet on port 22.
Sysdig Secure finds all occurrences of new security ingress rules for you with the Authorize Security Group Ingress rule.
AWS EC2 EBS Encryption
Encryption is, in general, a good thing. AWS EC2 EBS encryption is also a good thing, and should be enabled by default.
According to AWS, encryption operations occur on the servers that host EC2 instances, ensuring the security of both data-at-rest and data-in-transit between an instance and its attached EBS storage.
While it’s possible some rare and old instance types do not support encryption, that should be the exception rather than the rule, and handled on a case-by-case basis.
So, as shown here, if somebody goes and disables the default EBS Encryption setting for the entire AWS account, that is also a big deal from a security perspective, and you very much need to be alerted on this event:
Sysdig finds all occurrences of this risk event with the Disable EBS Encryption by Default rule.
Digging for Risks with CloudTrail
AWS wouldn’t leave us without a way to track our changes, and the great audit log in the sky named CloudTrail keeps track of everything.
Even better, CloudTrail doesn’t suffer from the same time-of-use problem, because it sees events as they happen, and doesn’t depend on the frequency of some function.
Here we can see some juicy AWS EC2 Security Group events, with all the right fidelity:
CloudTrail can of course have many millions of events, and it’s likely almost all of these will be good events which have no security implications at all. But how do you know which are which? Here a StackOverflow member posts a question that illustrates how painful finding interesting events can be:
Besides being essentially a giant log in the sky, millions of lines in size, CloudTrail doesn’t send us any alerts, but even if it could, it has no concept of right and wrong – so the alerting would not make any sense.
So, to make this useful, you could
- Use a Tool that knows right from wrong, and can send Alerts
- Roll up your sleeves and take a DIY approach using AWS native tools.
But as we wrote in an earlier blog post, it will be very difficult to get what you need using native tools .
Which brings us to Sysdig Secure for Cloud.
CloudTrail, meet Sysdig Secure
Sysdig watches CloudTrail constantly, so the moment CloudTrail knows about something, Sysdig knows about it too.
But unlike CloudTrail, Sysdig knows the difference between right and wrong, Sysdig will Alert us with the right data at the right time, and most importantly Sysdig works right out-of-the-box.
To show the breadth of high-risk Amazon EC2 events that Sysdig finds by default, let’s look at a couple of examples.
First, as we saw above, a Top 10 Worst-Practice is opening up a Security Group to 0.0.0/0 (the world) as shown here:
So for fun, let’s find the Who/What/When/Where if of a couple of high-risk events:
- Creating a 0.0.0.0 Security Group
- Disabling EBS Encryption
How can you achieve this? Find out for yourself right now by clicking start in the micro-demo below:
Sysdig and CloudTrail – Better Together
AWS EC2 provides a wide surface for attacks, and within that surface, security groups rank near the top. AWS Security Group limits are a critical component of a safe and secure deployment.
We also looked at another awful configuration event: Turning off encryption for AWS EBS. Both of these are events you definitely want to know about!
CloudTrail sees everything, but unfortunately doesn’t know right from wrong, and sometimes has millions of entries. Somewhere in CloudTrail, high-risk events are hiding like buried treasure.
Sysdig mines CloudTrail for these diamonds in the rough, finds them, runs them through your Sysdig security policy, and instantly gives you the Who/What/When/Where of the situation.
Sysdig Secure makes it easy to see if holes start to appear in your Amazon EC2 security, and makes it easy to fix.
Although this blog focused exclusively on EC2, Sysdig includes out-of-the-box security for pretty much all the important AWS services.
And Sysdig CloudTrail rules use the same format as the Open-Source Falco project (which has been downloaded more than 30 million times) giving you the power to make your own custom rules.
Getting a Single View of Risk with Sysdig
With Sysdig Secure, you can detect and respond to threats in the cloud and workloads (hosts, containers, and Kubernetes), including EC2.
When it comes to EC2, like other areas of the cloud, Sysdig provides protection in several ways:
- Vulnerability detection in OS and software packages on the EC2 Hosts
- Runtime protection of workloads on EC2 Hosts
- Periodic CSPM checking of EC2
- Continuous checking of changes to EC2 Hosts
It is important to understand these protections apply to container environments too.
You can manage configuration and permissions risk, meet compliance requirements, and manage vulnerabilities on containers and host VMs.
Sysdig provides a single view of risk with deep visibility and rich context to find and fix issues faster.
But don’t take our word for it – if you have a few minutes, get started today at no cost and see the proof for yourself.