Today, AWS announced the general availability of Amazon ECS Exec, a powerful feature to allow developers to run commands inside their ECS containers.
Amazon Elastic Container Service (ECS) is a fully managed container orchestration service by Amazon Web Services. ECS allows you to organize and operate container resources on the AWS cloud, and allows you to mix Amazon EC2 and AWS Fargate workloads for high scalability. With ECS, developers can use tools that they know, like using a docker-compose definition to orchestrate containers and get started quickly with a cloud-based container cluster.
The new ECS Exec feature brings to ECS an exec command-like functionality to provide shell access to a running container. This is a capability that many developers already use with platforms like Docker and Kubernetes to perform container setup and perform debug tasks.
In this post, we’ll describe use cases this new ECS Exec feature covers, how to audit its usage with AWS CloudTrail, and how to apply your security policies to the executed commands with Sysdig Secure.
The exec command in Docker and Kubernetes
Both Kubernetes have an exec command that allows you to execute shell commands in a running container. By running an interactive shell inside a container, developers can perform operations in the container in any way they want.
One typical use case for using an interactive shell, for instance, is debugging why a container is not performing correctly. This capability is also commonly used to perform tasks like changing environment variables or modifying config files.
docker -it exec container-name /bin/sh kubectl exec --stdin --tty pod-name -- /bin/bash
Enter Amazon ECS Exec
To support this capability for its customers, today, AWS announced the general availability of the ECS Exec feature.
You can check out how to enable this feature for ECS and Fargate on the AWS site.
Once enabled, a developer can run the following command to start an interactive shell within a container:
aws ecs execute-command \ --region $REGION \ --cluster $CLUSTER_NAME \ --task $TASKID \ --container $CONTAINER_NAME \ --command "/bin/sh" \ --interactive
With this new feature, cloud teams have a powerful debugging capability and can also perform management tasks as they are used to do with their local tools.
However, the ability to exec into containers does open a potential way to break into a running container in production. Given that the capability can have both helpful and potentially harmful results, it’s important to detect and monitor its use.
The security implications of exec
In production, directly executing a shell in a container is often considered a risky practice from a security perspective. To guard against this, AWS provides fine-grained IAM policies for the ECS Exec feature that help ensure its use is controlled and authorized.
Let’s see now how AWS enables you to audit the use of this tool, and how it integrates with your other security tools, like Sysdig.
Auditing ECS Exec commands with AWS CloudTrail
Amazon provides a global auditing tool for all your cloud services called AWS CloudTrail. CloudTrail will register all the exec commands that run in your containers.
We can leverage CloudTrail for runtime analysis of events to be aware when exec is used so we can investigate it.
Whenever ECS Exec is used, a CloudTrail event is recorded with a JSON structure like this:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "$PRINCIPAL_ID",
"arn": "arn:aws:iam::$USER_ID:user/$USER_NAME",
"accountId": "$USER_ID",
"accessKeyId": "$ACCESS_KEY_ID",
"userName": "$USER_NAME"
},
"eventTime": "2021-02-03T17:00:14Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "ExecuteCommand",
"awsRegion": "ap-southeast-1",
"sourceIPAddress": "$SOURCE_IP_ADDRESS",
"userAgent": "aws-cli/1.18.222 Python/2.7.16 Darwin/20.2.0 botocore/1.19.62",
"requestParameters": {
"cluster": "$CLUSTER_NAME",
"container": "$CONTAINER_NAME",
"command": "/bin/sh",
"interactive": true,
"task": "arn:aws:ecs:ap-southeast-1:$USER_ID:task/$CLUSTER_NAME/$TASK_ID"
},
"responseElements": {
"clusterArn": "arn:aws:ecs:ap-southeast-1:$USER_ID:cluster/$CLUSTER_NAME",
"containerArn": "arn:aws:ecs:ap-southeast-1:$USER_ID:container/20486b49-de19-4f3f-9569-4748a0cb6fef",
"containerName": "$CONTAINER_NAME",
"interactive": true,
"session": {
"sessionId": "ecs-execute-command-062049f67691b9fa8",
"streamUrl": "wss://ssmmessages.ap-southeast-1.amazonaws.com/v1/data-channel/ecs-execute-command-062049f67691b9fa8?role=publish_subscribe",
"tokenValue": "HIDDEN_DUE_TO_SECURITY_REASONS"
},
"taskArn": "arn:aws:ecs:ap-southeast-1:845151661675:task/fargate-test/637d524791b7457b9e6598414404162a"
},
"requestID": "1742d166-4d53-4012-8dc6-def771383b5b",
"eventID": "a8592389-8e99-4e01-8866-48089aec4f9c",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": "$USER_ID"
}
You should monitor these events to check for any unauthorized access to containers in production.
Detecting ECS exec with a Falco rule
Using the Sysdig Cloud Connector, you can filter CloudTrail events so you can focus on what’s important.
The Sysdig Cloud Connector is based in Falco, the open-source cloud-native runtime security project, and contains a collection of out-of-the-box rules to get you started quickly. Falco rules like “Terminal shell in container” and “Attach/Exec Pod” are crafted specifically to detect exec activity.
This is the code for a Falco rule to detect the execution of a terminal shell inside an ECS container:
- list: system_shells
items: [/bin/bash, /bin/csh, /bin/ksh, /bin/sh, /bin/tcsh, /bin/zsh]
- rule: Terminal Shell in ECS Container
desc: A terminal shell has been executed inside an ECS container.
condition:
jevt.value[/eventSource]="ecs.amazonaws.com" and
jevt.value[/eventName]="ExecuteCommand" and
jevt.value[/requestParameters/command] in (system_shells) and
jevt.value[/requestParameters/interactive]=true and
not jevt.value[/errorCode] exists
output:
An interactive terminal shell has been executed inside an ECS container
(requesting user=%jevt.value[/userIdentity/arn],
requesting IP=%jevt.value[/sourceIPAddress],
AWS region=%jevt.value[/awsRegion],
cluster=%jevt.value[/requestParameters/cluster],
container=%jevt.value[/requestParameters/container],
command=%jevt.value[/requestParameters/command],
task=%jevt.value[/requestParameters/task])
priority: WARNING
tags:
- cloud
- aws
- aws_ecs
- aws_fargate
- aws_ecs_exec
source: aws_cloudtrail
This “Terminal Shell in ECS Container” rule is included out of the box with the Cloud Connector.
Whenever this rule is triggered, a new finding with details of the activity will be sent to AWS Security Hub.
To detect other kinds of suspicious activity, like launching an interactive command or even detecting any command execution inside an ECS container, the condition can easily be changed like in the following examples:
- rule: Execute Interactive Command inside an ECS Container
desc: Detect execution of an interactive command inside an ECS container.
condition:
jevt.value[/eventSource]="ecs.amazonaws.com" and
jevt.value[/eventName]="ExecuteCommand" and
not jevt.value[/requestParameters/command] in (system_shells) and
jevt.value[/requestParameters/interactive]=true and
not jevt.value[/errorCode] exists
(...)
- rule: Execute Command inside an ECS Container
desc: Detect execution of a command inside an ECS container.
condition:
jevt.value[/eventSource]="ecs.amazonaws.com" and
jevt.value[/eventName]="ExecuteCommand" and
jevt.value[/requestParameters/interactive]=false and
not jevt.value[/errorCode] exists
(...)
To learn more about writing Falco rules for Sysdig Cloud Connector, you can visit this official AWS hands-on workshop for Secure DevOps.
Detecting ECS Exec with Sysdig Secure
The Sysdig Cloud Connector can also send those events to Sysdig Secure, Sysdig’s container and Kubernetes security solution.
Sysdig Secure features an easy-to-navigate user interface to simplify operating at cloud scale and extends what Falco has to offer. Using Sysdig secure you can detect ECS Exec activity to ensure proper usage.
Now completely integrated with the events feed, whenever a CloudTrail event triggers a Falco rule, you can see these events integrated into your Sysdig Secure dashboard as a single source of truth.
All cloud security rules, including detecting ECS Exec on ECS and Fargate, are available in the rules library. This lets you easily create policies and configure notification channels to direct alerts regarding ECS Exec usage to the right teams or contacts within your organization for review.
You can use the Sysdig Secure dashboard to easily customize any available rules to better tailor detection to your own needs.
The Falco rules for CloudTrail have been created taking into account standards and recommendations like NIST 800-53, SOC2, PCI DSS, MITRE ATT&CK®, AWS CIS, and AWS Foundational Security Best Practices standard.
All rules have tags to identify the standard and sections they apply to, so you can filter them and create your own policies.
Conclusion
The new ECS Exec feature from AWS provides a powerful tool for cloud teams to debug and interact with their ECS containers running on EC2 and Fargate.
If you’re using ECS and Fargate, check out this new capability, but be sure to put in place the tooling needed for visibility into use.
Solutions like Falco and Sysdig Secure can help you ensure the AWS Exec has a proper and authorized use to ensure the security of your running production workloads.
If you do not have a Sysdig account, you can register today for a trial. Deploy Sysdig Cloud Connector in less than 10 minutes by following the instructions in this article, and start having a secured AWS account today.
You can also find Sysdig on the AWS Marketplace.