What’s New in Sysdig is back again with the February 2023 edition! I am Michael Rudloff, an Enterprise Sales Engineer based in the United Kingdom, and I am very excited to update you with the latest feature releases from Sysdig.
This month, Sysdig Secure brings a couple of new features. We have added reports to Risk Spotlight – Risk Spotlight can show you which packages with vulnerabilities are currently in use in a running container across your whole Kubernetes environment. Also, new features in Sysdig Monitor include Cost Advisor, which provides predictable cost analysis and savings estimates for Kubernetes environments, and more!
Cost Advisor, the predictable cost analysis tool for Kubernetes, is available in preview. Cost Advisor features include:
- Visibility into Kubernetes cost allocation by team and business unit.
- Export detailed spending data to include in your chargeback model.
- Easily identify areas in your Kubernetes environments that can be optimized.
- Recommendations to reduce wasted resources by an average of 40%.
Cost advisor is currently supported only in AWS environments. Support for Azure & GCP is coming soon.
How customers are using this
Estimating the costs of the workloads running on Kubernetes clusters across cloud providers, and correlating these costs with your application performance metrics is a need for every company running its applications in the cloud.
Cloud billing reports don’t include the Kuberenetes context, making it impossible to group cost or break it down through different workloads.
Cost Advisor gives visibility into Kubernetes’ cost and helps automatically identify areas to reduce that cost, with cost advisor customers are reducing wasted spending by 40% on average.
Advisor has been improved to provide you the ability to:
- Navigate from a pod to Metrics Explorer without losing context.
- Create a scoped alert directly from an Advisory.
- View pod YAML, which is similar to kubectl get pod <pod> -o yaml. (Pod YAML requires agent 12.9.0 or newer)
You can now create a Custom Webhook Notification Channel and fully customize the HTTP payload of an alert notification forwarded to a third party webhook-based integration. Using Sysdig Templating Language, you can dynamically interpolate alert metadata such as alert name and severity as well as event context such as infrastructure labels and timestamp. This allows users to integrate with integrations beyond those natively supported by Sysdig Monitor.
You can now configure an optional warning threshold for metrics and events. For more information, see Multi-Threshold Alerts.
Alert on No Data
When a metric stops reporting, Sysdig Monitor shows no data where you would normally expect data points. To detect such incidents that fail silently, you can configure alerts to notify you when a metric ceases to report data. For more information, see Alert on No Data.
Dashboards & Explore enhancements
Dashboards have been improved to provide you the ability to:
- The PromQL Query Explorer module has received a refresh to offer an easier way to find, search and understand labels. Read more
Added the following integrations:
- Linux Host
- GCP Memorystore for Redis
- GCP Cloud SQL MySQL
- GCP Cloud SQL PostgreSQL
- GCP Cloud SQL SQLServer
- GCP Compute Engine
Dashboards and Alerts
- Updated the VM dashboards with new panels in the Windows and Linux dashboards
- Added “Exporter Down” alert to detect offline exporters before you notice missing metrics
- Added additional Windows alerts using Windows default metrics and process collector metrics
- Added new Cloud Provider labels to help scoping for TS consumption and AWS dashboards
- Added new Time Series (TS) dashboard (
Agents and Jobs Time Series) for tracking TS consumption from Monitor Integrations
- Refreshed Red Hat OpenShift (RHOS) dashboards and alerts
VM Reports Now Include Risk Spotlight (In Use) and Accepted Risks
The Risk Spotlight feature (In Use) and the Accepted Risks are now available in the VM Reporting feature as both an additional metadata column and a configurable filter. Every matching vulnerability will have these two new additional columns, plus the matching true/false filters.
How customers are using this
Many Sysdig customers use Risk Spotlight for Runtime Scan on a daily basis to determine which vulnerabilities are ‘In-Use’, can be fixed and which are exploitable. Our ability to show which packages are ‘In-Use’ in running Kubernetes containers is a game changer in many environments. While you could use the reporting feature to determine these fixable and exploitable vulnerabilities and packages already, by adding the functionality of reporting the ‘In-Use’ packages gives developers an actionable report without having to sort through the results manually.
Cli-scanner 1.3.3 and Jenkins Plugin 2.2.7 Released
Scanner Fixes: Bug fixes, some of which were impacting policy evaluations.
- Updates to the scanner
- Adjustments to the string representation of some policy rules in the report section
- Several bug fixes, including one that caused the build to fail when it shouldn’t
Non-Containerized Install Available for Host Scanning
While Helm is the recommended installation method, if you want to scan a host without using containers at all, we also offer a standalone binary and an RPM package.
Liveness and Readiness Probes Added to Helm Chart
Starting from sysdig-deploy Helm chart version
1.5.34, we have added livenessProbe and readinessProbe which checks for vulnerability runtime scanner component health, in agreement with the Kubernetes monitoring and scheduling practice.
Be aware, this requires having a vuln-runtime-scanner version >= v1.4.4″
New Filter and Grouping for Rules Library
This release enhances the Threat Detection rules library by showing the rules in a grouped manner as well as adding the ability to view only custom rules.
View Insights Grouped by User
The Insights visualization now permits viewing events grouped by user, greatly improving the ability to spot outliers. You can also see all events from a particular user in reverse chronological order. See the Insights documentation for details.
The latest Sysdig Agent release is v12.11.0. Below is a diff of updates since v12.10.1, which we covered in our January update.
There are no notable feature enhancements in this Months’ release.
Disable Containerd Events
You can configure Containerd events emission by using the events: >containerd: section in the YAML configuration.
Enhance Legacy Delegation
A fallback mechanism has been added to get the agent pod’s namespace. All the pods with label app: sysdig-agent and their namespace will be listed now.
Monitor UI now shows correct CPU utilization for the Linux host.
Communicate with Kubernetes Clusters with IPV6 Addresses
The cointerface process continues to communicate with Kubernetes clusters with IPs that only have IPV6 addresses.
Fix Cointerface Process Failure
Fixed a problem in agent 1v2.10.x that could cause the cointerface process to fail when k8s_delegated_nodes was set to 0.
Make CRI Socket Path Searchable in EKS+Bottlerocket Environments
The CRI socket path used by EKS+Bottlerocket is added to the set of paths automatically searched by the agent.
Send Stale Makers for Failed Scrapes
Fixed an issue that could intermittently cause the agent to send invalid Prometheus values instead of stale markers for failed scrapes.
Agent Starts as Expected on Fedora
Fixed agent startup failure on cloud variants of Fedora v35+ when no kernel headers are available.
End of Life
The local installer used to deploy the instrumentation stack is no longer supported.
The Serverless Agent 4.0.0 provides serverless-patcher, a new containerized template patcher that can run locally and be integrated into CI/CD pipelines
CloudFormation template instrumentation.yaml
The Serverless Agent 4.0.0 provides instrumentation.yaml, a new CloudFormation template to deploy the automation to instrument (that is to patch) templates on Cloud.
SecretsManager support for the orchestrator agent
Secrets like the Access Key and the Proxy Password can be now automatically fetched and provided to the orchestrator agent at deployment time.
Custom CA Certificates support for the orchestrator agent
The orchestrator agent supports the uploading of custom CA certificates. That allows the SSL certificate verification of OnPrem backends and proxies.
Workload agent logs fine-tuning
Logs can be tuned and controlled at the fine-grained component level. This can avoid excessive logging from certain components, or enable extra logging from specific components for troubleshooting
Runtime instrumentation exits
The runtime instrumentation exits when the main process exits, thus avoiding waiting for another process to finish and keeping the container alive.
Renamed parameter in CloudFormation template orchestrator-agent.yaml
The Gateway parameter has been renamed to NetworkType in the CloudFormation template orchestrator-agent.yaml.
Exact image tags
The CloudFormation stacks use exact tags now, instead of latest.
Redundant wildcard permissions
Redundant wildcard permissions have been removed from the TaskRole of the orchestrator-agent.
The runtime instrumentation propagates SIGINT and SIGTERM signals to the instrumented workload now.
Honor log silent mode in the workload agent
The silent log mode prevents environment variables from being printed now.
List separator to OptIn/OptOut containers to be/from being instrumented
Colons are now required as list separators to OptIn/OptOut containers. Commas are no longer supported.
In the TaskDefinition, Tags can be leveraged to explicitly instrument some containers of the task, or prevent a bunch of them from being instrumented. For example, the following tag prevents myContainer1 and myContainer2 from being instrumented when the template patching runs in OptOut mode (default):
Tags: - Name: "kilt-ignore-containers" Value: "myContainer1:myContainer2"Code language: Perl (perl)
Fixed the following vulnerabilities with the orchestrator agent:
Fixed the following vulnerabilities with the workload agent:
No known vulnerabilities with the serverless instrumentation.
SDK, CLI, and Tools
v0.7.14 is still the latest release. The instructions on how to use the tool and the release notes from previous versions are available at the following link:
v0.16.4 is still the latest release.
There is a new release v0.5.48.
- AWS Sysdig Secure for Cloud has been updated to v0.10.7.
- GCP Sysdig Secure for Cloud remains unchanged at v0.9.6.
- Azure Sysdig Secure for Cloud has been updated to v0.9.4.
Note: Please check release notes for potential breaking changes.
Falco VSCode Extension
v0.1.0 is still the latest release.
Sysdig Cloud Connector
AWS Sysdig Secure for Cloud has been updated to v0.16.28.
Sysdig Admission Controller has been updated to v3.9.16.
Runtime Vulnerability Scanner
The new vuln-runtime-scanner has been updated to v1.2.13.
Sysdig CLI Scanner
Sysdig CLI Scanner remains at v1.3.4.
Sysdig Node Image Analyzer has been updated to v0.1.24.
Sysdig Host Analyzer remains at v0.1.11.
Sysdig Secure Online Scan for Github Actions
The latest release is still v3.4.0.
Sysdig Secure Jenkins Plugin
Sysdig Secure Jenkins Plugin has been updated to v2.2.8.
The PromCat team officially released Prometheus Integrations v1.9.0!
- Feat: Install and configure windows machine in AWS in order to configure windows exporter
- Feat: New k8s-cAdvisor integration
- Fix: Etcd default job with new ports and certificate path
- Feat: PoC of Windows Installer with Windows Exporter and Prometheus Agent
- Fix: Fix unclosed parenthesis in ntp autodiscovery filter.
- Feat: GCP Memorystore for Redis integration.
- Feat: GCP Cloud SQL MySQL integration.
- Feat: GCP Cloud SQL PostgreSQL integration.
- Feat: GCP Cloud SQL SQLServer integration.
- Feat: GCP Compute Engine integration.
Last Hotfix available is 5.1.8 , list of fixes for that version are available here.
Falco rules changelog
- Added the following rules:
User Management Event Detected
Users Group Management Event Detected
OpenSSL File Read or Write
- Reduce false positives for the following rules:
Modify ld.so preload
Clear Log Activities
Read sensitive file untrusted
Read Shell Configuration File
- Improved condition for the following rules:
Delete Bash History
Delete or rename shell history
Detect malicious cmdlines
- Improve the
- Updated IoCs Ruleset with new findings.
- Add an exception for the
OpenSSL File Read or Writerule.
Our Falco team has been busy this month with multiple releases of new features. For more information on what has been released for the entire month of February, please review it here.
New Website Resources
- Kubernetes OOM and CPU Throttling
- End to End Vulnerability Scanning with Sysdig Secure
- How to Monitor CoreDNS
- Sysdig 2023 Cloud-Native Security and Usage Report
- Unveiling Sysdig’s new custom webhook
- Prometheus Alertmanager Best Practices
- GitHub & Supply Chain Risks
- Vulnerability Prioritization – Combating Developer Fatigue
- The Cloud Monitoring Journey
- How to Monitor Kubernetes Control Plane with Sysdig
- 8220 Gang Continues to Evolve With Each New Campaign
- How to Prevent a DDoS Attack in the Cloud
- Dig Deeper! Prioritize Cloud Vulnerabilities and Reduce Container Spending
- Crack the Exam Code to Become a Certified K8s Security Specialist in 2023
- April 17 – 21, KubeCon Europe 2023, Amsterdam, Netherlands
- April 24 – 27, RSA Conference 2023, San Francisco CA
The Sysdig Training team provides curated hands-on labs to learn and practice different topics. The selection of courses for the month of February:
- Monitoring Integrations – Automatically instrument your workloads with Sysdig Monitor and Prometheus