What's New in Sysdig - February 2023

What’s New in Sysdig is back again with the February 2023 edition! I am Michael Rudloff, an Enterprise Sales Engineer based in the United Kingdom, and I am very excited to update you with the latest feature releases from Sysdig.

This month, Sysdig Secure brings a couple of new features. We have added reports to Risk Spotlight – Risk Spotlight can show you which packages with vulnerabilities are currently in use in a running container across your whole Kubernetes environment. Also, new features in Sysdig Monitor include Cost Advisor, which provides predictable cost analysis and savings estimates for Kubernetes environments, and more!

Sysdig Monitor

Cost Advisor

Cost Advisor, the predictable cost analysis tool for Kubernetes, is available in preview. Cost Advisor features include:

Visibility into Kubernetes cost allocation by team and business unit.
Export detailed spending data to include in your chargeback model.
Easily identify areas in your Kubernetes environments that can be optimized.
Recommendations to reduce wasted resources by an average of 40%.

Cost advisor is currently supported only in AWS environments. Support for Azure & GCP is coming soon.

How customers are using this

Estimating the costs of the workloads running on Kubernetes clusters across cloud providers, and correlating these costs with your application performance metrics is a need for every company running its applications in the cloud.

Cloud billing reports don’t include the Kuberenetes context, making it impossible to group cost or break it down through different workloads.

Cost Advisor gives visibility into Kubernetes’ cost and helps automatically identify areas to reduce that cost, with cost advisor customers are reducing wasted spending by 40% on average.

Register interest for the Cost Advisor preview.

Advisor Enhancements

Advisor has been improved to provide you the ability to:

Navigate from a pod to Metrics Explorer without losing context.
Create a scoped alert directly from an Advisory.
View pod YAML, which is similar to kubectl get pod <pod> -o yaml. (Pod YAML requires agent 12.9.0 or newer)

Custom Webhook Notification Channel

You can now create a Custom Webhook Notification Channel and fully customize the HTTP payload of an alert notification forwarded to a third party webhook-based integration. Using Sysdig Templating Language, you can dynamically interpolate alert metadata such as alert name and severity as well as event context such as infrastructure labels and timestamp. This allows users to integrate with integrations beyond those natively supported by Sysdig Monitor.

Multi-Threshold Alerts

You can now configure an optional warning threshold for metrics and events. For more information, see Multi-Threshold Alerts.

Alert on No Data

When a metric stops reporting, Sysdig Monitor shows no data where you would normally expect data points. To detect such incidents that fail silently, you can configure alerts to notify you when a metric ceases to report data. For more information, see Alert on No Data.

Dashboards & Explore enhancements

Dashboards have been improved to provide you the ability to:

The PromQL Query Explorer module has received a refresh to offer an easier way to find, search and understand labels. Read more

Monitoring integrations

Added the following integrations:

Linux Host
GCP Memorystore for Redis
GCP Cloud SQL MySQL
GCP Cloud SQL PostgreSQL
GCP Cloud SQL SQLServer
GCP Compute Engine

Dashboards and Alerts

Updated the VM dashboards with new panels in the Windows and Linux dashboards
Added “Exporter Down” alert to detect offline exporters before you notice missing metrics
Added additional Windows alerts using Windows default metrics and process collector metrics
Added new Cloud Provider labels to help scoping for TS consumption and AWS dashboards
Added new Time Series (TS) dashboard (Agents and Jobs Time Series) for tracking TS consumption from Monitor Integrations
Refreshed Red Hat OpenShift (RHOS) dashboards and alerts

Sysdig Secure

VM Reports Now Include Risk Spotlight (In Use) and Accepted Risks

The Risk Spotlight feature (In Use) and the Accepted Risks are now available in the VM Reporting feature as both an additional metadata column and a configurable filter. Every matching vulnerability will have these two new additional columns, plus the matching true/false filters.

How customers are using this

Many Sysdig customers use Risk Spotlight for Runtime Scan on a daily basis to determine which vulnerabilities are ‘In-Use’, can be fixed and which are exploitable. Our ability to show which packages are ‘In-Use’ in running Kubernetes containers is a game changer in many environments. While you could use the reporting feature to determine these fixable and exploitable vulnerabilities and packages already, by adding the functionality of reporting the ‘In-Use’ packages gives developers an actionable report without having to sort through the results manually.

Cli-scanner 1.3.3 and Jenkins Plugin 2.2.7 Released

Sysdig has released version 1.3.3 version of the cli-scanner and 2.2.7 version of the Jenkins Plugin.

Scanner Fixes: Bug fixes, some of which were impacting policy evaluations.
Plugin Update:

Updates to the scanner
Adjustments to the string representation of some policy rules in the report section
Several bug fixes, including one that caused the build to fail when it shouldn’t

Non-Containerized Install Available for Host Scanning

While Helm is the recommended installation method, if you want to scan a host without using containers at all, we also offer a standalone binary and an RPM package.

Liveness and Readiness Probes Added to Helm Chart

Starting from sysdig-deploy Helm chart version 1.5.34, we have added livenessProbe and readinessProbe which checks for vulnerability runtime scanner component health, in agreement with the Kubernetes monitoring and scheduling practice.

Be aware, this requires having a vuln-runtime-scanner version >= v1.4.4″

New Filter and Grouping for Rules Library

This release enhances the Threat Detection rules library by showing the rules in a grouped manner as well as adding the ability to view only custom rules.

View Insights Grouped by User

The Insights visualization now permits viewing events grouped by user, greatly improving the ability to spot outliers. You can also see all events from a particular user in reverse chronological order. See the Insights documentation for details.

Sysdig Agents

Agent updates

The latest Sysdig Agent release is v12.11.0. Below is a diff of updates since v12.10.1, which we covered in our January update.

Feature enhancements

There are no notable feature enhancements in this Months’ release.

Defect fixes

Disable Containerd Events

You can configure Containerd events emission by using the events: >containerd: section in the YAML configuration.

Enhance Legacy Delegation

A fallback mechanism has been added to get the agent pod’s namespace. All the pods with label app: sysdig-agent and their namespace will be listed now.

Display Correct CPU Utilization for Linux Hosts

Monitor UI now shows correct CPU utilization for the Linux host.

Communicate with Kubernetes Clusters with IPV6 Addresses

The cointerface process continues to communicate with Kubernetes clusters with IPs that only have IPV6 addresses.

Fix Cointerface Process Failure

Fixed a problem in agent 1v2.10.x that could cause the cointerface process to fail when k8s_delegated_nodes was set to 0.

Make CRI Socket Path Searchable in EKS+Bottlerocket Environments

The CRI socket path used by EKS+Bottlerocket is added to the set of paths automatically searched by the agent.

Send Stale Makers for Failed Scrapes

Fixed an issue that could intermittently cause the agent to send invalid Prometheus values instead of stale markers for failed scrapes.

Agent Starts as Expected on Fedora

Fixed agent startup failure on cloud variants of Fedora v35+ when no kernel headers are available.

Agentless updates

End of Life

The local installer used to deploy the instrumentation stack is no longer supported.

New Features

Serverless Patcher

The Serverless Agent 4.0.0 provides serverless-patcher, a new containerized template patcher that can run locally and be integrated into CI/CD pipelines

CloudFormation template instrumentation.yaml

The Serverless Agent 4.0.0 provides instrumentation.yaml, a new CloudFormation template to deploy the automation to instrument (that is to patch) templates on Cloud.

SecretsManager support for the orchestrator agent

Secrets like the Access Key and the Proxy Password can be now automatically fetched and provided to the orchestrator agent at deployment time.

Custom CA Certificates support for the orchestrator agent

The orchestrator agent supports the uploading of custom CA certificates. That allows the SSL certificate verification of OnPrem backends and proxies.

Workload agent logs fine-tuning

Logs can be tuned and controlled at the fine-grained component level. This can avoid excessive logging from certain components, or enable extra logging from specific components for troubleshooting

Defect Fixes

Runtime instrumentation exits

The runtime instrumentation exits when the main process exits, thus avoiding waiting for another process to finish and keeping the container alive.

Renamed parameter in CloudFormation template orchestrator-agent.yaml

The Gateway parameter has been renamed to NetworkType in the CloudFormation template orchestrator-agent.yaml.

Exact image tags

The CloudFormation stacks use exact tags now, instead of latest.

Redundant wildcard permissions

Redundant wildcard permissions have been removed from the TaskRole of the orchestrator-agent.

SIGINT/SIGTERM propagation

The runtime instrumentation propagates SIGINT and SIGTERM signals to the instrumented workload now.

Honor log silent mode in the workload agent

The silent log mode prevents environment variables from being printed now.

List separator to OptIn/OptOut containers to be/from being instrumented

Colons are now required as list separators to OptIn/OptOut containers. Commas are no longer supported.

Example:

In the TaskDefinition, Tags can be leveraged to explicitly instrument some containers of the task, or prevent a bunch of them from being instrumented. For example, the following tag prevents myContainer1 and myContainer2 from being instrumented when the template patching runs in OptOut mode (default):

    Tags:
  - Name: "kilt-ignore-containers"
    Value: "myContainer1:myContainer2"
Code language: Perl (perl)

Vulnerabilities

Fixed the following vulnerabilities with the orchestrator agent:

CVE-2022-28948
CVE-2022-47629
CVE-2022-41721

Fixed the following vulnerabilities with the workload agent:

CVE-2022-47629

No known vulnerabilities with the serverless instrumentation.

SDK, CLI, and Tools

Sysdig CLI

v0.7.14 is still the latest release. The instructions on how to use the tool and the release notes from previous versions are available at the following link:

https://sysdiglabs.github.io/sysdig-platform-cli/

Python SDK

v0.16.4 is still the latest release.

https://github.com/sysdiglabs/sysdig-sdk-python/releases/tag/v0.16.4

Terraform Provider

There is a new release v0.5.48.

Documentation – https://registry.terraform.io/providers/sysdiglabs/sysdig/latest/docs

GitHub link – https://github.com/sysdiglabs/terraform-provider-sysdig/releases/tag/v0.5.48

Terraform modules

AWS Sysdig Secure for Cloud has been updated to v0.10.7.
GCP Sysdig Secure for Cloud remains unchanged at v0.9.6.
Azure Sysdig Secure for Cloud has been updated to v0.9.4.

Note: Please check release notes for potential breaking changes.

Falco VSCode Extension

v0.1.0 is still the latest release.

https://github.com/sysdiglabs/vscode-falco/releases/tag/v0.1.0

Sysdig Cloud Connector

AWS Sysdig Secure for Cloud has been updated to v0.16.28.

Admission Controller

Sysdig Admission Controller has been updated to v3.9.16.

Documentation – https://docs.sysdig.com/en/docs/installation/admission-controller-installation/

Runtime Vulnerability Scanner

The new vuln-runtime-scanner has been updated to v1.2.13.

Documentation – https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/runtime

Sysdig CLI Scanner

Sysdig CLI Scanner remains at v1.3.4.

Documentation – https://docs.sysdig.com/en/docs/sysdig-secure/vulnerabilities/pipeline/

Image Analyzer

Sysdig Node Image Analyzer has been updated to v0.1.24.

Host Analyzer

Sysdig Host Analyzer remains at v0.1.11.

Documentation – https://docs.sysdig.com/en/docs/installation/node-analyzer-multi-feature-installation/#node-analyzer-multi-feature-installation

Sysdig Secure Online Scan for Github Actions

The latest release is still v3.4.0.

https://github.com/marketplace/actions/sysdig-secure-inline-scan

Sysdig Secure Jenkins Plugin

Sysdig Secure Jenkins Plugin has been updated to v2.2.8.

https://plugins.jenkins.io/sysdig-secure/

Prometheus Integrations

The PromCat team officially released Prometheus Integrations v1.9.0!

Integrations:

Feat: Install and configure windows machine in AWS in order to configure windows exporter
Feat: New k8s-cAdvisor integration
Fix: Etcd default job with new ports and certificate path
Feat: PoC of Windows Installer with Windows Exporter and Prometheus Agent
Fix: Fix unclosed parenthesis in ntp autodiscovery filter.
Feat: GCP Memorystore for Redis integration.
Feat: GCP Cloud SQL MySQL integration.
Feat: GCP Cloud SQL PostgreSQL integration.
Feat: GCP Cloud SQL SQLServer integration.
Feat: GCP Compute Engine integration.

Sysdig on-premise

Last Hotfix available is 5.1.8 , list of fixes for that version are available here.

Falco rules changelog

Added the following rules:
- User Management Event Detected
- Users Group Management Event Detected
- OpenSSL File Read or Write
Reduce false positives for the following rules:
- Modify ld.so preload
- Clear Log Activities
- Read sensitive file untrusted
- Read Shell Configuration File
Improved condition for the following rules:
- Delete Bash History
- Delete or rename shell history
- Detect malicious cmdlines
Improve the sensitive_kernel_parameter_files list
Updated IoCs Ruleset with new findings.
Add an exception for the OpenSSL File Read or Write rule.

Our Falco team has been busy this month with multiple releases of new features. For more information on what has been released for the entire month of February, please review it here.

New Website Resources

Blogs

Threat research

Webinars

Tradeshows

April 17 – 21, KubeCon Europe 2023, Amsterdam, Netherlands
April 24 – 27, RSA Conference 2023, San Francisco CA

Education

The Sysdig Training team provides curated hands-on labs to learn and practice different topics. The selection of courses for the month of February:

Monitoring Integrations – Automatically instrument your workloads with Sysdig Monitor and Prometheus

What’s New in Sysdig – February 2023