Sysdig offers remote write and more custom metrics for its managed Prometheus service
Content
We are excited to announce that Prometheus Remote Write functionality is now generally available in Sysdig Monitor. This feature allows Prometheus users to easily push metrics directly from their Prometheus servers to Sysdig’s Managed Prometheus Service.
Sysdig Monitor provides not only a scalable long term storage solution for custom metrics but also radically simplifies Prometheus monitoring. Sysdig untangles complex querying with our easy-to-use form-based query interface and our ability to extend PromQL eliminating most of the joins. Monitoring cloud-native services and applications is done right using Prometheus-based monitoring integrations with autodetection, assisted configuration, and out-of-the-box dashboards and alerts.
Sysdig customers avoid the need to manually plumb together all the pieces of their Prometheus monitoring stack and eliminate most of the operational work leveraging our SaaS service.
Now, metrics can be received through Prometheus Remote Write in parallel to Sysdig agents, which also support Prometheus Service Discovery mechanism and
prometheus.yaml
job configurations.
When do you need Prometheus remote write?
Long term storage service for Prometheus
Leveraging an external metrics storage allows you to maintain your existing Prometheus setup while reducing operational overhead and with the benefits of having all your metrics from different environments under a centralized single pane of glass. You can leverage all the things you love about Prometheus …- PromQL to query metrics
- Connect your Grafana dashboards to Sysdig endpoint like if it was a Prometheus server
- Dispatch PromQL alerts to your notification channels
Monitor environments without installing agents
Prometheus remote write allows you to monitor Windows hosts, non-x86 based architectures like ARM or Power, IoT or edge computing devices, etc; any environment where you wouldn’t normally install the Sysdig agent. With this functionality, you can centralize metrics from different environments together with your Kubernetes monitoring. For Kubernetes, installing the Sysdig agent has additional benefits like collecting Kubernetes and container metadata for metric enrichment and events. This is very useful for adding context when troubleshooting using Sysdig’s eBPF-based metrics, including per process metrics, network metrics, and golden signals from your HTTP and TCP services.How to start using Prometheus remote write
You just need to configure your Prometheus servers to remote write to your Sysdig account, adding theremote_write
block to your configuration files.
To authenticate against the Prometheus Remote Write endpoint you should use an Authorization Header with your API token as Bearer Token (don’t confuse this with your instance’s Sysdig agent access key). You will have to configure the remote_write section in your prometheus.yml
configuration file like this:
global: external_labels: [ <labelname>: <labelvalue> ... ] remote_write: - url: "https://<region-url>/prometheus/remote/write" bearer_token: "<your API Token>"You can also use the
bearer_token_file
entry to refer to a file instead of directly including the API Token, this is the typical scenario if you store this in a Kubernetes secret, and we recommend this! :)
You will find the right endpoint URL for the Sysdig region you are in our docs.
From version v2.26, Prometheus allows a new way to configure the authorization by including a section within your remote_write
block called authorization
:
global: external_labels: [ <labelname>: <labelvalue> ... ] remote_write: - url: "https://<region-url>/prometheus/remote/write" authorization: credentials: "<your API Token>"Here you can also use the
credentials_file
option, like above.
Note: Prometheus does not reveal the bearer_token
value in the UI.
Controlling metrics sent to Prometheus remote write
By default, all metrics scraped by your Prometheus servers are sent to the Prometheus Remote Write endpoint when you configureremote write
. These metrics will include a remote_write: true
label when stored in Sysdig Monitor, so you can easily identify them.
You can specify custom label/value pairs to be sent along with each time series, using the external_labels
block within the global
section. This allows you to filter/scope metrics when using them, similar to setting an agent tag.
Imagine if you had two different Prometheus servers configured to remote write to Prometheus Remote Write. You could easily include an external label to identify them.
You would configure one server as:
global: external_labels: provider: prometheus1 remote_write: - url: ...And the other as:
global: external_labels: provider: prometheus2 remote_write: - url: ...With the general configuration, all metrics are written remotely to Prometheus Remote Write. You can include
relabel_config
entries to control which metrics you want to keep, drop, or replace.
In the following example, we are only sending metrics from one specific namespace called myapp-ns
.
remote_write: - url: https://<region-url>/prometheus/remote/write bearer_token_file: /etc/secrets/sysdig-api-token write_relabel_configs: - source_labels: [__meta_kubernetes_namespace] regex: ‘myapp-ns’ action: keepFor more information using Prometheus Remote Write on Sysdig Monitor, please refer to our documentation.