Monitoring Availability Metrics with Blackbox exporter and Sysdig

[Webinar] So Many Metrics, So Little Time: 5 Prometheus Exporter Best Practices

The Prometheus Blackbox exporter allows endpoints exploration over several protocols, such as HTTP(S), DNS, TCP, and ICMP. This exporter generates multiple metrics on your configured targets, like general endpoint status, response time, redirect information, or certificate expiration dates.

The Blackbox Exporter works out-of-the-box, as it just focuses on external visibility details. To get more detailed metrics, you can instrument your applications.

Since the Blackbox exporter usually runs as a self-hosted service in a container, it can be easily deployed in your Kubernetes cluster.

By default, when performing HTTP probes, this exporter uses the GET HTTP method to explore your targets and expects a status code similar to 2xx. This behavior is easily configurable along with others, like basic or bearer token-based authentication, custom headers, or the usage of a proxy server to connect to targets.

Some interesting use cases where you can use Blackbox exporter with Sysdig are:

  • Detecting certificate expiration.
  • Analyzing the latency of specific targets and paths of services running in the same cluster.
  • Detecting endpoint failures.
  • Monitoring latency and reachability of an external endpoint like third services.

Installing and configuring Blackbox exporter in Kubernetes

You can use Helm to install and configure your Blackbox installation in Kubernetes.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

Under the parameter config, you can include your probes configuration. For example, you can define several HTTP probes for different scenarios. You can include the following configuration in a Helm values file that has to be provided with the “-f” flag.

config:
  modules:
    http_2xx:
      prober: http
      timeout: 2s
      http:
        follow_redirects: true
    http_k8s:
      prober: http
      timeout: 2s
      http:
        follow_redirects: true
        bearer_token: /var/run/secrets/kubernetes.io/serviceaccount/token
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

In the http_2xx module, most options are kept by default. In the http_k8s example, it has been configured with a custom CA and bearer token-based authentication.

Once you have correctly configured your Helm values, you can install the Blackbox exporter:

$ helm install [RELEASE_NAME] prometheus-community/prometheus-blackbox-exporter -f values.yaml [-n NAMESPACE]

Scraping Blackbox using Sysdig Agent

The Blackbox exporter can be invoked using the native Prometheus service discovery, available with the Sysdig Agent.

To use the native Prometheus service discovery, you’ll need to enable the prometheus.prom_service_discovery configuration by adding the following lines in your Agent configuration:

prometheus:
  enabled: true
  prom_service_discovery: true

Target configuration for Blackbox exporter must be provided using Prometheus relabel_configs in a static scrape configuration. You can include the list of targets with the module you want to use as a parameter in your Sysdig Agent prometheus.yaml file.

prometheus.yaml: |
  global:
    scrape_interval: 10s
  scrape_configs:
  - job_name: blackbox
    metrics_path: /probe
    params:
      module:
      - http_2xx
    relabel_configs:
    - source_labels:
      - __address__
      target_label: __param_target
    - source_labels:
      - __param_target
      target_label: instance
    - replacement: blackbox-service:9115	# Blackbox hostname:port
      target_label: __address__
    static_configs:
    - targets:
      - http://front-end.sock-shop.svc.cluster.local
      - https://docs.sysdig.com/
      - https://promcat.io
      - https://api.promcat.io/apps
      - https://my.failing.app

In the relabel_configs block, replace “blackbox-service:9115” with your Blackbox exporter hostname:port. You can directly use the Blackbox Kubernetes Service and port.

In addition to static configuration, there is a common snippet where you do autodiscovery of services in the cluster and use Blackbox to check all the services with the annotation “prometheus.io/probe: true”.

prometheus.yaml: |
  global:
    scrape_interval: 10s
  scrape_configs:
  - job_name: kubernetes-services
    kubernetes_sd_configs:
    - role: service
    metrics_path: /probe
    params:
      module:
      - http_2xx
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_probe
    - source_labels:
      - __address__
      target_label: __param_target
    - replacement: blackbox-service:9115	# Blackbox hostname:port
      target_label: __address__
    - source_labels:
      - __param_target
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels:
      - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - source_labels:
      - __meta_kubernetes_service_name
      target_label: kubernetes_name

Visualize and Alert on available metrics

Once you set up Sysdig agent to use the Blackbox exporter to probe your targets, metrics will flow to your Sysdig instance:

probe_duration_seconds metric returns how long the probe took to complete. You can use the following query to analyze the probe duration by each one of your targets:

avg(probe_duration_seconds) by(instance)

When our applications consume third-party services, it is important to detect outages or wrong code responses. You can get this information with the probe_http_status metric.

Certificate expiration can produce miscommunications between your services. The probe_ssl_earliest_cert_expiry metric provides you the timestamp in seconds when the certificate chain will no longer be valid.

Represent the expiring time of all your endpoints in one dashboard with the following query:

probe_ssl_earliest_cert_expiry - time()

Set up your own alerts for certificates that will expire in less than 60 days by using the following PromQL query:

probe_ssl_earliest_cert_expiry - time()

You can also detect when your probes are failing by using the probe_success metric:

sum(probe_success) by (instance) != 1

You can create your own dashboard for a complete overview of all your targets based on the available information:

If you would like to try this integration, we invite you to sign up for a free trial of Sysdig Monitor.

Stay up to date

Sign up to receive our newest.

Related Posts

Kubernetes monitoring with Prometheus, the ultimate guide

Five Prometheus exporters best practices to increase your productivity

Top 5 key metrics for monitoring Amazon RDS