Unveiling Sysdig’s new custom webhook

Introduction

Sysdig Monitor has long integrated with a variety of Notification Channels, allowing users to forward alerts to a multitude of third-party services. Currently, users can choose to forward Sysdig Monitor’s alerting events to external services like PagerDuty, Slack, Email, Microsoft Teams, and many more.

At Sysdig, we believe integrating with third-party services should be a seamless experience. However, with an increasing number of notification channels to integrate, we needed to develop a flexible and generalized integration suitable for any third-party service provider.

For this exact purpose, Webhooks have always been a ubiquitous integration mechanism between services and applications. Being nothing more than an HTTP API on a remote server, they allow clients to access the services provided by a third party.

Sysdig Monitor has included a Webhook notification channel for almost as long as the product exists. However, it suffered from several limitations.

With this in mind, we have released a much more powerful and flexible Webhook notification channel. Today, we introduce the Custom Webhook notification channel.

The problem

If Webhooks are a ubiquitous method of integration and Sysdig Monitor has always included a Webhook notification channel, why was it deemed necessary to release a new version?

You’ll remember that calling upon a Webhook is simply performing a plain HTTP request against an external API. As an HTTP request, it requires a given external address, to which a payload gets sent, including a well-defined number of headers and a well-formatted body. The body can be plain text, but it can also be of several formats, such as JSON, YAML, etc.

Within Sysdig Monitor, the previous Webhook notification channel provided some relevant features, but it was also limited in a number of ways. It did allow users to specify all the headers for the request. It also allowed TLS verification to be skipped over HTTPS. At last, it allowed for some custom payload data to be defined by the user. With all these pieces set, the user could expect the final request to be the sum of the external address provided, plus the headers specified and the payload defined. It seemed to be left unclear which HTTP method would be used.

Payload example

Further examination of the payload delivered by the alert notification showed a surprising number of assumptions in the response:

1. First, it assumed the request should automatically be sent as HTTP POST. This is true for most integrations, but it is by no means guaranteed, as common REST APIs might use PUT or PATCH to completely or partially modify a resource, and DELETE to remove one.
2. Second, the actual user-defined headers do get transmitted, but three were set automatically: Content-type as application/json, Accept, as well as application/json and User-agent to Mozilla/5.0.
3. Third, the HTTP body contents of the Alert Notification were defined by Sysdig with little room for user adjustments. Furthermore, the apparent customizability given to the user through the custom data field seen previously lies only at the end of the larger JSON object structure (Image 2).

This approach, while having served our user’s needs until now, does not truly open up the potential for custom integrations, since any third-party integration might be very strict on the three factors mentioned: HTTP Method, Headers, and Body.

As such, any request that does not strictly match the format required by a given integration will often be rejected through a response of HTTP status 400 Bad Request. This is true for Slack, as well as Google Chat, and many more integrations. Clearly, our current Webhook did not completely address our requirements.

The need

Thus, there was a need within Sysdig Monitor for a truly customizable Webhook – one in which a user could exactly specify the contents of the payload, and flexible enough to be outfitted for any third-party API. The result would be one notification channel with the potential of integrating with anything the user wanted.

Initially, this wasn’t deemed a considerable challenge. Allowing, for instance, our users to customize their Webhook’s HTTP Method, and making sure the body to be sent matched exactly what they defined, was a change that could have been introduced in a matter of days.

Further investigation showed that a custom user-defined payload would not be enough to meet our customer’s needs. Customers use Sysdig Webhooks as part of automated workflows and these Webhooks rely on the dynamic context of the alert triggered (e.g., severity, labels, and host metadata).

Just allowing our users to customize a static body to be sent to third-party APIs would not enable these services to understand whether a Kubernetes pod went down, a Postgres instance reached its connection limit, or even more worrying situations.

How could we allow users to statically customize every single detail about the request being made, while also letting them dynamically include alert and event information specific to the runtime situation at hand?

The Solution: Sysdig’s Custom Webhook & Sysdig Templating Language

The solution was to create a Custom Webhook notification channel where, in addition to defining the exact payload structure, the user could interpolate runtime information specific to the alert context through an intermediate language called Sysdig Templating Language.

Sysdig Templating Language allows users to define exactly what the output body should be, while giving access to specific alert and event runtime variables. As such:

The alert that fired was {{@alert_name}}

For instance, the above is a valid Sysdig Templating Language sentence. This sentence only makes sense in the context of an alert firing. Under such conditions, evaluating this sentence can render the output “The alert that fired was High CPU Reached.”

Having users define the payloads to be sent using Sysdig Templating Language allows third-party systems to receive the dynamic runtime context information necessary in order to make sense of the incoming request, while still giving the freedom to only send exactly the information needed.

Accessing context

We’re shipping the Custom Webhook with Sysdig Templating Language support with a number of variables supported out of the box.

All these variables belong to one of two categories: alerts and events.

• Alert variables allow users to access the configuration for the alert responsible for the triggering event. These include alert name, description, scope, group by, condition, and many more.
• Event variables instead relate to the triggering event itself. These include, for instance, event state, triggering timestamp, infrastructure labels, etc.

Feel free to browse through the Custom Webhook complete reference to learn more about all the capabilities we’re shipping with.

Delving deeper: Conditional Logic

However, we didn’t want to stop with template variables, and realized that some use cases from our users could require more complex logic. Notification channels can indeed be triggered by many different alert types.

What if our users wanted different payloads to be sent based on the severity of the alert triggered (high, medium, low, etc.), the type of alert (metric, prometheus, downtime, etc.) triggered, or on whether the current event related to an alert resolution instead of the initial alert notification? It was for this reason that we have added built-in support for Conditional Logic.

Conditional Logic is important in situations where users might want to send different payloads based on some inherent properties of the alert in question. A simple use case could be to provide differentiation based on the configured severity of the alert.

Conditional Logic example

With Conditional Logic, it is possible that for a high severity alert a different payload gets sent when compared with a low severity one. It was for this reason that Sysdig Templating Language also introduces statements, in other words, language constructs that express some action to be carried out.

Unlike variables, however, they include a document body, so the initial tag requires a corresponding closing one. As such, the sentence:

{{#if_metric_alert}}
some text
{{/if}}Code language: PHP (php)

When evaluated in the context of triggering metric alert, renders as output:

“some text”

However, in the context of a triggering Prometheus alert, it would render no output.

For the first release of this feature, we’re shipping conditional statements for all alert types, all alert severities, and with the ability to distinguish between a triggering alert and resolving alert.

Conditional Logic example with else-if

Additionally, conditional statements can be extended with else-if logic, as such:

{{#if_metric_alert}}
do this
{{#else if_downtime_alert}}
do that
{{#else}}
nothing matched
{{/if}} Code language: JavaScript (javascript)

Conditional statements can thus consist of a main condition, followed by zero or more alternative branches, followed by an optional default branch.

As soon as you start typing in the editor window of the new Custom Webhook, you’ll receive immediate feedback on the grammatical soundness of the document being written, as well as auto-complete functionality with inline documentation on the meaning of each variable and statement.

Wrapping it up

The new Custom Webhook notification channel opens doors for potential integrations that the previous Webhook implementation may have left shut. It does so by closing the following gaps:

• External Address
• HTTP method
• HTTP Headers
• HTTP body payload with the ability to reference context
• Optional certificate verification

Any HTTP request can be made to a third-party service with full control of headers, payload, and alert context.

In the near future, you can expect further improvements to this feature, by giving the ability to easily define different payload formats.

All in all, we believe this new notification channel can truly revolutionize your workflows.

Use case: SMS sending for Alert Events with the new Custom Webhook

A common use case for integrating with alerts is sending an SMS. Given the general availability of a cell phone connection throughout the world, SMS’s are a convenient solution since they don’t require a smartphone or data plan.

Prior to the Custom Webhook, there was no way within Sysdig Monitor to integrate with third-party SMS services.

For the purposes of this use case, we will be using the third-party Plivo. Trial accounts provide some basic SMS delivery capabilities but do give access to their JSON based REST API.

Step 1 – Create an account and obtain your API keys

Go ahead and create a trial account on Plivo.com.

After registering, within its main dashboard, you can find your authentication id and auth token. Save this for later.

Note: Access the Messaging section, and then Settings > Geo Permissions, and make sure you have enabled the country to which you’d like to enable SMS sending to.

Step 2 – Create a new Custom Webhook channel within Sysdig

Access Sysdig Monitor. Hover over your profile name at the bottom left corner and then click Settings in the top right corner.

On the window just opened access Notification Channels.

On the newly opened page, click on the plus sign in the top right corner to add a new Notification Channel and choose “Custom Webhook”.

Step 3 – Insert the details for your notification channel

• 1. Type a name for your channel, such as “SMS delivery”
• 2. Insert the following Plivo address:

https://api.plivo.com/v1/Account/{accountId}/Message/Code language: JavaScript (javascript)

Where accountId should get substituted with the appropriate value obtained in Step 1.

• 3. On the Method and Headers box, you can leave the default HTTP method to POST, as that’s exactly what Plivo requires.
• 4. You should add a new Custom Header on the same box, and insert “Authorization” as the header name. Its value should be the string “Basic <base64Token>.” Base64 token should be the base 64 encoding of the string “<accountId>:<accountToken>,” the two values taken from step 1. You can use an online tool to obtain the resulting base 64 encoding.
• 5. Inside the editor window, type the following payload:

{
    "src": "Sysdig",
    "dst": "+<phone_number_with_country_code>",
    "text": "{{@event_title}}",
    "type": "sms",
    "url": "https://foo.com/sms status",
    "method": "POST",
    "log": "true",
    "trackable": false
}
Code language: JavaScript (javascript)

We are thus trying to contact Plivo with the format required by its API. In it, we define the destination phone number to be contacted, with the SMS text being exactly the same as the event title for the alert that triggered. This is a small sentence that should give enough context to the user on which situation is occurring.

• 6. You can now use the Configure Test Notification box to simulate a given alert firing over this channel. Choose an appropriate alert severity and an alert type and hit “Send Test Notification.”
• 7. You should have received your SMS. Hit save.

As soon as you configure a new alert within Sysdig Monitor and connect it to your new “SMS delivery” channel, the specified phone number will be warned about all the potential incidents occurring in your infrastructure.

We hope to have provided a use case interesting enough that greatly increases Sysdig’s capabilities, while also demonstrating the power of the new notification channel introduced.