Skip to main content

Alert Methods

When an incident is triggered, Nobl9 enables you to send an alert to a notification engine or tool (for example, PagerDuty). Nobl9 also supports integration with a web endpoint by using webhooks where you define the endpoint and parameters to pass.

Alerting on SLOs allows you to react immediately to incidents that matter from the perspective of the user experience of your Service (e.g., in terms of latency, errors, correctness, and other SLO-related concepts). Alerts improve the control of what’s going on in your system and enable you to do better-contributing factor analysis when something goes wrong.

Here are important things to keep in mind while setting up your Alerts:

  • Both, our attention and energy are limited resources. SLO Alerts must correspond to real and urgent issues of your system.

  • Keep in mind that to improve your monitoring, these Alerts have to be intentional (i.e., well-defined) and need to evolve together with your system.

Alert Policy & Alert Method Lifecycle​

Cooldown period​

You can configure Cooldown period for your Alert Policies. Follow YAML Guide to see how to set up the cooldown period through YAML.

What is a Cooldown Period?​

Cooldown period was designed to prevent from sending too many alerts. It is an interval measured from the last time stamp when all Alert Policy conditions were satisfied. Each SLO's objective triggers its own alerts and keeps its own cooldown period.

Assumptions:

  • When cooldown conditions are satisfied (i.e., no Alert events are triggered during its defined duration), an Alert event is resolved.

  • New Alert is triggered unless the cooldown period is satisfied.

  • The cooldown period may not be satisfied at a given time and won’t trigger any alerts. However, if, over time, all the alert conditions are satisfied again, the cooldown period is then reset and will be calculated from the time when any of the conditions stopped to be satisfied.

The diagram below shows a simplified lifecycle of an Alert Policy with a defined cooldown period:

lasts for condition
Image 1: Alerting lifecycle
tip

Check out the Alerting - Use Case in SLOcademy for more complex Alerting example.

Configuring Cooldown Period in the UI​

Refer to Getting Started guide for details.

Alert Policy Statuses​

When an alert policy is in Triggered state, no other new alert can be triggered unless the alert is resolved or canceled.

Alert Policy statuses adhere to the following criteria:

  • An alert is resolved when any of the conditions stopped to be true AND the cooldown period expired from that time.

  • An alert is canceled when Alert policy configuration has changed OR a new calendar window has started for the calendar aligned time window SLOs.

note

When an Alert event assumes the resolved status within the duration of AlertSilence, Nobl9 will send you an all-clear notification. For more details, see Silencing Alerts | Nobl9 Documentation.

Retrieving Triggered Alerts in sloctl​

Using sloctl, you can retrieve information when an alert stopped to be valid. To do so, run the following command in sloctl:

sloctl get alert

Here's an example of a triggered Alert that hasn't been resolved yet:

apiVersion: n9/v1alpha
  kind: Alert
  metadata:
    name: 6fbc76bc-ff8a-40a2-8ac6-65d7d7a2686e
    project: alerting-test
  spec:
    alertPolicy:
      name: burn-rate-is-4x-immediately
      project: alerting-test
    service:
      name: triggering-alerts-service
      project: alerting-test
    severity: Medium
    slo:
      name: prometheus-rolling-timeslices-threshold
      project: alerting-test
    status: Triggered
    thresholdValue: 950
    triggeredClockTime: "2022-01-16T00:28:05Z"

Labels and Alert Methods​

Adding Labels to Alert Methods​

Users can add one or more labels to an alert policy, which will be sent along with the alert notification when the policy’s conditions are met.

Other Relevant Resources​

For useful tips on how to get started with your first Alert check Your First Alert Policy!. Also see our Tips and Tricks.

If you describe infrastructure as code, you might also consider defining the Alert Methods with the same convention. You can find more details in our Terraform documentation.

Last updated on
Submit Feedback