Alerting on SLOs—overview
When an incident is triggered, Nobl9 can send an alert to a notification engine or tool (for example, PagerDuty). Nobl9 also supports integration with a web endpoint by using webhooks where you define the endpoint and parameters to pass.
Alerting on SLOs allows you to react immediately to incidents that matter from the perspective of the user experience of your service (e.g., in terms of latency, errors, correctness, and other SLO-related concepts). Alerts improve the control of what’s going on in your system and allow you to do better-contributing factor analysis when something goes wrong.
Here are important things to keep in mind while setting up your alerts:
Both our attention and energy are limited resources. SLO alerts must correspond to real and urgent issues of your system.
Keep in mind that to improve your monitoring, these alerts have to be intentional (i.e., well-defined) and need to evolve together with your system.
With Nobl9 system annotations feature, you can see an annotation added to the SLO objective charts by default each time an alert is triggered or resolved (they are displayed regardless of whether an alert policy is silenced). For more detailed information, refer to the SLO annotations documentation.
Alert policy & alert method lifecycle
You can configure Cooldown period for your alert policies. Follow YAML guide to see how to set up the cooldown period through YAML.
What is a cooldown period?
Cooldown period was designed to prevent sending too many alerts. It is an interval measured from the last time stamp when all alert policy conditions were satisfied. Each SLO objective triggers alerts and keeps its cooldown period.
When cooldown conditions are satisfied (i.e., no alert events are triggered during its defined duration), an alert event is resolved.
New alert is triggered unless the cooldown period is satisfied.
The cooldown period may not be satisfied at a given time and won’t trigger any alerts. However, if, over time, all the alerting conditions are satisfied again, the cooldown period is then reset. It will be calculated from the time when any of the conditions stopped to be satisfied.
The diagram below shows a simplified lifecycle of an alert policy with a defined cooldown period:
Check out the Alerting - use case in SLOcademy for the complex examples.
Configuring cooldown period in the UI
Refer to Getting started for details.
Alert policy statuses
When an alert policy is in
Triggered state, no other new alert can be triggered unless the alert is resolved or canceled.
Alert policy statuses adhere to the following criteria:
An alert is resolved when any of the conditions stopped to be true AND the cooldown period expires from that time.
An alert is canceled when alert policy configuration has changed OR a new calendar window has started for the calendar aligned time window SLOs.
When an alert event assumes the
resolved status within the duration of
Nobl9 will send you an all-clear notification.
Read mode in Silencing alerts.
Displaying triggered alerts in the UI
If any of your SLOs fires alerts, Nobl9 will display this information on the main pane of the Service Health dashboard, next to the SLO name in the SLO Grid view and the SLO Grid view tree:
Retrieving triggered alerts in
sloctl, you can retrieve information when an alert stopped to be valid. To do so, run the following command in
sloctl get alerts
For more details on the
sloctl get alerts command, check
sloctl User guide.
- unresolved alert
- resolved alert
Here's an example of a triggered alert that hasn't been resolved yet:
Here's an example of a resolved alert:
Alert list on the SLO grid view allows you to view alert events related to your SLO. You can access the alerts by accessing the SLO Details view and clicking the Alerts tab at the top:
You can click on each tile to check all details of every alert (see the video below):
Nobl9 limits displaying alerts to 1000 most recent alert events. You can use filters to narrow down the results. You can filter by:
- Alert statuses:
- SLO objective names
- Alert policy name
- Date (for example, last hour/week/month, current time window, or by a custom date range)
Keep in mind that the filters are linked by the
AND logical operator.
In rare situations, Nobl9 won't return some alerts. This can happen because:
- Nobl9 returns alerts for existing objects only (alert policies, SLOs, services, and objectives).
- If you delete any objects, Nobl9 won't return alerts for them in the alert list.
- If you delete an SLO, alert policy, or service and recreate it with the same name, Nobl9 won't return results for it.
- If you unlink an alert policy from an SLO, Nobl9 won't return alerts for it.
Alert list and RBAC
Nobl9 returns alerts triggered in a given project only (the alert project is the same as the SLO project). If you don't have permission to view SLO in a given project, you won't see these alerts.
Labels and alert methods
Adding labels to alert methods
Users can add one or more labels to an alert policy, which will be sent along with the alert notification when the policy’s conditions are met.
Other relevant resources
If you describe infrastructure as code, you might also consider defining the alert methods with the same convention. You can find more details in our Terraform documentation.