No data anomaly alerts

Reading time: 0 minute(s) (0 words)

Nobl9 users can stay on top of their SLOs by configuring no data notifications to detect and alert on missing data anomalies. These notifications ensure that gaps in monitoring data or reporting do not go unnoticed, thereby enhancing the reliability of your systems.

With this SLO monitoring feature, you can customize anomaly detection settings for your SLOs to define how and when to receive notifications if your SLO fails to report data.

Once a no data anomaly is detected, Nobl9 creates an SLO annotation and sends a notification using a supported alert method. The details include the information about the anomaly and a link to the affected SLO details.

By default, notifications are sent when the SLO reports no data during 15 minutes. However, you can set a custom time frame, from 5 minutes to 31 days.

Scope of support

No data anomaly notifications can be sent using the following alert methods:

• Email

• MS Teams

• Opsgenie

• Slack

• Webhook

  

  Slack notification for the "No data" anomaly.

Setting up anomaly notifications

Before configuring anomaly notifications for your SLO, ensure the following:

• At least one of the supported alert methods is configured
• You have access to it

You can set up anomaly notifications in the Nobl9 Web application, applying a YAML definition with sloctl apply, or using the Nobl9 Terraform provider.

Nobl9 Web application

To set up anomaly notifications in the Nobl9 Web application, select No data anomaly alert in Step 5 of the SLO wizard. Then, specify how long your SLO must wait for data before sending the notification and select your preferred alert method:

Setting up no data notification on the Nobl9 Web

sloctl

To set up anomaly notifications using sloctl, configure the noData parameters in your SLO YAML definition:

Example of anomalyConfig in an SLO YAML definition

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-slo
    project: my-project
  spec:
    alertPolicies: []
    anomalyConfig:
      noData:
        alertMethods:
          - name: my-alert-method-1
            project: my-project-1
          - name: my-alert-method-1
            project: my-project-2
        alertAfter: 1h

Then, apply it with sloctl apply.

Nobl9 Terraform provider

To set up anomaly alert using Nobl9 Terraform provider, follow the below pattern:

Example of anomaly notifications configuration to use with the Nobl9 Terraform provider

[...]
  anomaly_config {
     no_data {
       alertAfter = "15m"
  
       alert_method {
         name = "my-alert-method-1"
         project = "my-project-1"
       }
        
       alert_method {
         name = "my-alert-method-2"
         project = "my-project-2"
       }
     }
   }

Key takeaways

• No data anomalies differ from alerts.
  Alerts are events that occur when alert policy conditions are met, and you can retrieve them using sloctl get alerts. No data anomalies, however, are part of the SLO configuration and require an SLO to exist.

• No data anomaly alerting applies to SLO objectives.
  An notification is sent for every SLO objective that fails to report data within the specified time frame.

• You can add up to five alert methods for anomaly notifications per SLO.

• To get notified on no data anomalies, you must have access to the SLO and alert method used to send notifications.

• Query parameters, like query delay or interval, can influence when Nobl9 sends notifications about missing data and the annotation's duration.
  For example, if a query delay is applied, Nobl9 queries for data some time after it was initially registered in the data source. The notifications and annotations reflect this: the SLO charts display data point timestamps based on what's logged in the data source, while the notifications and annotated period are based on Nobl9's anomaly detection time, which depends on the responses from Nobl9 queries.

  

  Anomaly annotation is closed after a data stream resumes

  example for the query delay = 5 minutes and alert after = 10 minutes

  Data point	Time in a data source	Time in Nobl9	No data anomaly detection time
  Last data point before no data period	13:00:00	13:05:00	13:15:00
  First data point after no data period	14:00:00	14:05:00	14:15:00

Useful links

For a more in-depth look, consult additional resources:

Query delayNobl9 featuresNo data troubleshootingTroubleshootingSLOs general troubleshootingTroubleshootingData source troubleshootingTroubleshootingNetwork connectivity requirementsData sources: requirementsPermissions and accessRequirements