Data anomaly detection

Reading time: 0 minute(s) (0 words)

In Nobl9, data anomalies refer to data integrity issues that affect SLO reporting. Currently supported data anomaly is the failure to retrieve data from a data source (a No data anomaly).

Nobl9 can identify these anomalies in two ways:

• Auto-detection: Nobl9 automatically detects when no data has been reported for three hours and creates a system annotation on the SLO.
• Manual configuration: You can configure an SLO to send a notification through a specific alert method if no data is received within a custom time frame.

To distinguish between these methods, Nobl9 automatically labels each data anomaly annotation.

Detection method	Annotation label	Waiting time	Notifications using an alert method
Auto-detection	~anomaly-rule: auto	3 hours	No notifications
Manual setup	~anomaly-rule: manual	From 5 minutes to 31 days	Available for the supported alert methods

Manual detection setup

Before configuring data anomaly detection for your SLO, ensure the following:

• At least one of the supported alert methods is configured
• You have access to it

You can set up data anomaly detection in the following ways:

• In the Nobl9 Web application
• Applying a YAML definition with sloctl apply
• Using the Nobl9 Terraform provider

Nobl9 Web application

To configure data anomaly detection in the Nobl9 Web application, select No data anomaly alert in Step 5 of the SLO wizard. Then, specify how long your SLO must wait for data before sending the notification and select your preferred alert method:

Setting up no data notification on the Nobl9 Web

sloctl

To set up data anomaly detection using sloctl, configure the noData parameters in your SLO YAML definition:

Example of anomalyConfig in an SLO YAML definition

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-slo
    project: my-project
  spec:
    alertPolicies: []
    anomalyConfig:
      noData:
        alertMethods:
          - name: my-alert-method-1
            project: my-project-1
          - name: my-alert-method-1
            project: my-project-2
        alertAfter: 1h

Then, apply it with sloctl apply.

Nobl9 Terraform provider

To set up data anomaly detection using Nobl9 Terraform provider, follow the below pattern:

Example of data anomaly detection config to use with the Nobl9 Terraform provider

[...]
  anomaly_config {
     no_data {
       alertAfter = "15m"
  
       alert_method {
         name = "my-alert-method-1"
         project = "my-project-1"
       }
        
       alert_method {
         name = "my-alert-method-2"
         project = "my-project-2"
       }
     }
   }

Supported alert methods

Notifications on manually configured data anomalies can be sent using the following alert methods:

• Email
• MS Teams
• Opsgenie
• Slack
• Webhook

Key takeaways

• No data anomalies differ from alerts.
  Alerts are events that occur when alert policy conditions are met, and you can retrieve them using sloctl get alerts. No data anomalies, however, are part of the SLO configuration and require an SLO to exist.

• No data anomaly detection applies to SLO objectives.
  A notification is sent for every SLO objective that fails to report data within the specified time frame.

• You can add up to five alert methods for data anomaly notifications per SLO.

• To get notified on no data anomalies, you must have access to the SLO and alert method used to send notifications.

• Query parameters, like query delay or interval, can influence when Nobl9 sends notifications about missing data and the annotation's duration.
  For example, if a query delay is applied, Nobl9 queries for data some time after it was initially registered in the data source. The notifications and annotations reflect this: the SLO charts display data point timestamps based on what's logged in the data source, while the notifications and annotated period are based on Nobl9's anomaly detection time, which depends on the responses from Nobl9 queries.

  

  Anomaly annotation is closed after a data stream resumes

  example for the query delay = 5 minutes and alert after = 10 minutes

  Data point	Time in a data source	Time in Nobl9	No data anomaly detection time
  Last data point before no data period	13:00:00	13:05:00	13:15:00
  First data point after no data period	14:00:00	14:05:00	14:15:00

Useful links

Check out these related guides and references:

Query delayNobl9 featuresNo data troubleshootingTroubleshootingSLOs general troubleshootingTroubleshootingData source troubleshootingTroubleshootingNetwork connectivity requirementsData sources: requirementsPermissions and accessRequirements