Data anomaly detection

Reading time: 0 minute(s) (0 words)

Data anomaly detection is crucial for maintaining reliable SLO monitoring. When the data stream from your data source has any deviations, your SLOs cannot be calculated properly, creating blind spots in your observability.

Nobl9 offers two ways to detect data anomalies:

• Manually configured detection of no data. It is available for everyone—you set up notifications for your SLO. These notifications trigger when this SLO stops reporting data for a duration you set.
• Automatic data anomaly detection. This is an advanced feature for the Nobl9 Enterprise Edition. With it, five data anomaly types can be detected automatically. The auto-detection feature uses centralized defaults that can be customized on demand. Availability of individual anomaly types may depend on your organization's rollout configuration.

Once a data anomaly is detected, Nobl9 updates the tile on this SLO details page and creates an SLO annotation with details about the data anomaly and a link to the affected SLO.

Manual configuration and auto-detection both operate fully and seamlessly alongside each other.

This overview explains how data anomaly detection is configured and links to a dedicated page with examples and troubleshooting steps for each anomaly type.

No data manual configuration

With manually configured No data anomaly detection, you're setting up notifications for no data. The notifications are sent using one of the supported alert methods. Ensure you have access to at least one of the following:

• Email
• MS Teams
• Opsgenie
• Slack
• Webhook

You can set up data anomaly detection:

• In the SLO wizard within the Nobl9 Web application
• Applying an SLO YAML definition with sloctl apply
• Using the Nobl9 Terraform provider

Manual no-data alerts only

The settings in this section apply only to manually configured No data anomaly alert notifications for a specific SLO. They do not change Nobl9's automatic data anomaly detection in SLO oversight. In particular, automatic No data anomaly detection continues to use its own centralized rules and defaults.

Nobl9 Web application

To configure data anomaly detection in the Nobl9 Web application, select No data anomaly alert in Step 5 of the SLO wizard. Then, specify how long your SLO must wait for data before sending the notification, select your preferred alert method, and, if needed, enable Treat zero as no data.

When this option is enabled, zero-valued points are treated as no data for this manual no-data alerting flow. Leaving it unchecked preserves the default behavior.

Setting up no data notification on the Nobl9 Web

sloctl

To set up data anomaly detection using sloctl, configure the noData parameters in your SLO YAML definition:

Example of anomalyConfig in an SLO YAML definition

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-slo
    project: my-project
  spec:
    alertPolicies: []
    anomalyConfig:
      noData:
        alertMethods:
          - name: my-alert-method-1
            project: my-project-1
          - name: my-alert-method-1
            project: my-project-2
        alertAfter: 1h
        treatZeroAsNoData: true

Set treatZeroAsNoData: true to treat zero-valued points as no data for this manual no-data alert. If you omit this field or set it to false, the default behavior remains unchanged.

Then, apply it with sloctl apply.

Nobl9 Terraform provider

To set up data anomaly detection using Nobl9 Terraform provider, follow the below pattern:

Example of data anomaly detection config to use with the Nobl9 Terraform provider

[...]
   anomaly_config {
      no_data {
        alertAfter = "15m"
        treat_zero_as_no_data = true
  
        alert_method {
          name = "my-alert-method-1"
          project = "my-project-1"
        }
        
       alert_method {
         name = "my-alert-method-2"
         project = "my-project-2"
       }
     }
   }

Use treat_zero_as_no_data = true to enable the same behavior in Terraform. If you leave it unset or set it to false, zero-valued points are handled with the default behavior.

• You can add up to five alert methods for your manual no data anomaly notifications per SLO
• To receive notifications for no data anomalies, you must have access to both the SLO and the alert method it uses
• Query parameters, like query delay, can affect when Nobl9 sends notifications for missing data and the duration of the corresponding annotations. This creates a difference in timestamps: SLO charts use the time from the data source, while notifications and annotations are based on when Nobl9's query confirms the anomaly.

Anomaly annotation is closed after a data stream resumes

example for the query delay = 5 minutes and alert after = 10 minutes

Data point	Time in a data source	Time in Nobl9	No data anomaly detection time
Last data point before no data period	13:00:00	13:05:00	13:15:00
First data point after no data period	14:00:00	14:05:00	14:15:00

Refer to troubleshooting steps to address no data anomalies

Data anomaly auto-detection

Enterprise

Automatic data anomaly detection is available exclusively in the Nobl9 Enterprise Edition

The following data anomaly types are detected:

Data anomaly	Description	Default waiting time	Default cooldown
No data	No data is being reported by an SLO objective for one week	1 week	No cooldown
Constant burn	An SLO objective is constantly burning its error budget for an unusually long time	1 week	30 minutes
No burn	An SLO objective is not burning its error budget for an unusually long time	8 weeks	No cooldown
Incremental mismatch	A ratio SLO, configured with the incremental data count method, has received a non-incremental data point	Triggered immediately	1 day
Good over total	The good event count for a ratio SLO exceeds its total event count for the same calculated interval	Triggered immediately	30 minutes

Good over total availability

Good over total is being rolled out gradually and may not yet be enabled for your organization. Contact Nobl9 Support for availability details.

Customization capabilities

Contact Nobl9 Support to set a custom waiting time or cooldown interval for your organization.

No data

Automatic No data detection finds objectives that stop reporting points for an extended period. Its default observation period is one week.

Read how the No data anomaly works, including a timeline and troubleshooting steps.

Constant and no burn

Constant burn and No burn examine opposite long-running error budget patterns:

• Constant burn means the objective continuously consumes error budget.
• No burn means the objective does not consume error budget for an unusually long time.

Interpreting data anomalies

While we do our best to help you identify misconfigurations, the range of potential causes is too extensive to guarantee that all data anomalies are solely due to misconfigurations.

That is why a Constant burn data anomaly may be triggered by an actual incident on your system side, instead of a misconfiguration.

Similarly, for No burn, most cases should point to misconfigurations; however, some data anomalies may be produced for a normal, healthy SLO that simply has not burned its error budget.

Treat data anomalies as hints rather than definitive signals that demand an immediate response.

Incremental mismatch

Incremental mismatch detects a non-zero decrease in a count metric configured as incremental. A point at zero is treated as an explicit counter reset.

Read how the Incremental mismatch anomaly works, including reset behavior and examples.

Good over total

Good over total checks whether good events exceed total events in an interval. Incremental SLOs with good and total count metrics allow a recent total lead to explain delayed good events within a one-minute catch-up window by default. Non-incremental SLOs with good and total count metrics report the anomaly immediately. Objectives configured with bad and total count metrics are not evaluated for this anomaly.

Read how the Good over total anomaly works, including catch-up examples and the effect on reliability.

Data anomalies vs. alerts

While both data anomalies and alerts can create annotations on an SLO objective, they are distinct concepts with different purposes and configurations. The following table summarizes their key differences.

Parameter	Data anomaly	Alert
Definition	An automatically detected deviation from expected data patterns, common to all SLOs	A user-defined notification triggered when specific, configured conditions are met
Focus	The integrity of incoming data	SLO's error budget or error budget burn rate
Trigger	Built-in system logic that analyses data patterns	A specific, user-configured alert policy
Rules	System-defined and cannot be changed by the user	Customizable by the user
Notifications	Doesn't send notifications using alert methods Except for manual No data anomalies	Sends notifications using configured alert methods
Silencing	Cannot be silenced	Can be silenced based on user configuration

Manually configured "No data" detection rules

Manually configured No data detection rule follows the same logic as the auto-detection rule but must be enabled for an SLO explicitly and allow you to customize the waiting time and choose a specific alert method for notifications.

Useful links

Check out these related guides and references:

Query data anomalies with the APIAnomalies APIQuery delayNobl9 featuresData anomaly troubleshootingFAQ & troubleshootingSLOs troubleshootingFAQ & troubleshootingData source troubleshootingFAQ & troubleshootingNetwork connectivity requirementsData sources: requirementsPermissions and accessRequirements