Data anomaly troubleshooting
Any of the data anomalies detected by Nobl9 can be caused by reasons outside of Nobl9. This includes issues with a data source or with a data stream itself (like slight changes in a metric behavior).
However, to ensure that SLO data anomalies actually point to an external issue, we recommend checking the correctness of your SLO and SLO objective settings.
This troubleshooting guide will guide you through the issues that you can fix in your SLOs and data source configurations.
No data
The reasons for your SLO not receiving any data, receiving partial data, or not passing the query check verification can be grouped into two broader categories:
- Issues with the data source your SLO is connected to and network connectivity
- Issues with the query configured in your SLO
| Reason | How to address |
|---|---|
| Connection issues for data sources that use the agent method | • Confirm the connection status of your data source on your data source's details page. • Review your agent metrics. |
| Connection issues for data sources that use the direct method | Examine event logs. |
| Inappropriate data source's query parameters | The query interval and timeout must fit the data density: • If the timeout is small, and data is sparse, requests can fail before the data is emitted. • Check the query interval—requests must be sent with enough frequency to capture new data points. |
| Incorrect source-specific settings | Look into source-specific fields: • Ensure the authentication credentials, URL, and other provided values are correct. • Verify the validity of any tokens or API keys. |
| Rate limits hit | Nobl9 stops collecting data when the data source's API rate limit is reached. Collection resumes once the rate limit is reset. |
| Incorrect query | Ensure the syntax is correct according to your data source. |
| Network issues | Check for any network errors between Nobl9 and your data source. |
Nobl9 integrations with data sources (regardless of the connection method used) are resistant to temporary network failures while trying to receive data from them. When your data source becomes available again, Nobl9 catches up on the data lost during the brief outage.
If the data source remains unavailable for an extended period and doesn't recover, Nobl9 cannot collect data from it to resume calculations. In such cases, we recommend checking your data source's status page (see below).
Expand to see status pages
Tools you can use
- Checking data source connection
- For data sources connected using the agent method:
- For data sources connected using the direct method:
- Checking a query and targets
- Query checker for the Datadog, Dynatrace, and New Relic SLOs
- SLI Analyzer
Resources you can refer to for troubleshooting:
Specific Prometheus queries can impact SLO calculations
Nobl9 queries to Prometheus can not contain the following functions:
The reason is these three functions extrapolate missing data. Missing timestamps can lead to inconsistent data received by Nobl9.
Any function using a range vector (like rate, increase, irate)
can introduce another issue because Nobl9 requests data at a specific granularity
(e.g., 15 seconds for Prometheus).
Range vector queries operate over a different interval (e.g., [5m]).
Attempts to align these intervals potentially lead to unpredictable data.
Range vector queries introduce an interval (in PromQL, it’s represented by [x] where x is the duration, such as 5m), and it's hard to match the data intake interval with the aggregation function interval—those two might not overlap, and so the data will be unpredictable.
Constant and no burn
These data anomaly types are caused by either too strict or too lenient SLO objective settings. The following settings have an impact:
- Target
- Numerator (
goodorbad) query - Denominator (
total) query
| Burn type | Threshold SLOs | Ratio SLOs |
|---|---|---|
| Constant burn | The threshold target is too high | The ratio target is too high The numerator is too restrictive The denominator is too broad, or queries irrelevant data |
| No burn | The threshold target is too low | The ratio target is too low The numerator and denominator queries are nearly identical |
In either case, the SLO fails to reflect the actual state of your system.
Tools you can use:
- SLI Analyzer to experiment with different settings and determine a more optimal target
- Query checker for the Datadog, Dynatrace, and New Relic SLOs
Good over total
A Good over total anomaly applies to ratio SLO objectives configured with good and total count metrics. It does not apply to objectives configured with bad and total count metrics. The anomaly means that the good event count exceeded the total event count for the same calculated interval. Use the finding's affected period to compare the underlying good and total queries in your data source.
For incremental count SLOs, Nobl9 first matches positive differences against unmatched total events from the preceding one minute by default. This allows a delayed good series to catch up. Any positive remainder triggers the anomaly. Non-incremental count SLOs use strict interval-by-interval detection.
| Possible cause | How to address |
|---|---|
| The good and total queries use different filters or scopes | Ensure that every event counted as good is also included in the total query |
| The queries use different grouping or aggregation windows | Apply matching grouping dimensions and time windows |
| The queries return different units or apply different transformations | Return both counts in the same unit and review query transformations |
| The configured count method does not match the metric behavior | Verify whether the metrics are incremental or non-incremental |
| The good and total points are not aligned in time | Compare both query results at the timestamps covered by the annotation |
After correcting the query or source data, run Replay for the affected period. Then verify that the Good over total anomaly no longer appears for that period.
If the historical source data cannot be corrected, you can separately review whether a budget adjustment is appropriate for the affected period. Replay findings do not create budget adjustments automatically.
See Good over total anomaly for the detection pipeline, catch-up examples, credit expiry, and the effect of count repair on reliability.
Reimport your historical SLI data
Once the issue is resolved, we recommend replaying your SLO to refill it with historical SLI data for the period when the data anomaly was detected.
Useful links
Query parametersNobl9 featuresSLO troubleshootingFAQ & troubleshootingData source troubleshootingFAQ & troubleshootingNetwork connectivity requirementsRequirementsPermissions and accessData sources: requirementsHistorical data retrievalData sourcesData source connection methodsSLOcademy