Skip to main content

No Data - Troubleshooting

Reading time: 0 minute(s) (0 words)

The reasons for your SLO not receiving any data, partial data, or not passing the Query check verification might be grouped into two broader categories:

  • Issues with the data source your SLO is connected to.

  • Issues with the query configured in your SLO.

Issues with Data Sourcesโ€‹

Before more in-depth troubleshooting, we advise to:

  • Verify the connection status of your data source in the Integrations tab.

  • Check the configuration of your data source: ensure that your authentication credentials (that is, API Keys), URL, and other source-specific values are correct.

  • Check your data sourceโ€™s logs to identify any returned errors.

If youโ€™ve completed the above steps and youโ€™re still facing the issue with your data source, refer to the section below of this document.

Network Issuesโ€‹

Nobl9 SLO calculations are prone to errors in cases where the Nobl9 Agent canโ€™t gather all the necessary data from data sources. This issue might occur when, for example, there are network issues between Nobl9 and the respective data source. Refer to Agent Troubleshooting for more details.

Nobl9 agents are resistant to temporary network failures while trying to receive data from external sources. Whenever the source becomes available again, the agent catches up on the data lost during the brief outage. If the source becomes unavailable for an extended period and doesn't recover, Nobl9 can't collect data from it and process resume calculations. In such cases, we recommend checking your source's status page (see below).

Expand to see status pages

Rate Limitingโ€‹

When integrating with Data Sources, N9 agent must comply with the rate limits set by their APIs. Strict rate limits can be the reason for Nobl9 agent to stop collecting data.

note

For more details on the API rate limits for each data source, refer to the API Rate Limits sections in each data source documentation.

Issues with Queriesโ€‹

Any time you enter a wrong query in Nobl9, itโ€™s likely that your burn rate calculations will be incorrect. Nobl9 offers simple validations for the input for metrics that ensure you provide all the required values necessary to process your SLI data, but it doesnโ€™t handle more complex queries. Incorrect queries can have an impact on both threshold and ratio-type metrics. In general, burn rate calculations can be incorrect if:

  • Queries return unexplainable, unpredictable data or no data at all.

  • good or bad, and total queries are misplaced - this happened in the past and is easily discoverable. good will always be greater than the total in this case.

Incorrect SLO configurationโ€‹

Most often, queries return incorrect data when SLOโ€™s data collection method is set to incremental while its SLI data is in fact non-incremental.

caution

Most often, queries return incorrect data when SLOโ€™s data collection method is set to incremental while its SLI data is, in fact, non-incremental. Check the SLO calculations guide for more details about incremental metrics.

If your SLI data is non-incremental, remember to set the Data Count Method to non-incremental in the UI (screenshot below), or set the value to incremental: false in your YAML definition in sloctl. Here's a YAML definition for an incremental metric in Prometheus:

[...]
objectives:
- target: 0.75
countMetrics:
good:
amazonPrometheus:
promql: sum(production_http_response_time_seconds_hist_bucket{method=~"GET|POST",status=~"2..|3..",le="1"})
incremental: true
total:
amazonPrometheus:
promql: sum(production_http_response_time_seconds_hist_bucket{method=~"GET|POST",le="+Inf"})
displayName: available1
timeSliceTarget: 0.75
value: 1

Here's how you can set a ratio metric to the incremental/non-incremental type in the UI:

select-data-source-and-metric
Image 1: Choose incremental, non-incremental ratio metric type in the UI

The incremental flag impacts how SLO calculations should be processed. This flag is meant to be set to true for those SLOs whose queries provide Nobl9 with incremental data. By incremental data, we mean: value v that for each point in time, t, is always greater than or equal to previous value (which is a linearly increasing function):

v(t) โ‰ค v(t+1)

Specific Prometheus Queries Can Impact SLO Calculationsโ€‹

Nobl9 queries to Prometheus canโ€™t contain the following functions:

These three types of functions extrapolate missing data, so if different timestamps are missing each time the Nobl9 Agent queries the data source, itโ€™s possible to receive inconsistent data.

Any function with a range vector as a parameter (such as rate, increase, irate, and more) is potentially vulnerable to another type of issue since Nobl9 asks for data with a given granularity (15 sec for Prometheus).

Range vector queries introduce an interval (in PromQL, itโ€™s represented by [x] where x is the duration, such as 5m), and it's hard to match the data intake interval with the aggregation function interval - those two might not overlap, and so the data will be unpredictable.

note

For query issues related to other sources, check specific Sources documentation.

Troubleshooting Complete: Reimport your Historical SLI Dataโ€‹

Once youโ€™ve resolved your issues, we advise running Replay for your SLO to refill it with historical SLI data for the period when your SLO wasnโ€™t collecting any data or was collecting data only partially.

Refer to the Replay documentation for more details.