Sampling rate mismatch anomaly

Reading time: 0 minute(s) (0 words)

Sampling rate mismatch means Nobl9 sees good or bad samples at a different rate from total samples. For example, the good query might return four points per minute while total returns only one.

Applies to: Standard ratio SLOs with good/total or bad/total count metrics, incremental or non-incremental. Not supported for composite SLOs. Part of automatic anomaly detection in Enterprise Edition.

Availability

Sampling rate anomalies require enablement for your organization. Contact Nobl9 Support to request access.

This compares sample timing, not event counts. Good and total values can differ, and their timestamps don't need to match exactly. Detection adds an annotation; it doesn't change your error budget.

Why this matters

Your good or bad query and your total query should describe the same activity. Different sampling rates can be a sign that the queries use different resolutions, one is omitting results, or one is falling behind.

Example

Both streams start at four samples per minute. At 09:00, total slows to one while good stays at four. At 11:00, total returns to four.

Good stays steady while total slows down

Every sample from 08:40 to 11:40 UTC, on one continuous timeline.

GoodTotal

Detection and recovery

Before detecting anomalies, both streams first need at least 15 minutes and 20 completed sample intervals by default (a warmup window). Nobl9 then compares their recent rates, using total as the reference.

A difference above 50% triggers an anomaly. It resolves when the difference stays within 20%. Estimates take time to adjust, so recovery isn't immediate when total speeds up again.

Unlike Sampling rate change, a lasting mismatch doesn't become normal. One stream stopping can trigger it, both streams stopping doesn't prove recovery. Sparse streams use the same logic, but take longer to learn and confirm a mismatch.

Late data can trigger this too. Nobl9 waits 15 minutes by default, then checks the samples available for that time. Longer delays can leave live monitoring with an incomplete picture, even if the delayed stream later fills in at matching timestamps on the SLI chart. The annotation records that earlier observation and isn't removed when the chart fills in. See how delayed samples are handled.

What to investigate

1. Compare query steps, aggregation, and downsampling for both streams.
2. Run both queries at the source. Check whether one omits empty series or returns partial results. A zero-valued point counts; an omitted point doesn't.
3. Check event logs for failures or delays affecting one query.

See detection details and defaults or API access.