Sampling rate change anomaly

Reading time: 0 minute(s) (0 words)

Sampling rate change means Nobl9 sees your SLI reporting samples faster or slower than usual. A stream might drop from four samples per minute to one, even if its values barely change.

Applies to: Standard SLOs with raw or count metrics. Not supported for composite SLOs. Part of automatic anomaly detection in Enterprise Edition.

Availability

Sampling rate anomalies require enablement for your organization. Contact Nobl9 Support to request access.

This checks sample timing, not traffic volume. A point reporting zero requests still counts as a sample. Detection adds an annotation; it doesn't change your error budget.

Why this matters

How often you sample affects how much detail you can see. Less frequent sampling can make short-lived changes harder to spot, even when data continues to arrive.

Faster sampling isn't inherently a problem either. An unexpected change in either direction can reveal a query, collection, or aggregation change. This annotation helps you check whether that change was intentional, rather than mistaking a change in measurement for a change in your service.

Example

This stream starts at four samples per minute. At 09:00, it slows to one, then returns to four at 11:00. The points spread out and then get closer again, while the SLI values stay in the same range.

Sampling slows down, then returns to its original rate

Every sample from 08:40 to 11:40 UTC, on one continuous timeline.

SLI

Detection and recovery

Nobl9 learns each stream's usual spacing from its sample timestamps. Warmup requires at least 15 minutes and 20 completed intervals by default. Hourly streams take longer to learn; an empty minute isn't a problem by itself.

After warmup, a difference above 50% between the recent rate and the learned baseline triggers an anomaly if it lasts for two learned intervals. It resolves when the difference stays within 20% for two intervals.

The baseline keeps adapting. In this example, the first anomaly resolves while sampling stays at one per minute: the slower rate has become normal. Returning to four per minute triggers another change. Resolution means the pattern has settled, not that the collection settings are correct.

Long gaps can trigger this anomaly without another point arriving. Nobl9 waits 15 minutes for delayed data by default, but a longer delay can still leave live monitoring with an incomplete picture. Points arriving later may fill in the SLI chart, but the annotation stays as a record of that earlier observation. See how delayed samples are handled.

What to investigate

1. Check the affected stream in the annotation. Did its query step, collection interval, or downsampling change?
2. Run the query at the source. Look for missing samples or partial results.
3. Check event logs for timeouts, query failures, and rate limits.

See detection details and defaults or API access.