Entire/Remaining Budget Would Be Exhausted

Reading time: 0 minute(s) (0 words)

Ensuring that your SLOs meet their targets is one of the critical pillars of delivering a reliable and high-performing service to your users. But this doesn’t mean there can be no issues or failures along the way.

The key is to be proactive and ready for any incident. A proactive approach involves setting a time buffer to mitigate potential issues and safeguard the budget from being completely depleted in the event of an incident. Nobl9 allows creating such alerting conditions, for example:

Alert me if my Entire / Remaining budget would be exhausted in 4h (and the condition lasts for 15m | based on the alerting window 15m)

4h in the example above represents the amount of time the budget will exhaust, meaning that the reliability of such an SLO will drop below its target.

Choose a value (4h in the example above) that is actionable and adequate to the Service it’s set for. A 12h exhaustion time would help detect a slow degradation of the reliability of a Service, whereas 2h would represent an urgent issue that must be fixed immediately.

Entire vs. Remaining Budget

Exhaustion conditions predict how long it will take to use up the error budget based on the current situation of an SLO.

Remaining Budget Variant

This variant lets you know if your service needs attention based on how much error budget you have left and how fast you are burning it. If your SLO has a positive error budget and it looks like it will deplete it in two hours (or whatever condition you have set), Nobl9 will send an Alert to notify you.

The more your remaining budget decreases, the more sensitive this condition becomes. When there is no remaining budget for an SLO, any level of burn is enough to activate the Alert, as long as it meets the lasts_for condition for all cases where it is utilized.

In the Reliability Burn Down charts displayed below, you can see the progress of three Objectives within a single SLO:

• california with a high amount of the remaining budget left.

• frankfurt, with no remaining budget left.

• ohio with a small amount of remaining budget left.

If we used the Remaining budget would be exhausted condition for this SLO, the first objective (california) would require the highest value of burn rate to trigger an Alert. The third objective (ohio) would exhaust faster, so the burn rate needed to trigger an Alert based on the same condition would be lower than the first objective. And if there’s any burn, the second objective (frankfurt) would trigger the alert because there’s no remaining budget left.

Image 1: Reliability Burn Down charts for the Remaining Budget condition

Entire Budget Variant

In this variant, the current value of the remaining budget doesn't affect the calculation logic. The exhaustion prediction is based on the entire error budget allocation and how fast you burn it. If an SLO reaches exhaustion within, for example, 4 hours, the budget would be exhausted from 100% to 0% in this time frame, resulting in a drop in reliability below your set target. This variant is useful if you want to receive Alerts for similar incidents regardless of the value of the remaining error budget.

The value used in this condition (timeToBurnEntireBudget) should be greater or equal to the Error Budget Allocation for an SLO using this condition. It’s because error budget allocation represents the time within which the budget would be exhausted in the most pessimistic scenario, where all events are bad (that is below the threshold raw metrics, no good events for count metrics). It’s not possible to burn the budget faster than that.

The image below shows Total Error Budget for Objectives. In this section, we see that Error Budget Allocation for objectives is as follows:

• ok: 7h 12m

• slow: 3h 36m

• poor: 43m 12s

If the value of the entire error budget exhaustion used in the alerting condition is (for example) 4 hours, the only two objectives that will possibly alert are poor and slow because it’s impossible to burn the ok objective in less than 7 hours, 12 minutes.

Image 2: Reliability Burn Down charts for the Entire Budget condition

keep in mind

• During exhaustion, the remaining budget decreases over time.
• During recovery (only for the rolling time windows), the remaining budget increase over time and caps at 100%.
• Using Total Error Budget Allocation as a value in this condition will help to catch the most pessimistic burn possible:

Remember that such configuration is specific to the objective, so it may be less suited if you want to re-use such alert policy for different SLO configurations.

Some Definitions – Exhaustion

Exhaustion refers to the gradual depletion of the error budget over time.

An error budget is exhausted when it has no remaining budget. If there is a remaining error budget, then the budget is not exhausted.

Any positive amount of burn rate means that the budget is currently being exhausted (the exhaustion process is happening).

Key Take-Aways

Not every exhaustion is harmful.
When your budget is exhausting error budget very slowly, it doesn’t mean you will burn it.

Exhaustion of the error budget is natural.
As long as it doesn’t lead to burning the entire error budget. It should not be considered a failure to deliver reliable services.