Entire/Remaining budget would be exhausted
Ensuring that your SLOs meet their targets is one of the critical pillars of delivering a reliable and high-performing service to your users. But this doesnβt mean there can be no issues or failures along the way.
The key is to be proactive and ready for any incident. A proactive approach involves setting a time buffer to mitigate potential issues and safeguard the budget from being completely depleted in the event of an incident. Nobl9 allows creating such alert conditions, for example:
in 4h based on the alerting window 15m
4h
in the example above represents the amount of time the budget will exhaust, meaning that the reliability of such an SLO will drop below its target.
Choose a value (4h
in the example above) that is actionable and adequate to the service itβs set for.
A 12h
exhaustion time would help detect a slow service reliability degradation,
whereas 2h
would represent an urgent issue that must be fixed immediately.
Entire vs. remaining budgetβ
Exhaustion conditions predict how long it will take to use up the error budget based on the current situation of an SLO.
Remaining budget variantβ
This variant lets you know if your service needs attention based on how much error budget you have left and how fast you are burning it. If your SLO has a positive error budget, and it looks like it will deplete it in two hours (or whatever condition you have set), Nobl9 will send an alert to notify you.
The more your remaining budget decreases, the more sensitive this condition becomes.
When there is no remaining budget for an SLO and you're using lastsFor
observation model, any level of burn is enough to activate the alert,
as long as it meets the lastsFor
condition for all cases where it is utilized.
In the Reliability Burn Down charts displayed below, you can see the progress of three objectives within a single SLO:
-
california
with a high amount of the remaining budget left. -
frankfurt
, with no remaining budget left. -
ohio
with a small amount of remaining budget left.
If we used the Remaining budget would be exhausted condition for this SLO, the first objective (california
) would require the highest value of burn rate to trigger an alert.
The third objective (ohio
) would exhaust faster, so the burn rate needed to trigger an Alert based on the same condition would be lower than the first objective.
And if thereβs any burn, the second objective (frankfurt
) would trigger the alert because thereβs no remaining budget left.
![remaining budget RBBs](/img/remaining-budget-2.png)
Entire budget variantβ
In this variant, the current value of the remaining budget doesn't affect the calculation logic. The exhaustion prediction is based on the entire error budget allocation and how fast you burn it. If an SLO reaches exhaustion within, for example, 4 hours, the budget would be exhausted from 100%
to 0%
in this time frame, resulting in reliability dropping below your set target.
This variant is useful if you want to receive alerts for similar incidents regardless of the value of the remaining error budget.
The value used in this condition (timeToBurnEntireBudget
) should be greater or equal to the error budget allocation for an SLO using this condition. Itβs because error budget allocation represents the time within which the budget would be exhausted in the most pessimistic scenario, where all events are bad (that is below the threshold raw metrics, no good events for count metrics). Itβs not possible to burn the budget faster than that.
The image below shows the total error budget for objectives. In this section, we see that error budget allocation for objectives is as follows:
-
ok
: 7h 12m -
slow
: 3h 36m -
poor
: 43m 12s
If the value of the entire error budget exhaustion used in the alert condition is (for example) four hours, the only two objectives that will possibly alert are poor
and slow
because itβs impossible to burn the ok
objective in less than seven hours, 12 minutes.
![entire budget RBBs](/img/entire-budget.png)
- During exhaustion, the remaining budget decreases over time.
- During recovery (only for the rolling time windows), the remaining budget increase over time and caps at 100%.
- Using the total error budget allocation as a value in this condition will help to catch the most pessimistic burn possible:
![total error budget](/img/total-error-budget.png)
Remember that such configuration is specific to the objective, so it may be less suited if you want to re-use such alert policy for different SLO configurations.
Which variant (Remaining / Entire) of exhaustion should I use?β
For general use, the Remaining budget would be exhausted in condition is more appropriate if you want to prevent your budget from being exhausted. It gives better precision for exhaustion as the prediction is more sensitive as your remaining budget decreases.
When you have no remaining budget left, and you want to keep being alerted using time predictions, use the Entire budget would be exhausted in condition. It uses the total error budget allocation as a value Nobl9 predicts the exhaustion from, no matter how much budget is left. It's useful when you want to receive alerts for similar time-related incidents regardless of the value of the remaining error budget.
When your SLO has no remaining budget left and uses the Remaining budget would be exhausted in condition with the lastsFor
variant, Nobl9 will trigger an alert for any amount of burn rate, no matter how small it is (i.e., even a single bad event below the budget would trigger the alert).
If you want to keep being alerted using time predictions, use the Entire budget would be exhausted in condition .
YAML configurationβ
The following YAML defines AlertPolicy
with a Remaining budget exhaustion condition:
apiVersion: n9/v1alpha
kind: AlertPolicy
metadata:
name: remaining-exhaustion
project: default
spec:
alertMethods: []
conditions:
- alertingWindow: 30m
measurement: timeToBurnBudget
value: 3h
op: lt
coolDown: 5m
description: "Policy that triggers when the SLO's remaining budget would burn in 3h"
severity: Medium
The following YAML defines AlertPolicy
with a Entire budget exhaustion condition:
apiVersion: n9/v1alpha
kind: AlertPolicy
metadata:
name: entire-exhaustion
project: default
spec:
alertMethods: []
conditions:
- alertingWindow: 30m
measurement: timeToBurnEntireBudget
value: 12h
op: lt
coolDown: 5m
description: "Policy that triggers when the SLO's entire budget allocation would burn in 12h, regardless of its current value"
severity: Medium
entire-exhaustion
can only trigger alerts for SLOs which have a total budget allocation values lesser or equal to 12h.
Check if the defined alert condition has the alertingWindow
attribute (for example, by checking its YAML configuration through the sloctl get alertpolicies [alert_policy_name]
). It is possible to create a similar alert policy, but with the lastsFor
value defined instead.
However, we recommend configuring the burn rate policy with the alertingWindow
parameter, allowing more control over the evaluation window and providing more precise calculations.
What's budget exhaustion?β
Exhaustion refers to the gradual depletion of the error budget over time.
An error budget is exhausted when it has no remaining budget. If there is a remaining error budget, then the budget is not exhausted.
Any positive amount of burn rate means that the budget is currently being exhausted (the exhaustion process is happening).
Key takeawaysβ
When your budget is exhausting error budget very slowly, it doesnβt mean you will burn it.
As long as it doesnβt lead to burning the entire error budget. It should not be considered a failure to deliver reliable services.