Budget adjustments Beta
Error budgets are invaluable for visualizing trends and understanding your services' performance. However, real-world data isn't always clean or fully representative. Outliers and specific events—such as holidays, planned maintenance, or one-time occurrences—can distort error budget calculations and obscure meaningful insights.
This is where budget adjustments come in. With this feature, you can define exclusions to shield your error budget from events like scheduled downtime or deployments. By applying ad-hoc adjustments or scheduling cyclical events with flexible recurrence rules, you can ensure that these periods do not throw your SLOs off track.
With budget adjustments, you can:
Budget adjustments refine your error budget calculations to account for practical realities, helping you maintain accurate insights, focus on long-term trends, and make informed decisions without being misled by temporary fluctuations.
Budget adjustment is a beta functionality.
Currently, you can only apply it using sloctl, adjustments API or the Nobl9 Terraform provider.
RBAC
Only Organization admins can apply, update, and delete budget adjustments.
Limitations
When using budget adjustments, the following limits are in place:
- A single budget adjustment definition can include up to 30 SLOs.
- Each SLO allows modification for up to 30 unique events within one action.
- Past budget adjustments can be created up to 30 days in the past. This same restriction applies when modifying or deleting existing adjustment events.
How budget adjustments work
Budget adjustments allow you to take control of your error budget by excluding specific events and ensuring more reliable metrics about the performance of your services:
In a nutshell
Budget adjustments in Nobl9 rely on two key components:
See below for their differences and how they're connected.
Budget adjustment definition
A budget adjustment definition refers to kind: BudgetAdjustment,
specified in the YAML format and managed with sloctl or the Nobl9 Terraform provider.
The definition establishes parameters for how budget adjustments are applied to SLO(s).
Specifically, it outlines:
- Time period: Specifies when the adjustment is active, including start and end times.
- Recurrence: Determines whether the adjustment is a one-time event or repeats over time.
- Target SLOs: Lists the specific SLOs to which the adjustment applies (the adjustment applies to all objectives of the specified SLO).
A definition of a budget adjustment serves as the source for generating adjustment events. In other words, the budget adjustment definition passed in YAML has a one-to-many relationship with its associated events, meaning a single definition can result in multiple adjustment events.
Budget adjustment event
A budget adjustment event is a single occurrence of an adjustment triggered for a particular SLO based on the parameters defined in a budget adjustment definition.
Adjustment events are not self-standing and can't exist independently of the definition. They are always derived from a budget adjustment definition. The defined adjustment events are applied to all objectives within one SLO during the specified periods.
Adding budget adjustments
To add budget a adjustment event, apply a YAML definition in sloctl:
- General YAML
- Working YAML
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
name: string # Mandatory
displayName: string # Optional
spec:
description: string # Optional
firstEventStart: YYYY-MM-DDThh:mm:ssZ # Mandatory, defined start date-time point
duration: 1h
rrule: FREQ=DAILY;INTERVAL=1 # Optional
filters:
slos:
- name: string # Mandatory
project: string # Mandatory
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
name: monthly-maintenance-adjustment
displayName: Monthly maintenance adjustment
spec:
description: Recurring budget adjustment for latency, uptime, and throughput SLOs occurring on the first Tuesday of every month for maintenance purposes.
firstEventStart: 2024-01-01T12:00:00Z
duration: 1h
rrule: FREQ=MONTHLY;INTERVAL=1;BYDAY=1TU
filters:
slos:
- name: latency-slo
project: response-time
- name: uptime-slo
project: continuous-service
- name: throughput-slo
project: scale-max
| Field | Type | Description |
|---|---|---|
spec.firstEventStart mandatory | string | Scheduled start time for the first adjustment event. The expected value must be a string representing the date and time in the RFC3339 format. firstEventStart is equivalent to to the DSTART property in iCal.Constraints: • firstEventStart can be at most 30 days in the past. |
spec.duration mandatory | string | The duration of the budget adjustment event. Constraints: • The expected value for this field is a string formatted as a time duration • The duration must be at least 1 minute • The duration must be defined with a precision of 1 second • Example: 1h10m30s |
spec.rrule optional | string | The iCalendar recurrence rule for the budget adjustment event. Constraints: • The expected value is a string in the iCal RRULE format • Example: FREQ=MONTHLY;BYMONTHDAY=1• Use RRULE calculator to create the desired recurrence rule • rrule can't be applied to past events. |
spec.filters.slos[] mandatory | string | A list of SLOs that will be attached to the budget adjustment event. spec.filters.slos[n].name and spec.filters.slos[n].project are mandatory for each list item. |
Applying BudgetAdjustment with firstEventStart in the past and defined rrule returns the following error:
Error: Validation failed: Cannot apply BudgetAdjustment. firstEventStart is in the past, and RRULE cannot be applied to past events.
Types of budget adjustment events
Depending on SLO's last timestamp, the following budget adjustment types are available:
Let's assume that the last data point was received around Apr 24, 09:25:
Apr 24, 09:25 (a and b in the image below). Any changes to these events must be handled individually through the adjustments API.Apr 24, 09:25 (c in the image below).Any changes made to their definition may shorten or extend their duration. The shortest possible duration is limited to the time already processed for this ongoing adjustment event. Any duration changes don't affect an ongoing adjustment event's start time.
Apr 24, 09:25 (d in the image below). All definition changes affect future adjustments.
Actions applicable to budget adjustments
When working with non-recurring past adjustment definitions, keep in mind the following:
Non-recurring adjustments trigger historical events with defined start and end dates. If an adjustment event’s start and end dates are both in the past, editing the associated adjustment definition is restricted to avoid unintentional modifications to historical data.
To modify or delete such events, use these sloctl commands:
- Update the event:
sloctl budgetadjustment event update - Delete the event:
sloctl budgetadjustment event delete
Recurrence rule format
Using spec.rrule you can create one-time adjustments for ad hoc needs or define a rule for predictable events that happen regularly. The spec.rrule field follows iCalendar specification.
The format of the rrule field consists of key-value pairs separated by semicolons (;). Each key-value pair specifies a parameter of the recurrence rule. Nobl9 supports all iCalendar rules outlied in the iCalendar documentation.
Example:
apiVersion: n9/v1alpha
kind: BudgetAdjustment
...
spec:
rrule: FREQ=DAILY;INTERVAL=3;COUNT=10
Use the rrule generator to create a recurrence rule suited to your needs.
The FREQ value in the rrule definition specifies the frequency of the adjustment event. The value can be one of the following:
HOURLYDAILYWEEKLYMONTHLYYEARLY
The INTERVAL value specifies the interval between each recurrence. The value is an integer representing the number of units of the frequency type. For example, if FREQ=DAILY and INTERVAL=2, the event occurs every two days.
You can also include additional parameters such as BYDAY, BYMONTH, BYSETPS, for more complex recurrence patterns.
User experience
Impact on SLI data
Budget adjustment events don't affect SLI data. When the budget adjustment is active, Nobl9 collects data points and displays them on the SLI chart.
You can see the budget adjustment event on the SLI chart, marked by an annotation with the 
icon. When you hover over the budget adjustment area, you can see the collected data points:
When you hover over the Reliability burn down and the Error budget burn rate charts, you can see data gaps in the budget adjustment event's area:
Adjustments and Replay
When working with adjustments and Replay processes, note that only one calculation can be performed per SLO at a time. Any new requests involving the same SLO will be queued and executed sequentially:
-
Replay and adjustment conflict:
- If a Replay is running for SLO X and the user creates an adjustment for the same SLO X, the adjustment will be queued and will only begin once the replay is complete.
-
Adjustment and Replay conflict:
- If an adjustment is running for SLO X and the user initiates Replay for the same SLO X, the replay will be queued and will only start after the adjustment is complete.
-
Multiple adjustments conflict:
- If an adjustment is already running for SLO X and another adjustment is created for the same SLO X, the second adjustment will be queued and will only begin after the first adjustment is finished.
This ensures calculations are processed in the correct order without conflicts or data inconsistencies.
Managing adjustments for SLOs
To maintain accurate SLO tracking, you may need to exclude certain events or recurring time windows from error budget calculations. These short use cases show how to set up recurring adjustment definitions, manage past adjustments events, and create adjustments for historical events.
Setting up recurring adjustments for known downtime periods
In cases where downtime is predictable, such as routine maintenance or regular inactive hours, you can define a recurring adjustment to automatically exclude these periods from error budget calculations. This feature allows users to set up a schedule that repeats weekly, monthly, or at custom intervals, preventing the need to create new adjustments manually each time.
Let’s say that the service undergoes routine maintenance every Saturday from 2 a.m. to 4 a.m., during which it is temporarily taken offline. The adjustment definition for SLOs monitoring this service could look like this:
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
name: maintenance-budget-adjustment
displayName: Maintenance budget adjustment
spec:
description: Budget adjustment event happening weekly on the Saturday for 2 hours.
firstEventStart: 2024-01-01T00:00:00Z
duration: 2h
rrule: FREQ=WEEKLY;INTERVAL=1;BYDAY=SA
filters:
slos:
- name: latency-slo
project: project-alpha
- name: uptime-slo
project: project-alpha
- name: throughput-slo
project: project-alpha
Reviewing and modifying past adjustments for accuracy
When you need to modify a historical adjustment event, for example, in the case of any errors in the original exclusion setup or changes in actual downtime, you perform two common actions:
- Update a specific past adjustment event
- Delete an incorrect adjustment event
Review and update past adjustments process using adjustments API
Example scenario: During a routine maintenance window on Saturday, a regional outage extended the downtime beyond the scheduled period. Although the initial adjustment secured the planned maintenance time, it didn't cover the additional, unplanned downtime. After reviewing the historical adjustment event for that date, the team realized they needed to adjust the exclusion to capture the entire downtime period.
Access adjustment events history
The team sends the following GET request to the adjustments API:
curl -XGET
-H 'Organization: <organization_name>'
-H 'Authorization: Bearer <token>'
'http://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events?from=2024-01-01T00:00:00Z&to=2024-01-31T23:59:59Z'
The API returns the following response:
export default [
{
"eventStart": "2024-01-06T00:00:00Z",
"eventEnd": "2024-01-06T02:00:00Z",
"slos": [
{
"project": "project-alpha",
"name": "latency-slo"
},
{
"project": "project-alpha",
"name": "uptime-slo"
}
]
},
{
"eventStart": "2024-01-13T00:00:00Z",
"eventEnd": "2024-01-13T02:00:00Z",
"slos": [
{
"project": "project-alpha",
"name": "latency-slo"
},
{
"project": "project-alpha",
"name": "uptime-slo"
}
]
},
{
"eventStart": "2024-01-20T00:00:00Z",
"eventEnd": "2024-01-20T02:00:00Z",
"slos": [
{
"project": "project-alpha",
"name": "latency-slo"
},
{
"project": "project-alpha",
"name": "uptime-slo"
}
]
},
{
"eventStart": "2024-01-27T00:00:00Z",
"eventEnd": "2024-01-27T02:00:00Z",
"slos": [
{
"project": "project-alpha",
"name": "latency-slo"
},
{
"project": "project-alpha",
"name": "uptime-slo"
}
]
}
]
Update the required adjustment event
Having identified the event that needs to be updated, the team sends the following PUT request to the adjustment API:
curl -XPUT -H 'Organization: <organization>'
-H 'Authorization: Bearer <token>'
-H "Content-type: application/json" -d '[
{
"eventStart": "2024-01-20T00:00:00Z",
"eventEnd": "2024-01-20T02:00:00Z",
"slos": [
{
"project": "response-time",
"name": "latency-slo"
},
{
"project": "continuous-service",
"name": "uptime-slo"
}
],
"update": {
"eventStart": "2024-01-20T00:00:00Z",
"eventEnd": "2024-01-20T03:00:00Z"
}
}
]' 'https://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events/update'
You can also update past adjustment events with the sloctl budgetadjustments events command. For this:
- Retrieve the list of events for the specified adjustment and related SLO:
Runbudgetadjustments events get --adjustment-name=maintenance-budget-adjustment. - Identify the event that needs to be updated.
- Update the required adjustment:
Runbudgetadjustments events update, providing the updated values in a YAML file.
Refer to Adjustments use case for real-life examples managed through sloctl.
Delete an incorrect adjustment event using sloctl
Example scenario: On another Saturday, the maintenance was canceled because the service needed to remain fully operational due to high demand. Despite the cancellation, the adjustment was still applied as usual, which prevented the service degradation that occurred during this period from impacting the error budget. Reviewing the historical adjustment event for that date, the team realized they needed to remove the adjustment event.
Access adjustment events history
The team accesses adjustment events history by running the following command in sloctl:
sloctl budgetadjustments events get --adjustment-name=maintenance-budget-adjustment --from=2024-01-01T00:00:00Z --to=2024-01-31T23:59:59Z
Having run the command above, the team received the following response in sloctl:
- eventStart: 2024-01-06T00:00:00Z
eventEnd: 2024-01-06T02:00:00Z
slos:
- project: response-time
name: latency-slo
- project: continuous-service
name: uptime-slo
- eventStart: 2024-01-13T00:00:00Z
eventEnd: 2024-01-13T02:00:00Z
slos:
- project: response-time
name: latency-slo
- project: continuous-service
name: uptime-slo
- eventStart: 2024-01-20T00:00:00Z
eventEnd: 2024-01-20T02:00:00Z
slos:
- project: response-time
name: latency-slo
- project: continuous-service
name: uptime-slo
- eventStart: 2024-01-27T00:00:00Z
eventEnd: 2024-01-27T02:00:00Z
slos:
- project: response-time
name: latency-slo
- project: continuous-service
name: uptime-slo
Delete the required adjustment event
The team identifies that the following adjustment event must be deleted:
- eventStart: 2024-01-27T00:00:00Z
eventEnd: 2024-01-27T02:00:00Z
slos:
- project: response-time
name: latency-slo
- project: continuous-service
name: uptime-slo
And runs the following command in sloctl to delete it:
sloctl budgetadjustments events delete --adjustment-name=maintenance-budget-adjustment -f ./maintenance-event-to-delete.yaml
You can also delete a past budget adjustment event using adjustments API.
Retrospectively excluding a historical event
There may be cases where an event in the past should be excluded, but no adjustment definition was initially created. For example, an unanticipated maintenance period or a non-service-related incident (e.g., a regional outage) impacted the SLO. This feature allows users to create an adjustment definition after the event has occurred.
Example scenario: The team discovered that a misconfigured monitoring metric breaks an SLO, while the service was functioning correctly. Recognizing that the issue was due to inaccurate data beyond their control, they decided to retroactively exclude this event from the error budget:
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
name: outage-budget-adjustment
displayName: Outage budget adjustment
spec:
description: Budget exclusion due to external outage incident.
firstEventStart: 2024-01-04T10:00:00Z
duration: 4h
filters:
slos:
- name: latency-slo
project: response-time
