Budget adjustments

Beta

Reading time: 0 minute(s) (0 words)

Error budgets are invaluable for visualizing trends and understanding your services' performance. However, real-world data isn't always clean or fully representative. Outliers and specific events—such as holidays, planned maintenance, or one-time occurrences—can distort error budget calculations and obscure meaningful insights.

This is where budget adjustments come in. With this feature, you can define exclusions to shield your error budget from events like scheduled downtime or deployments. By applying ad-hoc adjustments or scheduling cyclical events with flexible recurrence rules, you can ensure that these periods do not throw your SLOs off track.

With budget adjustments, you can:

Shield your error budget

Protect your SLOs from expected downtime, such as maintenance or deployments.

Customize your budget adjustment needs

Tailor exclusions to specific needs using flexible scheduling options.

Control your error budget

Exclude past or future periods where SLO compliance isn't required.

Budget adjustments refine your error budget calculations to account for practical realities, helping you maintain accurate insights, focus on long-term trends, and make informed decisions without being misled by temporary fluctuations.

beta feature

Budget adjustment is a beta functionality. Currently, you can only apply it using sloctl, adjustments API or the Nobl9 Terraform provider.

RBAC

Role	Budget adjustments	Permission scope
Organization admin	View Apply Delete	Global
Project owner	View Apply Delete	Owned project All involved SLOs must be within owned projects. Otherwise, the action on a budget adjustment fails

Read about actions applicable to budget adjustments

Limitations

When using budget adjustments, the following limits are in place:

• A single budget adjustment definition can include up to 30 SLOs.
• Each SLO allows modification for up to 30 unique events within one action.
• Past budget adjustments can be created up to 30 days in the past. This same restriction applies when modifying or deleting existing adjustment events.
• Past budget adjustment events cannot be modified for composite SLOs.

How budget adjustments work

Budget adjustments allow you to take control of your error budget by excluding specific events and ensuring more reliable metrics about the performance of your services:

Identify events

Pinpoint specific timeframes or incidents that skew your data.

Apply adjustments

Apply adjustments to the relevant periods in historical data. Use sloctl or adjustments API to apply the budget adjustments to specific SLOs.

Recalculate budgets

Nobl9 automatically adjusts the chart’s values to account for the exclusion, recalculating Error budget and Reliability burn down by not taking Service level indicator values from the adjusted period into account.

In a nutshell

Budget adjustments in Nobl9 rely on two key components:

• Budget adjustment definition
• Adjustment event

See below for their differences and how they're connected.

Budget adjustment definition

A budget adjustment definition refers to kind: BudgetAdjustment, specified in the YAML format and managed with sloctl or the Nobl9 Terraform provider. The definition establishes parameters for how budget adjustments are applied to SLO(s). Specifically, it outlines:

• Time period: Specifies when the adjustment is active, including start and end times.
• Recurrence: Determines whether the adjustment is a one-time event or repeats over time.
• Target SLOs: Lists the specific SLOs to which the adjustment applies (the adjustment applies to all objectives of the specified SLO).

A definition of a budget adjustment serves as the source for generating adjustment events. In other words, the budget adjustment definition passed in YAML has a one-to-many relationship with its associated events, meaning a single definition can result in multiple adjustment events.

Budget adjustment event

A budget adjustment event is a single occurrence of an adjustment triggered for a particular SLO based on the parameters defined in a budget adjustment definition.

Adjustment events are not self-standing and can't exist independently of the definition. They are always derived from a budget adjustment definition. The defined adjustment events are applied to all objectives within one SLO during the specified periods.

Adjusting SLO error budget

To adjust an SLO's error budget, create a budget adjustment definition.

You can create a one-time budget adjustment definition or recurring.

Recurring budget adjustments limitations

Creating recurring budget adjustment definitions that start in the past is not possible.

To create an adjustment definition, apply the YAML configuration with sloctl. Your YAML configuration must follow the format provided below:

General YAML

General YAML definition for kind: BudgetAdjustment

apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: string # Mandatory
  displayName: string # Optional
spec:
  description: string # Optional
  firstEventStart: YYYY-MM-DDThh:mm:ssZ # Mandatory, defined start date-time point
  duration: 1h
  rrule: FREQ=DAILY;INTERVAL=1 # Optional
  filters:
    slos:
      - name: string # Mandatory
        project: string # Mandatory

Working YAML

Working YAML sample for kind: BudgetAdjustment

apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: monthly-deployment-adjustment
  displayName: Monthly deployment adjustment
spec:
  description: Adjustment for deployment happening monthly on the first Tuesday of each month for 1 hour
  firstEventStart: 2024-01-01T12:00:00Z
  duration: 1h
  rrule: FREQ=MONTHLY;INTERVAL=1;BYDAY=1TU
  filters:
    slos:
      - name: api-server-latency
        project: default
      - name: api-server-uptime
        project: default
      - name: proxy-throughput
        project: proxy

Field	Type	Description
spec.firstEventStart mandatory	string	Scheduled start time for the first adjustment event. The expected value must be a string representing the date and time in the RFC3339 format. firstEventStart is equivalent to to the DSTART property in iCal. Constraints: • firstEventStart can be at most 30 days in the past.
spec.duration mandatory	string	The duration of the budget adjustment event. Constraints: • The expected value for this field is a string formatted as a time duration • The duration must be at least 1 minute • The duration must be defined with a precision of 1 second • Example: 1h10m30s
spec.rrule optional	string	The iCalendar recurrence rule for the budget adjustment event. Constraints: • The expected value is a string in the iCal RRULE format • Example: FREQ=MONTHLY;BYMONTHDAY=1 • Use RRULE calculator to create the desired recurrence rule • rrule can't be applied to past events.
spec.filters.slos[] mandatory	string	A list of SLOs that will be attached to the budget adjustment event. spec.filters.slos[n].name and spec.filters.slos[n].project are mandatory for each list item.

Types of budget adjustment events

Depending on SLO's last timestamp, the following budget adjustment types are available:

Let's assume that the last data point was received around Apr 24, 09:25:

Past adjustment events

Refer to budget adjustment events completed before Apr 24, 09:25 (a and b in the image below). Any changes to these events must be handled individually through the adjustments API.

Ongoing adjustment events

Currently active budget adjustment events that started before Apr 24, 09:25 (c in the image below).

Any changes made to their definition may shorten or extend their duration. The shortest possible duration is limited to the time already processed for this ongoing adjustment event. Any duration changes don't affect an ongoing adjustment event's start time.

Future adjustment events

Future budget adjustments will start after Apr 24, 09:25 (d in the image below). All definition changes affect future adjustments.

Image 4: An SLO with past (a, b), ongoing (c), and future (d) adjustment events

Actions applicable to budget adjustments

Select a method type from the drop-down menu to see how you can manage your budget adjustment events and definitions:

Create

Create

Read

Update

Delete

Period

As a Nobl9 user I want to

Type

sloctl

API

Terraform

Additional info

Future

Exclude a future adjustment event.

Definition

- - -

Future

Exclude multiple recurring future adjustment events.

Definition

- - -

Past

Exclude a past event.

Definition

- - -

Past

Exclude recurring past adjustment events (e.g., previous two weekends).

Event

Workaround: To exclude a recurring past period, e.g. the previous two weekends, user should create two separate budget adjustment definitions, one for each weekend.

Editing budget adjustment definitions

• Editing recurring budget adjustment definitions only impacts future events. Any events already triggered by an edited definition remain without changes.
• Editing one-time budget adjustment definitions has the following restrictions:
  • Completed definitions are read-only.
  • Ongoing definitions allow you to edit display names, descriptions, and duration, provided the updated event ends in the future.
  • Not started definitions require the firstEventStart to be in the future.

editing past budget adjustment events

You can edit past budget adjustment events if they occurred within the last 30 days.
This action recalculates the SLO's error budget, which can distort how your system's reliability is reported. Use it with caution.

Recurrence rule format

Using spec.rrule you can create one-time adjustments for ad hoc needs or define a rule for predictable events that happen regularly. The spec.rrule field follows iCalendar specification.

The format of the rrule field consists of key-value pairs separated by semicolons (;). Each key-value pair specifies a parameter of the recurrence rule. Nobl9 supports all iCalendar rules outlied in the iCalendar documentation.

Example:

Budget adjustment will repeat every three days for a total of 10 occurrences.

apiVersion: n9/v1alpha
kind: BudgetAdjustment
...
spec:
  rrule: FREQ=DAILY;INTERVAL=3;COUNT=10

tip

Use the rrule generator to create a recurrence rule suited to your needs.

Deeper dive

Want to know more?

The FREQ value in the rrule definition specifies the frequency of the adjustment event. The value can be one of the following:

• HOURLY
• DAILY
• WEEKLY
• MONTHLY
• YEARLY

The INTERVAL value specifies the interval between each recurrence. The value is an integer representing the number of units of the frequency type. For example, if FREQ=DAILY and INTERVAL=2, the event occurs every two days.

You can also include additional parameters such as BYDAY, BYMONTH, BYSETPS, for more complex recurrence patterns.

User experience

Impact on SLI data

Budget adjustment events don't affect SLI data. When the budget adjustment is active, Nobl9 collects data points and displays them on the SLI chart.

You can see the budget adjustment event on the SLI chart, marked by an annotation with the icon. When you hover over the budget adjustment area, you can see the collected data points:

Image 1: SLI chart with a marked budget adjustment event

When you hover over the Reliability burn down and the Error budget burn rate charts, you can see data gaps in the budget adjustment event's area:

Image 2: Reliability burn down chart marked with adjustment event

Image 3: Burn rate chart marked with adjustment event

Adjustments and Replay

When working with adjustments and Replay processes, note that only one calculation can be performed per SLO at a time. Any new requests involving the same SLO will be queued and executed sequentially:

1. Replay and adjustment conflict:

   • If a Replay is running for SLO X and the user creates an adjustment for the same SLO X, the adjustment will be queued and will only begin once the replay is complete.

2. Adjustment and Replay conflict:

   • If an adjustment is running for SLO X and the user initiates Replay for the same SLO X, the replay will be queued and will only start after the adjustment is complete.

3. Multiple adjustments conflict:

   • If an adjustment is already running for SLO X and another adjustment is created for the same SLO X, the second adjustment will be queued and will only begin after the first adjustment is finished.

This ensures calculations are processed in the correct order without conflicts or data inconsistencies.

Managing adjustments for SLOs

To maintain accurate SLO tracking, you may need to exclude certain events or recurring time windows from error budget calculations. These short use cases show how to set up recurring adjustment definitions, manage past adjustments events, and create adjustments for historical events.

Setting up recurring adjustments for known downtime periods

In cases where downtime is predictable, such as routine maintenance or regular inactive hours, you can define a recurring adjustment to automatically exclude these periods from error budget calculations. This feature allows users to set up a schedule that repeats weekly, monthly, or at custom intervals, preventing the need to create new adjustments manually each time.

Let’s say that the service undergoes routine maintenance every Saturday from 2 a.m. to 4 a.m., during which it is temporarily taken offline. The adjustment definition for SLOs monitoring this service could look like this:

Sample 2-hour budget adjustment event happening weekly on Saturdays

apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: maintenance-budget-adjustment
  displayName: Maintenance budget adjustment
spec:
  description: Budget adjustment event happening weekly on the Saturday for 2 hours.
  firstEventStart: 2024-01-01T00:00:00Z
  duration: 2h
  rrule: FREQ=WEEKLY;INTERVAL=1;BYDAY=SA
  filters:
    slos:
      - name: latency-slo
        project: project-alpha
      - name: uptime-slo
        project: project-alpha
      - name: throughput-slo
        project: project-alpha

Reviewing and modifying past adjustments for accuracy

When you need to modify a historical adjustment event, for example, in the case of any errors in the original exclusion setup or changes in actual downtime, you perform two common actions:

• Update a specific past adjustment event
• Delete an incorrect adjustment event

Review and update past adjustments process using adjustments API

Example scenario: During a routine maintenance window on Saturday, a regional outage extended the downtime beyond the scheduled period. Although the initial adjustment secured the planned maintenance time, it didn't cover the additional, unplanned downtime. After reviewing the historical adjustment event for that date, the team realized they needed to adjust the exclusion to capture the entire downtime period.

Access adjustment events history

The team sends the following GET request to the adjustments API:

GET past budget adjustments

  curl -XGET
  -H 'Organization: <organization_name>'
  -H 'Authorization: Bearer <token>'
  'http://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events?from=2024-01-01T00:00:00Z&to=2024-01-31T23:59:59Z'

The API returns the following response:

GET 200 response

export default [
  {
    "eventStart": "2024-01-06T00:00:00Z",
    "eventEnd": "2024-01-06T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-13T00:00:00Z",
    "eventEnd": "2024-01-13T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-20T00:00:00Z",
    "eventEnd": "2024-01-20T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-27T00:00:00Z",
    "eventEnd": "2024-01-27T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  }
]

Update the required adjustment event

Having identified the event that needs to be updated, the team sends the following PUT request to the adjustment API:

PUT budget adjustment (update)

  curl -XPUT -H 'Organization: <organization>'
  -H 'Authorization: Bearer <token>'
  -H "Content-type: application/json" -d '[
      {
          "eventStart": "2024-01-20T00:00:00Z",
          "eventEnd": "2024-01-20T02:00:00Z",
          "slos": [
              {
                  "project": "response-time",
                  "name": "latency-slo"
              },
              {
                  "project": "continuous-service",
                  "name": "uptime-slo"
              }
          ],
          "update": {
              "eventStart": "2024-01-20T00:00:00Z",
              "eventEnd": "2024-01-20T03:00:00Z"
          }
      }
  ]' 'https://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events/update'

You can also update past adjustment events with the sloctl budgetadjustments events command. For this:

1. Retrieve the list of events for the specified adjustment and related SLO:
   Run budgetadjustments events get --adjustment-name=maintenance-budget-adjustment.
2. Identify the event that needs to be updated.
3. Update the required adjustment:
   Run budgetadjustments events update, providing the updated values in a YAML file.

Refer to Adjustments use case for real-life examples managed through sloctl.

Delete an incorrect adjustment event using sloctl

Example scenario: On another Saturday, the maintenance was canceled because the service needed to remain fully operational due to high demand. Despite the cancellation, the adjustment was still applied as usual, which prevented the service degradation that occurred during this period from impacting the error budget. Reviewing the historical adjustment event for that date, the team realized they needed to remove the adjustment event.

Access adjustment events history

The team accesses adjustment events history by running the following command in sloctl:

sloctl budgetadjustments events get --adjustment-name=maintenance-budget-adjustment --from=2024-01-01T00:00:00Z --to=2024-01-31T23:59:59Z

Having run the command above, the team received the following response in sloctl:

Budget adjustments history sloctl response

- eventStart: 2024-01-06T00:00:00Z
  eventEnd: 2024-01-06T02:00:00Z
  slos:
    - project: response-time
      name: latency-slo
    - project: continuous-service
      name: uptime-slo
- eventStart: 2024-01-13T00:00:00Z
  eventEnd: 2024-01-13T02:00:00Z
  slos:
    - project: response-time
      name: latency-slo
    - project: continuous-service
      name: uptime-slo
- eventStart: 2024-01-20T00:00:00Z
  eventEnd: 2024-01-20T02:00:00Z
  slos:
    - project: response-time
      name: latency-slo
    - project: continuous-service
      name: uptime-slo
- eventStart: 2024-01-27T00:00:00Z
  eventEnd: 2024-01-27T02:00:00Z
  slos:
    - project: response-time
      name: latency-slo
    - project: continuous-service
      name: uptime-slo

Delete the required adjustment event

The team identifies that the following adjustment event must be deleted:

Delete a budget adjustment event

- eventStart: 2024-01-27T00:00:00Z
  eventEnd: 2024-01-27T02:00:00Z
  slos:
    - project: response-time
      name: latency-slo
    - project: continuous-service
      name: uptime-slo

And runs the following command in sloctl to delete it:

sloctl budgetadjustments events delete --adjustment-name=maintenance-budget-adjustment -f ./maintenance-event-to-delete.yaml

You can also delete a past budget adjustment event using adjustments API.

Retrospectively excluding a historical event

There may be cases where an event in the past should be excluded, but no adjustment definition was initially created. For example, an unanticipated maintenance period or a non-service-related incident (e.g., a regional outage) impacted the SLO. This feature allows users to create an adjustment definition after the event has occurred.

Example scenario: The team discovered that a misconfigured monitoring metric breaks an SLO, while the service was functioning correctly. Recognizing that the issue was due to inaccurate data beyond their control, they decided to retroactively exclude this event from the error budget:

Exclude past event from the error budget

apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: outage-budget-adjustment
  displayName: Outage budget adjustment
spec:
  description: Budget exclusion due to external outage incident.
  firstEventStart: 2024-01-04T10:00:00Z
  duration: 4h
  filters:
    slos:
      - name: latency-slo
        project: response-time

Useful links

For a more in-depth look, consult additional resources:

Global annotations listNobl9 featuresOverview of SLO calculationsguidesReliability and error budget calculationsguidesYAML guideYAMLnobl9_budget_adjustmentTerraform