Skip to main content

Using sloctl

Reading time: 0 minute(s) (0 words)

Using sloctl, you can perform bulk operations on budget adjustments and retrieve adjustment events for composite SLOs.

Adding budget adjustments​

To adjust an SLO's error budget, prepare a YAML definition and apply it with the sloctl apply command. Your budget adjustment definition must follow the template below:

General YAML definition for kind: BudgetAdjustment
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: string # Mandatory
  displayName: string # Optional
spec:
  description: string # Optional
  firstEventStart: YYYY-MM-DDThh:mm:ssZ # Mandatory, defined start date-time point
  duration: 1h
  rrule: FREQ=DAILY;INTERVAL=1 # Optional
  filters:
    slos:
      - name: string # Mandatory
        project: string # Mandatory
FieldTypeDescription
spec.firstEventStart
mandatory
stringScheduled start time for the first adjustment event. The expected value must be a string representing the date and time in the RFC3339 format. firstEventStart is equivalent to to the DSTART property in iCal.
Constraints:
• firstEventStart can be at most 30 days in the past.
spec.duration
mandatory
stringThe duration of the budget adjustment event.
Constraints:
• The expected value for this field is a string formatted as a time duration
• The duration must be at least 1 minute
• The duration must be defined with a precision of 1 second
• Example: 1h10m30s
spec.rrule
optional
stringThe iCalendar recurrence rule for the budget adjustment event.
Constraints:
• The expected value is a string in the iCal RRULE format
• Example: FREQ=MONTHLY;BYMONTHDAY=1
• Use RRULE calculator to create the desired recurrence rule
• rrule can't be applied to past events.
spec.filters.slos[]
mandatory
stringA list of SLOs that will be attached to the budget adjustment event. spec.filters.slos[n].name and spec.filters.slos[n].project are mandatory for each list item.

Recurrence rule format​

Using spec.rrule, you can create ad hoc one-time adjustments or define a rule for expected events that happen regularly. The spec.rrule field follows iCalendar specification.

The format of the rrule field consists of key-value pairs separated by semicolons (;). Each key-value pair specifies a parameter of the recurrence rule. Nobl9 supports all iCalendar rules outlied in the iCalendar documentation.

Example:

Budget adjustment will repeat every three days for a total of 10 occurrences.
apiVersion: n9/v1alpha
kind: BudgetAdjustment
...
spec:
  rrule: FREQ=DAILY;INTERVAL=3;COUNT=10
Need assistance?

The RRULE tool is available to help with your advanced rrule configuration and validation.

Deeper dive

  • The FREQ value in the rrule definition specifies the frequency of the adjustment event. The value can be one of the following:

    • HOURLY
    • DAILY
    • WEEKLY
    • MONTHLY
    • YEARLY

  • The INTERVAL value specifies the interval between each event. The value is an integer representing the number of frequency units. For example, if FREQ=DAILY and INTERVAL=2, the event occurs every other day.

  • You can also include additional parameters such as BYDAY, BYMONTH, BYSETPS, for more complex recurrence patterns.

  • Use the UNTIL and COUNT parameters configure when the recurrence must stop.

    • UNTIL—specify an end date and time
    • COUNT—set the maximum number of occurrences Examples:
    • FREQ=DAILY;INTERVAL=3;UNTIL=20251016T000000Z: Triggers adjustment events every three days until October 16, 2025. The timestamp is decoded as follows:
    YearMonthDayTimeHourMinSecUTC
    20251016T000000Z
    • FREQ=WEEKLY;INTERVAL=2;COUNT=10: Triggers adjustment events every two weeks for a total of 10 occurrences. Then, it becomes completed.

Use cases​

The short use cases in this section demonstrate how to set up recurring adjustment definitions, manage past adjustments events, and create adjustments for historical events.

Setting up recurring adjustment definitions​

In cases where downtime is predictable, such as routine maintenance or regular inactive hours, you can define a recurring adjustment to automatically exclude these periods from error budget calculations. This feature allows users to set up a schedule that repeats weekly, monthly, or at custom intervals, preventing the need to create new adjustments manually each time.

Let’s say that the service undergoes routine maintenance every Saturday from 2 a.m. to 4 a.m., during which it is temporarily taken offline. The adjustment definition for SLOs monitoring this service could look like this:

Sample 2-hour budget adjustment event happening weekly on Saturdays
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: maintenance-budget-adjustment
  displayName: Maintenance budget adjustment
spec:
  description: Budget adjustment event happening weekly on the Saturday for 2 hours.
  firstEventStart: 2024-01-01T00:00:00Z
  duration: 2h
  rrule: FREQ=WEEKLY;INTERVAL=1;BYDAY=SA
  filters:
    slos:
      - name: latency-slo
        project: project-alpha
      - name: uptime-slo
        project: project-alpha
      - name: throughput-slo
        project: project-alpha

Reviewing and modifying past adjustments​

When you need to modify a historical adjustment event, for example, in the case of any errors in the original exclusion setup or changes in actual downtime, you perform two common actions:

  • Update a specific past adjustment event
  • Delete an incorrect adjustment event

For any operations with the budget adjustment events, you need to retrieve the events list first.

To retrieve the list of past adjustment events, call the GET budget adjustments endpoint:

GET past budget adjustments
  curl -XGET
  -H 'Organization: <organization_name>'
  -H 'Authorization: Bearer <token>'
  'https://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events?from=2024-01-01T00:00:00Z&to=2024-01-31T23:59:59Z'

The following response is received:

GET past budget adjustments 200 response
export default [
  {
    "eventStart": "2024-01-06T00:00:00Z",
    "eventEnd": "2024-01-06T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-13T00:00:00Z",
    "eventEnd": "2024-01-13T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-20T00:00:00Z",
    "eventEnd": "2024-01-20T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  },
  {
    "eventStart": "2024-01-27T00:00:00Z",
    "eventEnd": "2024-01-27T02:00:00Z",
    "slos": [
      {
        "project": "project-alpha",
        "name": "latency-slo"
      },
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ]
  }
]

Update an adjustment event​

Example scenario: During a routine maintenance window on Saturday, a regional outage extended the downtime beyond the scheduled period. Although the initial adjustment secured the planned maintenance time, it didn't cover the additional, unplanned downtime. After reviewing the past adjustment event for that date, the team realized they needed to extend the exclusion, so it captures the entire downtime period.

To update the required adjustment event, call:

PUT budget adjustment (update)
  curl -XPUT -H 'Organization: <organization>'
  -H 'Authorization: Bearer <token>'
  -H "Content-type: application/json" -d '[
      {
          "eventStart": "2024-01-06T00:00:00Z",
          "eventEnd": "2024-01-06T02:00:00Z",
          "slos": [
              {
                  "project": "project-alpha",
                  "name": "latency-slo"
              }
          ],
          "update": {
              "eventStart": "2024-01-06T00:00:00Z",
              "eventEnd": "2024-01-06T01:30:00Z"
          }
      }
  ]' 'https://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events/update'

Delete an adjustment event​

Example scenario: On another Saturday, the maintenance was canceled because the service needed to remain fully operational due to high demand. Despite the cancellation, the adjustment was still applied as usual, which prevented the service degradation that occurred during this period from impacting the error budget. Reviewing the historical adjustment event for that date, the team realized they needed to remove the adjustment event.

To delete an adjustment event, call:

POST budget adjustment (delete)
curl -XPOST -H 'Organization: <organization>'
-H 'Authorization: Bearer <token>'
-H "Content-type: application/json" -d '[
    {
    "slos": [
      {
        "project": "project-alpha",
        "name": "uptime-slo"
      }
    ],
    "eventStart": "2024-01-20T00:00:00Z",
    "eventEnd": "2024-01-20T02:00:00Z"
    }
    ]' 'http://app.nobl9.com/api/budgetadjustments/v1/maintenance-budget-adjustment/events/delete'

Adjusting budget for a past period​

You may need to exclude a timeframe in the past, when no adjustment definition was initially created. For this, you can create a budget adjustment definition with a past date for event start and end.

Past budget adjustment limitations

You can create a one-time definition to adjust the error budget for a past period. Past budget adjustment definitions can't be recurring.

Example scenario: The team discovered that a misconfigured monitoring metric breaks an SLO, while the service was functioning correctly. Recognizing that the issue was due to inaccurate data beyond their control, they decided to retroactively exclude this event from the error budget:

Adjusting error budget for a past period
apiVersion: n9/v1alpha
kind: BudgetAdjustment
metadata:
  name: misconfigured-slo-adjustment
  displayName: Misconfigured SLO adjustment
spec:
  description: Budget adjustment for misconfigured SLO.
  firstEventStart: 2024-01-04T10:00:00Z
  duration: 4h
  filters:
    slos:
      - name: latency-slo
        project: response-time
Check out these related guides and references:
Global annotations listNobl9 featuresOverview of SLO calculationsguidesReliability and error budget calculationsguidesYAML guideYAMLnobl9_budget_adjustmentTerraform
Last updated on
On this page