Composite SLOs guide

Reading time: 0 minute(s) (0 words)

Nobl9 has introduced composite SLOs, a concept that allows aggregating multiple SLO objectives within a single SLO to represent an overall reliability performance view of your service. SLOs can now capture an end-to-end user journey for a complex service.

tip

Check the Use cases of composite SLOs to see how composite SLOs can be used in real-life scenarios.

Composite SLOs - rules at a glance

Like all other resources in Nobl9, composite SLOs are grouped under projects and services:

• Each SLO you define can only be part of one composite SLO.
• You can combine a set of objectives within a single SLO into a composite SLO.
  • You can define up to 12 objectives for an SLO.
  • You can modify the query for each objective.

Some definitions

Composite SLO is a combination of multiple error budgets in a single SLO into one error budget that burns the budget if any of the component objectives are burning.

Composite SLO creates a new error budget calculation called "composite.” For the composite error budget burn, each objective contributes equally to this composite error budget.

The composite error budget is calculated minute by minute. In each minute, if any component objectives have a burn rate over the threshold you set, the minute is considered a bad minute for the composite.

Composite SLOs and annotations

You can add an annotation to a composite SLO. The added annotation appears in all the charts for the composite SLO and the objectives.

note

For more information on annotations, refer to the SLO annotations.

Composite SLOs and error budget calculations

Composite SLOs have a new error budget calculation method called the Composite error budget. It aggregates the error budgets of all the objectives in the SLO. Here's a glance at some rules that underlie changes in the calculation methods:

• The composite error budget burns if the error budget for any of the SLO objectives within the composite SLO is burning.
  • For a Time Slice SLO, this is binary, depending on whether it was a good or bad minute. If it was a bad minute for any individual objective, it’s considered a bad minute for the composite SLO.
  • For an Occurrences SLO, the composite error budget is calculated based on the error budget burn rate, which measures how fast the error budget is consumed following the greater than condition. The default burn rate value is 1x and can be customized.

tip

The target of a composite SLO and its objectives can be different, as such:

• A single bad point for the objective can burn the whole composite SLO’s error budget.
• A very high error budget burn of the SLO objective is treated the same as any other burn if it matches the composite SLO burn rate condition.
  

Image 1: Composite SLO and error budget calculations

Let’s put these assumptions into an example:

Composite SLO has the error budget Burn value set as 2x. Consider the following:

• scenario A: SLO objective’s error budget Burn value = 2x
• scenario B: SLO objective’s error budget Burn value = 20x
  

A and B scenarios result in the same composite SLO calculations. SLO objective’s error budget Burn values don't affect the composite SLO calculations. In this case, both objectives burn the same amount of reliability but can exhaust composite SLO’s error budget sooner.

For more details, check the section below

Tips for configuring composite SLOs

Occurrences method

• Setting a high target for the standard objective will cause its burn rate to spike more into high values. The maximum burn rate value is an inverse of the budget.

  • For example, a budget of 10% (target: 90%) will have a maximum burn rate of 10x (1/0.1), while a budget of 3% (target: 97%) will have a maximum burn rate of 33.33x (1/0.03). This is true for both standard and composite objectives.

• By manipulating standard objective targets and the composite’s burn rate threshold, you can choose how sensitive the composite is to the burning of its aggregated objective’s budget:

  Composite’s burn rate condition threshold	Standard objective target	How fast composite’s budget is burning
  Low	High	Very sensitive
  High	High	Sensitive
  Low	Low	Insensitive
  High	Low	Very Insensitive

Timeslices method

• The target of standard objectives doesn’t affect the composite’s error budget_.
  Calculations are based on whether individual minutes of standard objectives are good or bad, regardless of how much a bad minute would burn its objective’s budget.

  • Each bad minute of both composite or standard objective will lower reliability burn down by the same amount that is equal to:
  

  • For example, each bad minute in one week rolling SLO will lower reliability burn down by around 0.01% = 1/(7*24*60).

  • Note that the budget target doesn’t affect how fast reliability burn-down drops.
    In the case of timeslices, it matters how many maximum minutes (number of minutes in time window) we allow to be bad. Following the previous example, a 90% target gives 1008 minutes in the budget 7*24*60*(1-90%), so one bad minute will burn around 0,1% of the budget.

• The time window's length significantly affects how fast a budget will burn for both composite and standard objectives.

Observations on composite budgets

Composite budgets will usually burn faster than individual aggregated standard objective budgets.

That is because a composite budget is burnt when any aggregated budgets are burnt fast enough (occurrences) or have a bad minute (timeslices); it will be burnt more often than any of its individual objectives.

The more standard objectives composite aggregates, the higher the chance that at least one will burn the budget fast enough (occurrences) or have a bad minute (timeslices).

This means that composites with more aggregated objectives will usually burn their budget faster than those with fewer.

tip

Composite will usually burn at least as fast as its fastest-burning aggregated objective.

To get a more actionable composite SLO, set more relaxed parameters (i.e., budget target, threshold value, time slice target) on aggregated objectives than you usually would if they were individual SLOs.

SLO is actionable if it can warn you about the potential exhaustion of the error budget early enough so you can do something about it before it happens. A composite SLO that exhausts its budget as soon as any of its constituent budgets are burnt isn't actionable.

Reliability burn down

Composite’s reliability burn down is not an average of reliability burn downs of its aggregated objectives.

This means that any intuitions you might have regarding an average will likely not be true regarding the composite.

tip

A composite’s reliability burn down can be higher or lower than any or all aggregated objectives.

Composite’s budget calculations require information from all its aggregated objectives.

• If data for any aggregated objective query is missing or delayed, then the composite’s budget calculation will also be missing or delayed.

• Composite’s reliability burn down chart data resolution will be equal to or lower than aggregated objective with the lowest data resolution.

tip

Composing objectives with dense and sparse SLI metrics will give a budget that behaves as one calculated from sparse metrics.

Uniqueness of threshold values

To enable uniqueness in error budget calculations in composite SLOs, Nobl9 uses the threshold value to identify specific objectives. You must set a unique target value across all objectives within one composite SLO.

If you want to use similar threshold values for different objectives in your composite SLO, we recommend differentiating them by setting varying decimal points for each objective. For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one, as in the example below:

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-composite-slo
    project: my-composite-slo
    ...
    objectives:
    - countMetrics:
      ...
      name: objective-1
      target: 0.95
      value: 1.0000001
    - countMetrics:
      ...
      name: objective-2
      target: 0.91
      value: 1.0000002

Defining composite SLOs

You can define your composite SLOs through the Nobl9 Web or by applying a YAML file through sloctl.

Creating composite SLOs on the Nobl9 Web

To create a composite SLO on the Nobl9 Web:

1. Navigate to Service Level Objectives.
2. Click the button.
3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.
4. In step 2, select a data source for your SLO.
5. Select the type of Metric. The available types vary depending upon the data source used. Check the Getting Started Guide for more details.
6. Enter a Query for the metric you selected.
7. In step 3, define a Time Window for the SLO.
8. In step 4, specify the Error Budget Calculation Method and your Objective(s). You can define up to 12 objectives for an SLO:

• For the Occurrences method:

  • Enter SLO objective Display Name (optional).
    The Display Name allows you to enter a name with spaces.

  • Enter a unique SLO objective Name.
    The Name is required and can only contain lowercase, alphanumeric characters, and dashes. For example: objective-one.
    You can’t edit the SLO objective Name once you’ve created the SLO.

    note

    During the SLO objective unique identifier grace period, you can update the name for existing SLOs only through sloctl, and you can only do that once. Refer here for more information.

  • Enter the Target (mandatory).
    Define the percentage of good events out of the total events you want to meet the target values for the SLI.

  • Enter the target Values (mandatory).
    Compare the values gathered from the metric source. For example, if Values less than 200 is the objective for a good experience, values below 200 are considered good.

  • You can modify the query for each added objective.

• For the Time Slices method:

  • Enter SLO objective Display Name (optional).
    The Display Name allows you to enter a name with spaces.

  • Enter a unique SLO objective Name.
    The Name is required and can only contain lowercase, alphanumeric characters, and dashes. For example: objective-one.
    You can’t edit the SLO objective Name once you’ve created the SLO.

    note

    During the SLO objective unique identifier grace period, you can update the name for existing SLOs only through sloctl, and you can only do that once. Refer here for more information.

  • Enter the Target (mandatory).
    Define the percentage of good events out of the total events you want to meet the target values for the SLI.

  • Enter the Time Slice Allowance (mandatory).
    Define the percentage of the time slice you want to meet that target. The allowance determines whether a time slice should be considered good or bad.

  • Enter the target Values (mandatory).
    Compare the values gathered from the metric source. For example, if Values less than 200 is the objective for a good experience, values below 200 are considered good.

  • You can modify the query for each added objective.

For details on further steps, refer to the Getting Started Guide.

Creating composite SLOs in sloctl

Refer to the YAML guide to see how you can define a composite SLO in a YAML file.

Editing composite SLOs

You can edit a composite SLO by clicking the Edit button next to the composite SLO in the grid view.

• In step 4 of the SLO wizard, you can edit:

  • The Target for both the Occurrences and Time Slices error budget calculation methods
  • The Time Slice Allowance for the Time Slices error budget calculation method
  • The Values for both the Occurrences and Time Slices error budget calculation methods
  • The Query for each added objective by clicking View Query and modifying it

• You can also add a new Objective to the composite SLO. Once you’ve saved your changes, error budget calculations proceed with this objective included. Similarly, you can remove a defined objective. Error budget calculations proceed without this objective after you’ve saved your changes. Calculations do not get reset.

• In step 5 of the SLO wizard, you can uncheck the Create Composite SLO checkbox. When you save your changes, all the previously added objectives become individual, standard SLOs (the composite SLO no longer appears on the Nobl9 Web and no history is preserved). Similarly, you can edit a standard SLO and check the Create Composite SLO checkbox in this wizard step. When you save your changes, the SLO becomes a composite SLO.

note

The changes you make to the composite SLO are reflected within a few minutes. The composite SLO then starts to show an updated burn down.

Viewing composite SLOs in grid view

Like standard SLOs, you can view your composite SLOs in the grid view on the Nobl9 Web. Here, you can see the composite SLO’s reliability burn down chart, remaining error budget, and such.
You can quickly identify them by their special icon and blue header:

Image 2: Composite SLO in grid view

To open the composite SLO's details view, click the name of the composite SLO.

You can inspect the Reliability Burn Down and Error Budget Burn Rate charts in the composite tab.

Image 3: Composite SLO details view

In the Objectives tab, you can see a list of all underlying objectives with their data (the SLI chart and reliability burn down chart). This lets you see how each contributes to burning the composite SLO’s error budget.

Image 4: Composite SLO Objectives tab

Composite SLOs and data export

With the rollout of composite SLOs, we've added a new column (composite) at the end of each CSV row. The new column indicates whether the row in the CSV file contains composite SLOs-related data.

When the value for the composite column is true, you can find the data for your composite SLOs in the following columns in the CSV file:

Column name	Data type	Description
budget_target	DOUBLE	Objective’s target value (percentage in its decimal form) or composite’s target value.
objective_value	DOUBLE	Objective value or composite SLO’s burn rate condition value.
objective_operator	STRING	An operator used with raw metrics or composite’s burn rate condition operator. One of the following values: lte - less than or equal lt - less than get - greater than equal gt - greater than
composite	BOOLEAN	Indicates whether the row contains composite-related data (true means a row contains composite data).

Useful links

Composite SLO - troubleshooting

Composite SLOs

Replay and composite SLOs