Composite SLOs

Reading time: 0 minute(s) (0 words)

Nobl9 has introduced Composite SLOs, a concept that allows aggregating multiple SLO objectives within a single SLO to represent an overall reliability performance view of your service. SLOs can now capture an end-to-end user journey for a complex service.

tip

Check the Use Cases of Composite SLOs to see how Composite SLOs can be used in real-life scenarios.

Composite SLOs - Rules at a Glance

Like all other resources in Nobl9, Composite SLOs are grouped under Projects and Services:

• Each SLO you define can only be part of one Composite SLO.
• You can combine a set of objectives within a single SLO into a Composite SLO.
  • You can define up to 12 objectives for an SLO.
  • You can modify the query for each objective.

Composite SLOs and Annotations

You can add an annotation to a Composite SLO. The added annotation appears in all the charts for the Composite SLO and the objectives.

note

For more information on annotations, refer to the SLO Annotations | Nobl9 Documentation.

Composite SLOs and Error Budget Calculations

Composite SLOs have a new error budget calculation method called the Composite Error Budget. It aggregates the error budgets of all the objectives in the SLO. Here's a glance of some rules that underlie changes in the calculation methods:

• The Composite Error Budget burns if the error budget for any of the SLO objectives within the Composite SLO is burning.
  • For a Time Slice SLO, this is binary depending on whether it was a good or bad minute. If it was a bad minute for any individual objective, it’s considered a bad minute for the Composite SLO.
  • For an Occurrences SLO, the Composite Error Budget is calculated based on the error budget burn rate which measures how fast the error budget is being consumed following the greater than condition. The default burn rate value is 1x and can be customized.

tip

The target of a Composite SLO and its Objectives can be different, as such:

• single bad point for the Objective can burn the whole Composite SLO’s error budget.
• very high error budget burn of the SLO Objective is treated the same as any other burn as long as it matches the Composite SLO burn rate condition.
  

Image 1: Composite SLO and Error Budget Calculations

Let’s put these assumptions into an example:

Composite SLO has the Error Budget Burn value set as 2x. Consider the following:

• scenario A: SLO Objective’s Error Budget Burn value = 2x
• scenario B: SLO Objective’s Error Budget Burn value = 20x
  

Both scenarios, A and B, result in the exact same Composite SLO calculations. Composite SLO calculations are not affected by SLO Objective’s Error Budget Burn values. In this case, both Objectives burn the same amount of reliability but can exhaust Composite SLO’s error budget sooner.

For more details, check the section below

Tips for Configuring Composite SLOs

Occurrences Method

• Setting a high target for the standard objective will cause its burn rate to spike more into high values. The maximum burn rate value is an inverse of the budget.

  • For example, a budget of 10% (target: 90%) will have a maximum burn rate of 10x (1/0.1), while a budget of 3% (target: 97%) will have a maximum burn rate of 33.33x (1/0.03). This is true for both standard and composite objectives.

• By manipulating standard objective targets and composite’s burn rate threshold, you can choose how sensitive the composite is to the burning of its aggregated objective’s budget:

  Composite’s burn rate condition threshold	Standard objective target	How fast composite’s budget is burning
  Low	High	Very sensitive
  High	High	Sensitive
  Low	Low	Insensitive
  High	Low	Very Insensitive

Timeslices Method

• The Target of standard objectives doesn’t affect composite’s error budget.
  Calculations are based on whether individual minutes of standard objectives are good or bad, regardless of how much an individual bad minute would burn its objective’s budget.

  • Each bad minute of both composite or standard objective will lower reliability burn down by the same amount that is equal to:

    
    • For example: each bad minute in one week rolling SLO will lower reliability burn down by around 0.01% = 1/(7*24*60).

  • Note that the budget target doesn’t affect how fast reliability burn-down drops.
    In the case of timeslices, it matters how many of maximum minutes (number of minutes in time window) we allow to be bad. Following the previous example, a 90% target gives 1008 minutes in the budget 7*24*60*(1-90%), so 1 bad minute will burn around 0,1% of the budget.

  • The time window length is a major factor in how fast a budget will burn for both Composite and standard objectives.

Observations on Composite Budgets

Composite budgets will usually burn faster than individual aggregated standard objective budgets.

That is because a composite budget is being burnt when any of its aggregated budgets are being burned fast enough (occurrences) or have a bad minute (timeslices), so it will be burnt more often than any of its individual objectives.

The more standard objectives Composite aggregates, the higher the chance that at least one of them will burn the budget fast enough (occurrences) or have a bad minute (timeslices).

This means that Composites with a greater number of aggregated objectives will usually burn their budget faster than Composite with a smaller number of aggregated objectives.

tip

Composite will usually burn at least as fast as its fastest-burning aggregated objective.

To get a more actionable Composite SLO, set more relaxed parameters (i.e, budget target, threshold value, time slice target) on aggregated objectives than you normally would if they were individual SLOs.

SLO is actionable if it can warn you about the potential exhaustion of the error budget early enough so you can do something about it before it happens. Composite SLO that exhausts its budget almost immediately when any of its aggregated budgets burns some budget is not actionable.

Reliability Burn Down

Composite’s reliability burn down is not an average of reliability burn downs of its aggregated objectives.

This means that any intuitions you might have regarding an average will likely not be true regarding the Composite.

tip

The Composite’s reliability burn down can be higher or lower than any or all aggregated objectives.

Composite’s budget calculations require information from all its aggregated objectives.

• If data for any query of aggregated objectives is missing or delayed, then the composite’s budget calculation will also be missing or delayed.

• Composite’s reliability burn down chart data resolution will be equal or lower than aggregated objective with the lowest data resolution.

  tip

  Composing objectives with dense and sparse SLI metrics will give a budget that behaves as one calculated from sparse metrics.

Defining Composite SLOs

You can define your Composite SLOs through the Nobl9 UI or by applying a YAML file through sloctl.

Creating Composite SLOs in the UI

To create a Composite SLO in the Nobl9 UI:

1. Navigate to Service Level Objectives.
2. Click the button.
3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.
4. In step 2, select a Data Source for your SLO.
5. Select the type of Metric. The available types vary depending upon the data source used. Check the Getting Started Guide for more details.
6. Enter a Query for the metric you selected.
7. In step 3, define a Time Window for the SLO.
8. In step 4, specify the Error Budget Calculation Method and your Objective(s). You can define up to 12 objectives for an SLO:

• For the Occurrences method:

  • Enter SLO objective Display Name (optional).
    The Display Name allows you to enter a name with spaces.

  • Enter unique SLO objective Name.
    The Name is required and can only contain lowercase, alphanumeric characters, and dashes. For example: objective-one.
    You can’t edit the SLO objective Name once you’ve created the SLO.

    note

    During the SLO objective unique identifier grace period, you can update the name for existing SLOs only through sloctl, and you can only do that once. Refer here for more information.

  • Enter the Target (mandatory).
    Define the percentage of good events out of total events that you want to meet the target values for the SLI.

  • Enter the target Values (mandatory).
    Compare the values gathered from the metric source. For example, if Values less than 200 is the objective for a Good experience, values below 200 are considered good.

  • You can modify the query for each added objective.

• For the Time Slices method:

  • Enter SLO objective Display Name (optional).
    The Display Name allows you to enter a name with spaces.

  • Enter unique SLO objective Name.
    The Name is required and can only contain lowercase, alphanumeric characters, and dashes. For example: objective-one.
    You can’t edit the SLO objective Name once you’ve created the SLO.

    note

    During the SLO objective unique identifier grace period, you can update the name for existing SLOs only through sloctl, and you can only do that once. Refer here for more information.

  • Enter the Target (mandatory).
    Define the percentage of good events out of total events that you want to meet the target values for the SLI.

  • Enter the Time Slice Allowance (mandatory).
    Define the percentage of the time slice for which you want to meet that Target. This is used to determine if a time slice should be considered good or bad.

  • Enter the target Values (mandatory).
    Compare the values gathered from the metric source. For example, if Values less than 200 is the objective for a Good experience, values below 200 are considered good.

  • You can modify the query for each added objective.

For details on further steps, refer to the Getting Started Guide.

Creating Composite SLOs in sloctl

Refer to the YAML Guide | Nobl9 Documentation to see how you can define a Composite SLO in a YAML file.

Editing Composite SLOs

You can edit a Composite SLO by clicking the Edit button next to the Composite SLO in the Grid View.

• In step 4 of the SLO wizard, you can edit:

  • The Target for both the Occurrences and Time Slices error budget calculation methods
  • The Time Slice Allowance for the Time Slices error budget calculation method
  • The Values for both the Occurrences and Time Slices error budget calculation methods
  • The Query for each added Objective by clicking View Query and modifying it

• You can also add a new Objective to the Composite SLO. Once you’ve saved your changes, error budget calculations proceed with this objective included. Similarly, you can remove a defined Objective. Error budget calculations proceed without this objective after you’ve saved your changes. Calculations do not get reset.

• In step 5 of the SLO wizard, you can uncheck the Create Composite SLO checkbox. When you save your changes, all the previously added objectives become individual, regular SLOs (the Composite SLO no longer appears in the Nobl9 UI and no history is preserved). Similarly, you can edit a regular SLO and check the Create Composite SLO checkbox in this step of the wizard. When you save your changes, the SLO becomes a Composite SLO.

note

The changes you make to the Composite SLO are reflected within a few minutes. The Composite SLO then starts to show an updated burn down.

Viewing Composite SLOs in Grid View

Like standard SLOs, you can view your Composite SLOs in the Grid View in the Nobl9 UI. Here, you can see the Composite SLO’s Reliability Burn Down chart, remaining error budget, and such.
You can quickly identify them by their special icon and blue header:

Image 2: Composite SLO in Grid View

To open the Composite SLO's details view, click the name of the Composite SLO.

In the Composite tab, you can inspect the Reliability Burn Down and Error Budget Burn Rate charts.

Image 3: Composite SLO details view

In the Objectives tab, you can see a list of all underlying Objectives with their data (the SLI chart and Reliability Burn Down chart). This allows you to see how each of them contributes to burning the Composite SLO’s error budget.

Image 4: Composite SLO Objectives tab

Composite SLOs and Data Export

With the rollout of Composite SLOs, we've added a new column (composite) at the end of each CSV row. The new column indicates whether the row in the CSV file contains composite SLOs-related data.

When the value for the composite column is true, you can find the data for your composite SLOs in the following columns in the CSV file:

Column name	Data type	Description
budget_target	DOUBLE	Objective’s target value (percentage in its decimal form) or composite’s target value.
objective_value	DOUBLE	Objective value or composite SLO’s burn rate condition value.
objective_operator	STRING	An operator used with raw metrics or composite’s burn rate condition operator. One of the following values: lte - less than or equal lt - less than get - greater than equal gt - greater than
composite	BOOLEAN	Indicates whether the row contains composite-related data (true means a row contains composite data).

Useful Links

Composite SLO - Troubleshooting

Composite SLOs | Nobl9 Terraform Documentation

Replay and Composite SLOs