Skip to main content

SLO annotations

Reading time: 0 minute(s) (0 words)

When you experience any outstanding event related to your services, it can be helpful to add a note using the user annotations about what happened and have that live alongside your metrics. You can also leverage annotations to record and track recent or upcoming deployments and hotfixes.

Thanks to system annotations that are added each time to the SLO objective charts when an alert is triggered or resolved, you can find answers to the following questions more easily:

  • Why was an alert triggered?
  • How does the triggered alert correlate with the information on the SLO charts?
  • What are the next steps that should be taken?

Limitations​

The Nobl9 SLO annotations service adheres to the following criteria:

  • Annotations are stored for one year.
  • The maximum number of characters per annotation is 1000.

Overview​

Annotations can be added using the Nobl9 UI, sloctl, or through the Nobl9 API.

As you can create annotations for a specific SLO objective via sloctl or Nobl9 API, annotations (both, created by the user or system annotations added by default) attached to a specific objective are displayed only on the charts related to that objective:

  • If you select all of the underlying SLO objectives to be displayed on the charts in the SLO Details tab, the SLI chart will display every annotation created for all the objectives and the SLO itself.
  • If you select only one objective to be displayed on the SLI chart, only the annotations created for that particular objective will be displayed.

System annotations​

System annotations are added to the SLO objective charts each time an alert is triggered or resolved - they are created by default and are not related to any specific user.

System annotations regarding alerts are created based on metric timestamp. Those annotations represent the time when an SLI indicates that alert was triggered or resolved. For the delayed metrics, Nobl9 will draw annotation in the past - note that it is not related to time when you receive a notification.

note

System annotations are displayed regardless of whether an Alert policy is silenced. That is, even if you silence an Alert policy for a given SLO, Nobl9 will create the annotation for the silenced alert and display it on the SLO charts.

RBAC​

The following are the RBAC limitations for system annotations:

  • You can't edit system annotations.
  • Only Organization Admins can delete system annotations, for example, when a false alert was triggered.
  • Users with other permissions can only view system annotations.

Managing system annotations in the UI​

You can choose to show or hide system or user annotations in the SLO details. Refer to the Managing system annotations section for detailed instructions.

User annotations​

When Nobl9 users experience any outstanding event related to their services, it can be helpful for them to add a note about what happened and have that live alongside their metrics. This information is crucial when performing historical analyses of product reliability. It can help answer the following questions:

  • What caused my error budget to burn so quickly?
  • Why did my SLI spike?
  • When and how long was my Service down?

Nobl9 users can also leverage annotations to record and track recent or upcoming deployments and hotfixes. Such annotations, alongside your SLI metrics, can help investigate the following questions:

  • We just deployed a new release; how can I track when the deployment occurred?
  • Was the spike in my SLI caused by a hotfix (scheduled or unscheduled)?”

Adding annotations in the UI​

Refer to the Managing annotations in the UI section for detailed instructions.

Managing annotations in sloctl​

The following sloctl commands are available to use in the SLO annotations service:

caution

sloctl does not return system annotations. When applying the sloctl get annotation[s] command, sloctl will return only annotations created by the user.

Applying annotations​

You can add or update annotations using the sloctl apply -f {yamlFile} command.

The following YAML example shows how to add an annotation to your SLO:

apiVersion: n9/v1alpha
kind: Annotation
metadata:
  name: annotation-name
  project: default
spec:
  slo: test-slo
  description: test annotation description
  objectiveName: objective-1
  startTime: 2021-11-02T17:10:05Z
  endTime: 2021-11-22T17:15:05Z

Notes on fields specific to Annotations:

FieldTypeDescription
metadata.name
mandatory
stringA unique annotation name, required for distinguishing project annotations.
spec.slo
mandatory
stringThe name of the SLO the annotation applies to.
spec.description
optional
stringA string (plain text) describing the annotation. The maximum number of characters is 1000.
spec.startTime
mandatory
stringDefines the date-time point where the annotation will be placed in the graph. The value must be in the YYYY-MM-DDTh:mm:ssZ format that complies with ISO8601. If startTime == endTime, the annotation will be placed at a single time point.
spec.endTime
mandatory
stringDefines the date-time point where the annotation will be placed in the graph. The value must be in the YYYY-MM-DDTh:mm:ssZ format that complies with ISO8601. If startTime == endTime, the annotation will be placed at a single time point.
spec.objectiveName
optional
stringOptional field allowing you to add an annotation to a particular, already existing SLO objective. If not specified, the annotation is attached to the SLO.
note

By default, sloctl always returns time in UTC. You can adjust this for your time zone by replacing the Z (for Zulu; a shorthand for UTC) with the offset from UTC, prefaced with either + or -. For instance, to have sloctl return times adjusted for Eastern Standard Time you would use startTime: 2021-11-02T17:10:05-05:00.

Deleting annotations​

To delete an annotation, run the following command: sloctl delete -f {yamlFile}.

Validation errors​

The following are possible errors you may run into while applying annotations through sloctl:

  • If the specified SLO does not exist, as in this example:

    apiVersion: n9/v1alpha
    kind: Annotation
    metadata:
      name: annotation-name
      project: my-project
    spec:
      slo: bad-slo-name
      description: test annotation description
      startTime: 2006-01-02T17:10:05Z
      endTime: 2006-01-02T17:15:05Z

    sloctl returns the following error:

    applying Annotation annotation-test-1 in 'default' project failed, because object SLO 'bad-slo-name' referenced in its spec does not exist in 'my-project' project

  • If the startTime or endTime format is invalid, as shown here:

    apiVersion: n9/v1alpha
    kind: Annotation
    metadata:
      name: annotation-name
      project: my-project
    spec:
      slo: bad-slo-name
      description: test annotation description
      startTime: 2006-01-02T17:10:05Z
      endTime: 2006-01-02T15:04:05Z07:00

    sloctl returns the following error:

    'Annotation.spec.startTime' Error:Field validation for 'startTime' failed on the 'iso8601dateTimeFormatRequired' tag

  • If the annotation name is blank:

    apiVersion: n9/v1alpha
    kind: Annotation
    metadata:
      name:
      project: my-project
    spec:
      slo: bad-slo-name
      description: test annotation description
      startTime: 2006-01-02T17:10:05Z
      endTime: 2006-01-02T15:04:05Z07:00

    sloctl returns the following error:

    Annotation.ObjectHeader.MetadataHolder.metadata.name' Error:Field validation for 'name' failed on the 'required' tag

  • If the endTime is before the startTime:

    apiVersion: n9/v1alpha
    kind: Annotation
    metadata:
      name:
      project: my-project
    spec:
      slo: bad-slo-name
      description: test annotation description
      startTime: 2020-01-02T17:10:05Z
      endTime: 2019-01-02T15:04:05Z07:00

    sloctl returns the following error:

    'Annotation.spec.endTime' Error:Field validation for 'endTime' failed on the 'endTimeBeforeStartTime' tag

  • If the required description is missing:

    apiVersion: n9/v1alpha
    kind: Annotation
    metadata:
      name:
      project: my-project
    spec:
      slo: bad-slo-name
      description: test annotation description
      startTime: 2021-01-02T17:10:05Z
      endTime: 2021-01-02T15:04:05Z07:00

    sloctl returns the following error:

    'Annotation.spec.description' Error:Field validation for 'description' failed on the 'required' tag

  • If the value in the description field is too long (>1000 characters), sloctl returns the following error:

    'Annotation.spec.description' Error:Field validation for 'description' failed on the 'max' tag

Last updated on
On this page