Silencing alerts

Reading time: 0 minute(s) (0 words)

SLOs with attached alert policies can get into a state where the user does not want to receive any alerts. For instance, this situation could be caused by a configuration issue that cannot be immediately resolved, which causes an alert to be triggered on and off. You can also have a known-bad SLO that won't recover for some time.

In such circumstances, Nobl9 allows you to silence the alerts that originate from a specific SLO for a predefined period. During the defined time, you won’t receive any alerts from the silenced alert policy for the SLO.

tip

Do you prefer silencing your alerts via Nobl9 UI? Go here for a comprehensive overview.

About AlertSilence

You can apply AlertSilence at an SLO level for the selected alert policy.

You can apply only one AlertSilence object per an SLO at the same time.

When an alert event assumes the resolved status within the duration of AlertSilence, Nobl9 will send you an all-clear notification. For more details, see Cooldown period | Nobl9 documentation.

You can manage AlertSilence both in the Nobl9 UI and in sloctl.

Managing AlertSilence in sloctl

Applying AlertSilence

You can add AlertSilence to an SLO using the sloctl apply -f {yamlFile} command.

The following YAML example shows how to add an AlertSilence to your SLO:

apiVersion: n9/v1alpha
kind: AlertSilence
metadata:
  name: alert-silence-name
  project: default
spec:
  description: AlertSilence description
  slo: slo-name # the name of the SLO AlertSilence will be applied to
  alertPolicy:
    name: alert-policy-name # the name of the AlertPolicy AlertSilence will be applied to
  period: # define period for the AlertSilence
    startTime: 2025-01-01T12:00:00+02:00 # define the start date-time point where the Alerts will be silenced
    endTime: 2025-01-01T13:00:00+02:00 # define the start date-time point where the Alerts will be silenced
    duration: 1h

Important notes:

• spec.alertPolicy - alertPolicy that AlertSilence will be applied to

  caution

  alertPolicy must be in the same project as AlertSilence and SLO

• spec.period.startTime, spec.period.endTime:

  • These values define the date-time point when the alerts will be silenced.
  • Both timestamps are inclusive, meaning alerts silence starts at startTime and stops at endTime.
  • They must be in the YYYY-MM-DDThh:mm:ssZ format that complies with ISO8601.

• spec.period.duration

  • This value defines the duration for which alerts will be silenced from the applying time.
  • Accepted units: s, m, h.

Configuring startTime, endTime, and duration

You can combine startTime, endTime, and duration as in the examples below:

• startTime and endTime - AlertPolicy will be silenced from the defined startTime time-point to the endTime time-point:

  apiVersion: n9/v1alpha
  kind: AlertSilence
  metadata:
    name: alert-silence-name
    project: default
  spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
      name: alert-policy-name
    period:
      startTime: 2022-10-01T12:00:00+02:00
      endTime: 2022-10-01T13:00:00+02:00

• spec.period.startTime and spec.period.duration - the AlertPolicy will be silenced for one hour from the startTime time-point as in the example below:

  apiVersion: n9/v1alpha
  kind: AlertSilence
  metadata:
    name: alert-silence-name
    project: default
  spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
      name: alert-policy-name
    period:
      startTime: 2022-10-01T12:00:00+02:00
      duration: 1h

• When you define only spec.period.duration, the AlertPolicy will be silenced for the defined period (1h in the example below) since the application time:

  apiVersion: n9/v1alpha
  kind: AlertSilence
  metadata:
    name: alert-silence-name
    project: default
  spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
      name: alert-policy-name
    period:
      duration: 1h

• When you define only endTime, the AlertPolicy will be silenced from the application time to the defined endtime time-point:

  apiVersion: n9/v1alpha
  kind: AlertSilence
  metadata:
    name: alert-silence-name
    project: default
  spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
      name: alert-policy-name
    period:
      endTime: 2025-01-01T13:00:00+02:00

Removing AlertSilence

You can remove AlertSilence from an SLO by:

• Using the sloctl delete alertsilences alert-silence-name command.
• Updating the value for period.endTime and setting it to the current date.

Getting AlertSilence

Using the sloctl get alertsilences command, you can only see these AlertSilence objects that are active or are planned for the future.

Validation errors

• If you apply AlertSilence to an SLO that doesn't exist, sloctl returns the following error:

  Applying AlertSilence ('my-alert-silence') in the 'default' project failed because the object SLO ('my-slo') referenced in its spec does not exist in the 'default' project.

• If you apply AlertSilence to an AlertPolicy that is not attached to the SLO, sloctl returns the following validation error:

  Applying AlertSilence ('my-alert-silence') failed because the object referenced in the field alertPolicy ('my-alert-policy') couldn't be found on the list of assigned alert policies.

• If you configure simultaneously more then one AlertSilence to one SLO, sloctl returns the following validation error:

  You can't set more than one AlertSilence for the AlertPolicy ('my-alert-policy') assigned to the SLO ('my-slo').

• If you use the sloctl get command for an AlertSilence that ended in the past, sloctl returns the following validation error:

  No resources found in the 'default' project.

Alert silence and RBAC

• Users with the Organization admin and Organization responder roles or users with Project owner or editor, or Project responder roles can apply or delete silence on an alert via sloctl.
• Users having View permissions can get silenced alerts in the specific project.