Skip to main content

Silencing Alerts

SLOs with attached Alert Policies can get into a state where the user does not want to receive any alerts. For instance, this situation could be caused by a configuration issue that cannot be immediately resolved, which causes an alert to be triggered on and off. You can also have a known-bad SLO that won't recover for some time.

In such circumstances, Nobl9 allows you to silence the alerts that originate from a specific SLO for a predefined period. During the defined time, you won’t receive any alerts about the silenced Alert Policy for the SLO.

About AlertSilence

You can set AlertSilence at an SLO level for the selected Alert Policy.

You can apply only one AlertSilence object per an SLO at the same time.

When an Alert event assumes the resolved status within the duration of AlertSilence, Nobl9 will send you an all-clear notification. For more details, see Cooldown period | Nobl9 Documentation.

You can manage AlertSilence both in the Nobl9 UI and in sloctl.

Managing AlertSilence in sloctl

Applying AlertSilence

You can add AlertSilence to an SLO using the sloctl apply -f {yamlFile} command.

The following YAML example shows how to add an AlertSilence to your SLO:

apiVersion: n9/v1alpha
kind: AlertSilence
metadata:
name: alert-silence-name
project: default
spec:
description: AlertSilence description
slo: slo-name # the name of the SLO AlertSilence will be applied to
alertPolicy:
name: alert-policy-name # the name of the AlertPolicy AlertSilence will be applied to
period: # define period for the AlertSilence
startTime: 2025-01-01T12:00:00+02:00 # define the start date-time point where the Alerts will be silenced
endTime: 2025-01-01T13:00:00+02:00 # define the start date-time point where the Alerts will be silenced
duration: 1h

Notes

  • spec[n].alertPolicy - alertPolicy that AlertSilence will be applied to

    caution

    alertPolicy must be in the same project as AlertSilence and SLO

  • period[n].startTime, period[n].endTime:

    • These values define the date-time point where the Alerts will be silenced.
    • They must be in the YYYY-MM-DDTh:mm:ssZ format that complies with ISO8601.
  • period[n].duration

    • This value defines the duration for which Alerts will be silenced from the applying time.
    • Accepted units: s, m, h.

Configuring startTime, endTime and duration

You can combine startTime, endTime and duration as in the examples below:

  • startTime and endTime - AlertPolicy will be silenced from the defined startTime time-point to the endTime time-point:

    apiVersion: n9/v1alpha
    kind: AlertSilence
    metadata:
    name: alert-silence-name
    project: default
    spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
    name: alert-policy-name
    period:
    startTime: 2022-10-01T12:00:00+02:00
    endTime: 2022-10-01T13:00:00+02:00
  • startTime and duration - the AlertPolicy will be silenced for 1 hour from the startTime timepoint as in the example below:

    apiVersion: n9/v1alpha
    kind: AlertSilence
    metadata:
    name: alert-silence-name
    project: default
    spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
    name: alert-policy-name
    period:
    startTime: 2022-10-01T12:00:00+02:00
    duration: 1h
  • When you define only duration, the AlertPolicy will be silenced for the defined period (1h in the example below) since the application time:

    apiVersion: n9/v1alpha
    kind: AlertSilence
    metadata:
    name: alert-silence-name
    project: default
    spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
    name: alert-policy-name
    period:
    duration: 1h
  • When you define only endTime, the AlertPolicy will be silenced from the application time to the defined endtime timepoint:

    apiVersion: n9/v1alpha
    kind: AlertSilence
    metadata:
    name: alert-silence-name
    project: default
    spec:
    description: AlertSilence description
    slo: slo-name
    alertPolicy:
    name: alert-policy-name
    period:
    endTime: 2025-01-01T13:00:00+02:00

Removing AlertSilence

You can remove AlertSilence from an SLO by:

  • Using the sloctl delete alertsilences alert-silence-name command.
  • Updating the value for period[n].endTime and setting it to the current date.

Getting AlertSilence

Using the sloctl get alertsilences command, you can only see these AlertSilence objects that are active or are planned for the future.

Validation Errors

  • If you apply AlertSilence to an SLO that doesn't exist, sloctl returns the following error:

    Applying AlertSilence ('my-alert-silence') in the 'default' project failed because the object SLO ('my-slo') referenced in its spec does not exist in the 'default' project.

  • If you apply AlertSilence to an AlertPolicy that is not attached to the SLO, sloctl returns the following validation error:

    Applying AlertSilence ('my-alert-silence') failed because the object referenced in the field alertPolicy ('my-alert-policy') could'nt be found on the list of assigned alert policies.

  • If you configure simultaneously more then one AlertSilence to one SLO, sloctl returns the following validation error:

    You can't set more than one AlertSilence for the AlertPolicy ('my-alert-policy') assigned to the SLO ('my-slo').

  • If you use the sloctl get command for an AlertSilence that ended in the past, sloctl returns the following validation error:

    No resources found in the 'default' project.

Alert Silence and RBAC

  • Only Organization Admins or users with Project Owner/Edit permissions can apply/delete silence on an Alert via sloctl
  • Users having View permissions can get Silenced alerts in the specific project.