Skip to main content

Amazon CloudWatch

Reading time: 0 minute(s) (0 words)

Amazon CloudWatch is a monitoring and observability service and a repository that aggregates data from more than 70 AWS data sources. CloudWatch also allows users to publish custom metrics from their services. Creating SLOs using this data is a powerful tool to monitor large portfolios of products.

Nobl9 integration with CloudWatch supports CloudWatch Metrics Insights. Leveraging Metrics Insights, Nobl9 users can retrieve metrics even faster and gain added flexibility in querying raw service level indicator (SLI) data to use for their SLOs.

Using CloudWatch as a Source in Nobl9, users can configure their SLOs by leveraging data in CloudWatch-specific groupings – i.e., by region, namespaces, and dimensions.

Amazon CloudWatch parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent, Direct
Replay and SLI Analyzer: Historical data limit 15 days
Event logs: Supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 1 min
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9cloudwatch
Query delay environment variable: CW_QUERY_DELAY
Replay and SLI Analyzer: 0.65.0
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0
Additional notes:
Support for AWS cross-account observability
Bad over total ratio metrics
No support for high-resolution metrics and metrics that use more than one Unit
Learn more Opens in a new tab

Creating SLOs with CloudWatch​

Using Amazon CloudWatch, you can create SLOs by:

  • Entering standard threshold and ratio metrics

  • Entering an SQL query

  • Entering multiple queries through JSON

All three methods are available both in the UI and through applying YAML (see the Creating CloudWatch SLOs - YAML section).

Nobl9 Web​

Follow the instructions below to create your SLOs with CloudWatch in the UI:

  1. Navigate to Service Level Objectives.
  2. Click .
  3. Select a Service.
    It will be the location for your SLO in Nobl9.
  4. Select your Amazon CloudWatch data source.
  5. Modify Period for Historical Data Retrieval, if necessary.
    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on Amazon CloudWatch.
    • A longer period can extend the data loading time for your SLO.
    • Must be a positive whole number up to the maximum period value you've set when adding the Amazon CloudWatch data source.

    Non-editable Replay period
    Non-editable Replay period indicates that the maximum period for historical data retrieval set for your Amazon CloudWatch data source is set to zero.
    Adjust the data source settings to create the SLO with Replay.
  6. Metric refers to the way of calculating and interpreting calculate and interpret data from your data source.
    • Threshold metric is defined by a single numerical value (the threshold) that separates satisfactory performance from unsatisfactory performance. It's represented by a single time series evaluated against the threshold.
    • Ratio metric expresses the performance as a fraction or proportion, typically by dividing the number of successful or unsuccessful events by the total number of potential events (successes + failures). It's represented by two-time series for comparison for good or bad events and total events.
      For ratio metrics, select the Data count method.

      SLI values for good and total
      When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
      • Must be positive.
      • While we recommend using integers, fractions are also acceptable.
        • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
      • Shouldn't be larger than 1e+20.
  1. Select the query format:
  • Configurations to specify a standard threshold or ratio metrics
  • SQL to build an SQL query
  • JSON to query multiple CloudWatch metrics and use math expressions to create new time series based on these metrics
  1. Account ID (optional, only in Beta) is a 12-digit identification number of your AWS account. It allows you to access SLO data from multiple accounts within a Region.
    Beta feature

    AWS cross-account observability is available for configuration-type metrics.
    This field appears only if your Amazon CloudWatch data source was added to Nobl9 using the Beta release channel.

  2. Region is a region code in AWS.
  3. Namespace (mandatory) is a container for CloudWatch metrics. It must be up to 255 characters and can contain alphanumeric characters, period, a hyphen, underscore, forward slash, hash, or colon.
  4. Metric Name (mandatory) can include up to 255 characters.
  5. Statistic function is an aggregation of metric data over specified periods. For example, you can use Maximum, Minimum, Sum, Average.
  6. Dimensions (optional, list) refer to a name/value pair that is part of a metric’s identity. You can assign up to 10 dimensions per metric.
    • Both the Key and Value can be up to 255 ASCII characters long and must include at least one non-whitespace character.
    • For the Key, any spaces at the beginning or end are preserved
  1. Define the Time window for your SLO:
  2. Configure the Error budget calculation method and Objectives:
    • Occurrences method counts good attempts against the count of total attempts.
    • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
    • You can define up to 12 objectives for an SLO.

      • Similar threshold values for objectives
      To use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
      For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one.
  3. Add the Display name, Name, and other settings for your SLO:
    • Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
      Use only lowercase letters, numbers, and dashes.
    • Select No data anomaly alert to receive notifications when your SLO stops reporting data for a specified period:
      • Choose up to five supported Alert methods.
      • Specify the delay period before Nobl9 sends an alert about the missing data.
        From 5 minutes to 31 days. Default: 15 minutes
    • Add alert policies, labels, and links, if required.
      Limits per SLO: 20 alert policies or links, 30 labels.
  4. Click CREATE SLO.

    5. SLO configuration use case
    Check the SLO configuration use case for a real-life SLO example.

YAML​

Standard configuration​

Sample Amazon CloudWatch threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example CloudWatch SLO
  indicator:
    metricSource:
      name: cloud-watch
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200.0
      name: ok
      target: 0.95
      rawMetric:
        query:
          cloudWatch:
            region: us-west-2
            namespace: AWS/RDS
            metricName: ReadLatency
            stat: Average
            dimensions:
              - name: LoadBalancer
                value: app/api-server
            accountId: "123456789012"
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01 00:00:00
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
  noData:
    alertMethods:
      - name: slack-notification
        project: default
    alertAfter: 1h
Click to open field reference
FieldTypeDescription
apiVersion
mandatory
stringAPI version. Use n9/v1alpha
kind
mandatory
stringThe resource type. Use SLO
Metadata
metadata.name
mandatory
stringName identifier for the SLO. Use only lowercase alphanumeric characters
metadata.displayNamestringUser-friendly SLO name
metadata.project
mandatory
stringThe name identifier of the project where you need to host your SLO
metadata.labelsobject (map: string[])Grouping labels for filtering or viewing
metadata.annotationsobject (map: string)Flat string annotations
Spec
spec.descriptionstringSLO description
spec.indicator.metricSource.name
mandatory
stringData source name
spec.indicator.metricSource.project
mandatory
stringProject containing the data source
spec.indicator.metricSource.kind
mandatory
stringData source connection method. Can be Agent or Direct
spec.budgetingMethod
mandatory
enumError budget calculation method. Can be Occurrences or Time slices
spec.objectives
mandatory
arrayYour SLO objective definition, up to 12 objectives per SLO.
spec.objectives[].displayNamestringUser-friendly objective name
spec.objectives[].value
mandatory
numberData point values that is considered "good" (e.g., 200.0).
In SLOs with two or more objectives, keep each objective's value unique.
In ratio (count) metrics, value is retained for legacy purposes.
spec.objectives[].name
mandatory
stringName identifier for this objective
spec.objectives[].op
mandatory
string (enum)Operator for objective. One of:
lte (less than or equal to)
lt (less than)
gte (greater than or equal to)
gt (greater than)
spec.objectives[].target
mandatory
floatThe percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is 0.95, the good performance is expected to be observed in at least 95% of the time window)
spec.objectives[].rawMetric/.countMetric
mandatory
objectThe metric type indicator. Set:
rawMetric for a threshold metric
countMetric for a ratio metric.
A ratio metric requires the additional fields:
countMetric.incremental (boolean) the data count method
countMetric.good/.bad and countMetric.total a numerator and denominator queries
spec.objectives[].countMetric.incremental
mandatory
booleanThe data count method for a ratio (countMetric) metric type
spec.objectives[].primarybooleanThe indicator of a primary SLO objective
spec.service
mandatory
stringThe name identifier of a service to host this SLO. The service must exist in the project specified in metadata.project
spec.timeWindows
mandatory
arrayDefines SLO time window for error budget calculation. Set:
isRolling: true for the rolling time window type
isRolling: false for the calendar-aligned type
spec.timeWindows.unit
mandatory
integerThe time window units. One of:
Day | Hour | Minute for the rolling time window
Year | Quarter | Month | Week | Day for the calendar-aligned time window
spec.timeWindows.count
mandatory
integerThe number of units in a time window
spec.timeWindows.startTimestringMandatory for calendar-aligned time windows. Date and time in the format YYYY-MM-DDTHH:mm:ss
spec.timeWindows.timeZonestringMandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
spec.timeWindows.isRolling
mandatory
boolean
true for the rolling time window type
false for the calendar-aligned type
spec.alertPoliciesarrayThe name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
spec.attachmentsarrayLinks to any additional attributes of this SLO
spec.anomalyConfigobjectSettings for a manual no data anomaly detection rule
spec.noData.alertMethodsarrayList of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the name and project fields
spec.noData.alertAfterstringWaiting time before sending a no-data notification. Must be 5m to 31d.
Default: 15m
Source-specific fields
cloudwatch.region
mandatory
stringAWS region for metric source
cloudwatch.namespace
mandatory
stringCloudWatch metric container. Up to 255 alphanumeric characters, including hyphens (-), periods (.), underscores (_), forward slashes (/), hashes (#), and colons (:)
cloudwatch.metricName
mandatory
stringCloudWatch metric name. Up to 255 characters
cloudwatch.stat
mandatory
stringMetric data aggregation (e.g. Sum, Average, p95, TC(0.005:0.030)
cloudwatch.dimensions
mandatory
arrayArray of dimension objects (name-value pairs). Refer to a metric identity part. Up to 10 dimensions per metric. Requirements for name and value: up to 255 characters each, must contain only ASCII characters and include at least one non-whitespace character
cloudwatch.accountId
mandatory
stringAWS account identifier

Using CloudWatch SQL query​

Sample Amazon CloudWatch threshold SLO with the SQL query
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example CloudWatch SLO
  indicator:
    metricSource:
      name: cloud-watch
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200.0
      name: ok
      target: 0.95
      rawMetric:
        query:
          cloudWatch:
            region: us-west-2
            sql: SELECT AVG(CPUUtilization) FROM "AWS/EC2”
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01 00:00:00
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
  noData:
    alertMethods:
      - name: slack-notification
        project: default
    alertAfter: 1h
Click to open field reference
FieldTypeDescription
apiVersion
mandatory
stringAPI version. Use n9/v1alpha
kind
mandatory
stringThe resource type. Use SLO
Metadata
metadata.name
mandatory
stringName identifier for the SLO. Use only lowercase alphanumeric characters
metadata.displayNamestringUser-friendly SLO name
metadata.project
mandatory
stringThe name identifier of the project where you need to host your SLO
metadata.labelsobject (map: string[])Grouping labels for filtering or viewing
metadata.annotationsobject (map: string)Flat string annotations
Spec
spec.descriptionstringSLO description
spec.indicator.metricSource.name
mandatory
stringData source name
spec.indicator.metricSource.project
mandatory
stringProject containing the data source
spec.indicator.metricSource.kind
mandatory
stringData source connection method. Can be Agent or Direct
spec.budgetingMethod
mandatory
enumError budget calculation method. Can be Occurrences or Time slices
spec.objectives
mandatory
arrayYour SLO objective definition, up to 12 objectives per SLO.
spec.objectives[].displayNamestringUser-friendly objective name
spec.objectives[].value
mandatory
numberData point values that is considered "good" (e.g., 200.0).
In SLOs with two or more objectives, keep each objective's value unique.
In ratio (count) metrics, value is retained for legacy purposes.
spec.objectives[].name
mandatory
stringName identifier for this objective
spec.objectives[].op
mandatory
string (enum)Operator for objective. One of:
lte (less than or equal to)
lt (less than)
gte (greater than or equal to)
gt (greater than)
spec.objectives[].target
mandatory
floatThe percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is 0.95, the good performance is expected to be observed in at least 95% of the time window)
spec.objectives[].rawMetric/.countMetric
mandatory
objectThe metric type indicator. Set:
rawMetric for a threshold metric
countMetric for a ratio metric.
A ratio metric requires the additional fields:
countMetric.incremental (boolean) the data count method
countMetric.good/.bad and countMetric.total a numerator and denominator queries
spec.objectives[].countMetric.incremental
mandatory
booleanThe data count method for a ratio (countMetric) metric type
spec.objectives[].primarybooleanThe indicator of a primary SLO objective
spec.service
mandatory
stringThe name identifier of a service to host this SLO. The service must exist in the project specified in metadata.project
spec.timeWindows
mandatory
arrayDefines SLO time window for error budget calculation. Set:
isRolling: true for the rolling time window type
isRolling: false for the calendar-aligned type
spec.timeWindows.unit
mandatory
integerThe time window units. One of:
Day | Hour | Minute for the rolling time window
Year | Quarter | Month | Week | Day for the calendar-aligned time window
spec.timeWindows.count
mandatory
integerThe number of units in a time window
spec.timeWindows.startTimestringMandatory for calendar-aligned time windows. Date and time in the format YYYY-MM-DDTHH:mm:ss
spec.timeWindows.timeZonestringMandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
spec.timeWindows.isRolling
mandatory
boolean
true for the rolling time window type
false for the calendar-aligned type
spec.alertPoliciesarrayThe name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
spec.attachmentsarrayLinks to any additional attributes of this SLO
spec.anomalyConfigobjectSettings for a manual no data anomaly detection rule
spec.noData.alertMethodsarrayList of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the name and project fields
spec.noData.alertAfterstringWaiting time before sending a no-data notification. Must be 5m to 31d.
Default: 15m
Source-specific fields
cloudwatch.region
mandatory
stringAWS region for metric source. CloudWatch SQL query is available in all AWS regions, except China
cloudwatch.sql
mandatory
stringCloudWatch SQL query to compare, aggregate, and group metrics by labels

Multiple-metric SLO with JSON​

Sample Amazon CloudWatch multiple-metric threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example CloudWatch SLO
  indicator:
    metricSource:
      name: cloud-watch
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          cloudWatch:
            region: us-west-2
            json: |-
              [
                {
                  "Id": "e1",
                  "Expression": "m1 / m2",
                  "Period": 60
                },
                {
                  "Id": "m1",
                  "MetricStat": {
                    "Metric": {
                      "Namespace": "AWS/NATGateway",
                      "MetricName": "ConnectionEstablishedCount",
                      "Dimensions": [
                        {
                          "Name": "NatGatewayId",
                          "Value": "nat-016e2012a5fa8a151"
                        }
                      ]
                    },
                    "Period": 60,
                    "Stat": "SampleCount"
                  },
                  "ReturnData": false
                },
                {
                  "Id": "m2",
                  "MetricStat": {
                    "Metric": {
                      "Namespace": "AWS/NATGateway",
                      "MetricName": "ConnectionAttemptCount",
                      "Dimensions": [
                        {
                          "Name": "NatGatewayId",
                          "Value": "nat-016e2012a5fa8a151"
                        }
                      ]
                    },
                    "Period": 60,
                    "Stat": "SampleCount"
                  },
                  "ReturnData": false
                }
              ]
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: '2022-12-01 00:00:00'
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default
      alertAfter: 1h
Click to open field reference
FieldTypeDescription
apiVersion
mandatory
stringAPI version. Use n9/v1alpha
kind
mandatory
stringThe resource type. Use SLO
Metadata
metadata.name
mandatory
stringName identifier for the SLO. Use only lowercase alphanumeric characters
metadata.displayNamestringUser-friendly SLO name
metadata.project
mandatory
stringThe name identifier of the project where you need to host your SLO
metadata.labelsobject (map: string[])Grouping labels for filtering or viewing
metadata.annotationsobject (map: string)Flat string annotations
Spec
spec.descriptionstringSLO description
spec.indicator.metricSource.name
mandatory
stringData source name
spec.indicator.metricSource.project
mandatory
stringProject containing the data source
spec.indicator.metricSource.kind
mandatory
stringData source connection method. Can be Agent or Direct
spec.budgetingMethod
mandatory
enumError budget calculation method. Can be Occurrences or Time slices
spec.objectives
mandatory
arrayYour SLO objective definition, up to 12 objectives per SLO.
spec.objectives[].displayNamestringUser-friendly objective name
spec.objectives[].value
mandatory
numberData point values that is considered "good" (e.g., 200.0).
In SLOs with two or more objectives, keep each objective's value unique.
In ratio (count) metrics, value is retained for legacy purposes.
spec.objectives[].name
mandatory
stringName identifier for this objective
spec.objectives[].op
mandatory
string (enum)Operator for objective. One of:
lte (less than or equal to)
lt (less than)
gte (greater than or equal to)
gt (greater than)
spec.objectives[].target
mandatory
floatThe percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is 0.95, the good performance is expected to be observed in at least 95% of the time window)
spec.objectives[].rawMetric/.countMetric
mandatory
objectThe metric type indicator. Set:
rawMetric for a threshold metric
countMetric for a ratio metric.
A ratio metric requires the additional fields:
countMetric.incremental (boolean) the data count method
countMetric.good/.bad and countMetric.total a numerator and denominator queries
spec.objectives[].countMetric.incremental
mandatory
booleanThe data count method for a ratio (countMetric) metric type
spec.objectives[].primarybooleanThe indicator of a primary SLO objective
spec.service
mandatory
stringThe name identifier of a service to host this SLO. The service must exist in the project specified in metadata.project
spec.timeWindows
mandatory
arrayDefines SLO time window for error budget calculation. Set:
isRolling: true for the rolling time window type
isRolling: false for the calendar-aligned type
spec.timeWindows.unit
mandatory
integerThe time window units. One of:
Day | Hour | Minute for the rolling time window
Year | Quarter | Month | Week | Day for the calendar-aligned time window
spec.timeWindows.count
mandatory
integerThe number of units in a time window
spec.timeWindows.startTimestringMandatory for calendar-aligned time windows. Date and time in the format YYYY-MM-DDTHH:mm:ss
spec.timeWindows.timeZonestringMandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
spec.timeWindows.isRolling
mandatory
boolean
true for the rolling time window type
false for the calendar-aligned type
spec.alertPoliciesarrayThe name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
spec.attachmentsarrayLinks to any additional attributes of this SLO
spec.anomalyConfigobjectSettings for a manual no data anomaly detection rule
spec.noData.alertMethodsarrayList of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the name and project fields
spec.noData.alertAfterstringWaiting time before sending a no-data notification. Must be 5m to 31d.
Default: 15m
Source-specific fields
cloudwatch.region
mandatory
stringAWS region for metric source
cloudwatch.json
mandatory
stringA JSON query to retrieve multiple CloudWatch metrics and use math expressions to create new time series based on these metrics. The JSON query is validated against the following rules: it must be valid JSON and structured as an array of metrics; only one `ReturnData` field can be set to true (and it defaults to true when not set), while all other `ReturnData` fields in different metrics must be explicitly set to false; and the `Period` field in `MetricStat` is required and must be set to 60, or if `MetricStat` does not exist, the `Period` field in the base object should be set to 60.

Querying the CloudWatch server​

Once the SLO is set up, Nobl9 queries the CloudWatch server every 60 seconds.

CloudWatch API rate limits​

For GetMetricData API, CloudWatch has limit of 50TPS per Region set by default. This is the maximum number of operation requests you can make per second. For more information, refer to the CloudWatch service quotas | CloudWatch documentation.

CloudWatch has minimum query and store period - one second. By default, CloudWatch stores data with a 1-minute period.

CloudWatch retains metric data differently for various store period. For more information, refer to the GetMetricData | CloudWatch documentation.

Known limitations​

CloudWatch SQL query is available in all AWS Regions, except China.

Check out these related guides and references:
Add Amazon CloudWatch as data sourceAdding data sourcesPut Metric DataCloudWatch docsGet Metric DataCloudWatch docsAmazon CloudWatch ConceptsCloudWatch docsCloudWatch Statistics DefinitionsCloudWatch docsUsing Metric MathCloudWatch docsAWS Regional EndpointsCloudWatch docsCloudWatch Metrics InsightsCloudWatch docsCloudWatch Service QuotasCloudWatch docsCreating SLOs via TerraformTerraform
On this page