Skip to main content

Google Cloud Monitoring

Reading time: 0 minute(s) (0 words)

Google Cloud Monitoring (GCM) provides visibility into the performance, uptime, and overall health of cloud-powered applications. It collects metrics, events, and metadata from Google Cloud, hosted uptime probes, and application instrumentation.

Google Cloud Monitoring parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent, Direct
Replay and SLI Analyzer: Historical data limit 30 days
Event logs: Supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 2 min
Jitter: 15 sec
Timeout: 50 sec
Agent details and minimum required versions for supported features:
Plugin name: n9gcm
Query delay environment variable: GCM_QUERY_DELAY
Replay and SLI Analyzer: 0.79.0-beta
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0
Additional notes:
Support for PromQL queries in Nobl9 agent v0.83.0-beta + / 0.88.0 +
Support for Google Cloud Monitoring metrics
Learn more Opens in a new tab

Creating SLOs with Google Cloud Monitoring​

Nobl9 Web​

Follow the instructions below to create your SLOs with Google Cloud Monitoring in the Nobl9 UI:

  1. Navigate to Service Level Objectives.

  2. Click .

  3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.

  4. In step 2, select Google Cloud Monitoring as the data source for your SLO.

  5. Enter a Project ID.

  6. Specify the Metric. You can choose either a Threshold Metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).

    1. Choose the Data Count Method for your ratio metric:
      • Non-incremental: counts incoming metric values one-by-one. So the resulting SLO graph is pike-shaped.
      • Incremental: counts the incoming metric values incrementally, adding every next value to previous values. It results in a constantly increasing SLO graph.

  7. Specify Query or a Good Query and Total Query for the metric you selected.

    • Each query must return only one metric and one time series.
    • Since Nobl9 asks for data every 1 minute, we recommend setting the period for the align delta function to 1 minute, i.e. align delta(1m).
      As a result, Nobl9 receives the difference in a given minute and records it as an SLI.
    • Nobl9 processes a single dataset at a time and doesn't aggregate GCM metrics.
      Make sure your group_by aggregator points to the single dataset—exactly that one you want to observe.
      You can find the available groups on your Google Cloud Observability Monitoring dashboard > Metrics explorer.
Monitoring Query Language

MQL is no longer recommended by Google as a query language for Cloud Monitoring. Following this, MQL is deprecated in Nobl9 as well. PromQL is a recommended replacement.

PromQL in GCM is supported by Nobl9 agent version0.83.0-beta / 0.88.0 or higher.

Refer to the PromQL Cheat Sheet for additional guidance.

Details

PromQL query examples Threshold query:
"sum(rate(serviceruntime_googleapis_com:api_request_latencies_sum{monitored_resource=\"consumed_api\",service=\"bigquery.googleapis.com\"}[1m]))/sum(rate(serviceruntime_googleapis_com:api_request_latencies_count{monitored_resource=\"consumed_api\",service=\"bigquery.googleapis.com\"}[1m]))"

Ratio query:
Good counter "sum(rate(serviceruntime_googleapis_com:api_request_count{monitored_resource=\"consumed_api\",response_code=\"200\",service=\"monitoring.googleapis.com\"}[1m]))"
Total counter "sum(rate(serviceruntime_googleapis_com:api_request_count{monitored_resource=\"consumed_api\",service=\"monitoring.googleapis.com\"}[1m]))"

SLI values for good and total
When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
  • Must be positive.
  • While we recommend using integers, fractions are also acceptable.
    • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
  • Shouldn't be larger than 1e+20.
  1. In step 3 of the SLO wizard, define a Time Window for the SLO.

  • Rolling time windows are better for tracking the recent user experience of a service.

  • Calendar-aligned windows are best suited for SLOs that are intended to map to business metrics measured on a calendar-aligned basis, such as every calendar month or every quarter.

  1. In step 4, specify the Error Budget Calculation Method and your Objective(s).
  • Occurrences method counts good attempts against the count of total attempts.
  • Time Slicesmethod measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
  • You can define up to 12 objectives for an SLO.

See the use case example and the SLO calculations guide for more information on the error budget calculation methods.

  1. In step 5, add the Display name, Name, and other settings for your SLO:
  • Create a composite SLO
  • Set notification on data, if this option is available for your data source.
    When activated, Nobl9 notifies you if your SLO hasn't received data or received incomplete data for more than 15 minutes.
  • Add alert policies, labels, and links, if required.
    You can add up to 20 links per SLO.
  1. Click Create SLO.

sloctl​

Monitoring Query Language

MQL is no longer recommended by Google as a query language for Cloud Monitoring. Following this, MQL is deprecated in Nobl9 as well. PromQL is a recommended replacement.

PromQL in GCM is supported by Nobl9 agent version0.83.0-beta / 0.88.0 or higher.

Refer to the PromQL Cheat Sheet for additional guidance.

Monitoring Query Language is deprecated. We recommend using PromQL to write queries for Google Cloud Monitoring.

Sample GCM rawMetric SLO YAML definition with the MQL query (deprecated)
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Google Cloud Monitoring SLO
  indicator:
    metricSource:
      name: google-cloud-monitoring
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          gcm:
            query: |-
              fetch api-server
              | metric 'serviceruntime.googleapis.com/api/request_latencies'
              | filter (resource.service == 'monitoring.googleapis.com')
              | align delta(1m)
              | every 1m
              | group_by [resource.service],
                  [value_request_latencies_mean: mean(value.request_latencies)]
            projectId: my-project-id
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01T00:00:00.000Z
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default

Expected query output​

Nobl9 accepts single time series only. Therefore, at each point in the time series, the GCM query must return a single value.
When your query includes multiple tables, for example, using ident, make sure it returns a single value.
You can test your query result with the projects.timeSeries.query method

Sample expected query output returning a single value
{
  "timeSeriesDescriptor": {
    "pointDescriptors": [
      {
        "key": "good_total_ratio",
        "valueType": "DOUBLE",
        "metricKind": "GAUGE",
        "unit": "1"
      }
    ]
  },
  "timeSeriesData": [
    {
      "pointData": [
        {
          "values": [
            {
              "doubleValue": 0.9877300613496932
            }
          ],
          "timeInterval": {
            "startTime": "2024-06-06T08:00:03.532075Z",
            "endTime": "2024-06-06T08:00:03.532075Z"
          }
        }
      ]
    }
  ]
}

Querying the Google Cloud Monitoring server​

Google Cloud Monitoring API rate limits​

To verify the limits to API usage, go to the Quotas dashboard in the GCM UI. For an API, click the All Quotas button to see your quota.

For a more in-depth look, consult additional resources:
Getting started with authenticationGoogle Cloud docsCreating and managing projectsGoogle Cloud docsMonitoring Query Language referenceGoogle Cloud docsprojects.timeSeries.queryGoogle Cloud docsAgent metricsAgentCreating SLOs via TerraformTerraformCreating agents via TerraformTerraform
Last updated on
On this page