Skip to main content

Datadog

Reading time: 0 minute(s) (0 words)

Datadog is a cloud-scale application observability solution that monitors servers, databases, tools, and services. Nobl9 connects with Datadog to collect SLI measurements and compare them to SLO targets. Nobl9 can activate processes and notifications when the error budget burn rate is too high or has been surpassed because it calculates error budgets of acceptable thresholds.

Users can pass business context through monitoring data, developing and measuring reliability targets, and aligning activities against the error budget's priorities using Nobl9 integration with Datadog.

Datadog parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent, Direct
Replay and SLI Analyzer: Historical data limit 30 days
Event logs: Supported
Query checker: Supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 2 min
Query delay: 1 min
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9datadog
Query delay environment variable: DATADOG_QUERY_DELAY
Replay and SLI Analyzer: 0.65.0
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0

Creating SLOs with Datadog​

Nobl9 Web​

Follow the instructions below to create your SLOs with Datadog in the UI:

  1. Navigate to Service Level Objectives.

  2. Click .

  3. In step 1 of the SLO wizard, select the Service your SLO will be associated with.

  4. In step 2, select Datadog as the Data Source for your SLO.

  5. Modify Period for Historical Data Retrieval, when necessary.
    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on Datadog.
    • A longer period can extend the data loading time for your SLO.
    • Must be a positive whole number up to the maximum period value you've set when adding the Datadog data source.

  6. Select the metric type:

    • A Threshold metric where a single time series is evaluated against a threshold.
    • A Ratio metric that allows you to enter two-time series for comparison.
        • For the ratio metric, select the Data Count Method:
      • Non-incremental: counts incoming metric values one-by-one. So the resulting SLO graph is pike-shaped.
      • Incremental: counts the incoming metric values incrementally, adding every next value to previous values. It results in a constantly increasing SLO graph.

  7. Enter a Query or Good Query and Total Query for the metric you selected.

      A sample threshold metric for Datadog:

    • avg:trace.http.request.duration{service:my-service}.as_count()

      A sample ratio metric for Datadog:

    • Good avg:trace.http.request.hits.by_http_status{service:my-service,!http.status_class:5xx}.as_count()
    • Total avg:trace.http.request.hits.by_http_status{service:my-service}.as_count()

    SLI values for good and total
    When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
    • Must be positive.
    • While we recommend using integers, fractions are also acceptable.
      • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
    • Shouldn't be larger than 1e+20.
  1. Define the Time window for your SLO:
  2. Configure the Error budget calculation method and Objectives:
    • Occurrences method counts good attempts against the count of total attempts.
    • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
    • You can define up to 12 objectives for an SLO.

      • Similar threshold values for objectives
      To use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
      For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one.
      Learn more about threshold value uniqueness.
  3. Add the Display name, Name, and other settings for your SLO:
    • Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
      Use only lowercase letters, numbers, and dashes.
    • Create composite SLO: with this option selected, you create a composite SLO 1.0. Composite SLOs 1.0 are deprecated. They're fully operable; however, we encourage you to create new composite SLOs 2.0.
      You can create composite SLOs 2.0 with sloctl using the provided template. Alternatively, you can create a composite SLO 2.0 with Nobl9 Terraform provider.
    • Set Notifications on data. With it, Nobl9 will notify you in the cases when SLO won't be reporting data for more than 15 minutes.
    • Add alert policies, labels, and links, if required.
      Up to 20 items of each type per SLO is allowed.
  4. Click CREATE SLO.

    5. SLO configuration use case
    Check the SLO configuration use case for a real-life SLO example.

sloctl​

Sample Datadog threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Datadog SLO
  indicator:
    metricSource:
      name: datadog
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          datadog:
            query: avg:trace.http.request.duration{*}
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01T00:00:00.000Z
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default

Important notes:

Learn more about Datadog queries.

warning

Queries to Datadog must return only one time series.

Example:

❌ Grouping metrics will often result in a multiple time series:

  • avg:system.load.1{*} by {host}
Image 1: Sample query with grouping metrics

✔ Same query without grouping

  • avg:system.load.1{*}
Image 2: Same query without grouping metrics
Rollup functions

Try avoiding the .rollup() or .moving_rollup() functions in your queries.

The Nobl9 agent uses enforced rollup described in the Rollup Interval: Enforced vs Custom to control the number of points returned from the queries. Using .rollup() or .moving_rollup() can affect the number of returned points or the way they are aggregated. This fact, in conjunction with the time range of each query by the Nobl9 agent can skew the calculated error budgets.

Learn more about Rollup.

Querying the Datadog server and API rate limits​

The Nobl9 agent leverages the Query Timeseries API parameters at a two-minute interval.

Requests to Datadog APIs are rate limited. The following is applied:

  • The default rate limit for the Query Timeseries API call is 1600 per hour per organization.
  • Nobl9 batches queries to Datadog into single 1024-character requests, including commas. Identical queries are sent only once to prevent redundancy.

For this reason, the amount of SLIs per interval depends on query length. With no batching, sending one query per API request, Nobl9 sends 52 SLIs per two-minute interval minimum (where an SLI represents a single time series).

When you need greater limits, refer to your Datadog contact.

Ensure query correctness

An incorrect query can occasionally cause delays for all queries in that batch. Identifying the problematic query may take time, leading to temporary processing delays.

For a more in-depth look, consult additional resources:
Nobl9 integration with datadogDatadog documentationRollup interval: Enforced vs customDatadog documentationRate limitsDatadog documentationQuery timeseries APIDatadog documentationAgent metricsAgentCreating SLOs via TerraformTerraformCreating agents via TerraformTerraform
Last updated on
On this page