Skip to main content

ServiceNow Cloud Observability (formerly Lightstep)

Reading time: 0 minute(s) (0 words)

ServiceNow Cloud Observability (formerly Lightstep) features distributed tracing that can be used to rapidly pinpoint the causes of failures and poor performance across the deeply complex dependencies among services, teams, and workloads in modern production systems. Nobl9 integration with ServiceNow Cloud Observability facilitates organizations to establish service level objectives from performance data captured through distributed traces in the ServiceNow Cloud Observability platform.

Scope of support​

ServiceNow Cloud Observability parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent, Direct
Replay and SLI Analyzer: Historical data limit 30 days
Event logs: Supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 2 min
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9lightstep
Query delay environment variable: LS_QUERY_DELAY
Replay and SLI Analyzer: 0.65.0
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0

You can configure Nobl9 SLOs with ServiceNow Cloud Observability by using one of the following metric types:

  • ServiceNow Cloud Observability Unified Query Language (UQL)

    • Nobl9 supports constant, metrics, and spans query types in the UQL for both, Threshold and Ratio metric types

      caution

      Nobl9 does not support creating SLOs with the following ServiceNow Cloud Observability UQL queries: spans_sample and assemble.

  • Latency Threshold for Threshold metric type

  • Error Threshold for Threshold metric type

  • Error Ratio for Ratio metric type

Learn more about available metric types.

Creating SLOs with ServiceNow Cloud Observability​

Nobl9 Web​

Follow the instructions below to create your SLOs with ServiceNow Cloud Observability in the UI:

  1. Navigate to Service Level Objectives.

  2. Click .

  3. In step 2, select ServiceNow Cloud Observability as the data source for your SLO.

  4. Specify the Metric. You can choose either a Threshold metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).

    For the threshold metric, you can create SLO using one of the following metrics:

    UQL query:


    Latency Threshold metric that is the n-th percentile of latency in milliseconds:

    • Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
      For more information, refer to the Authentication section.
    • Select a Percentile.

    Error Threshold metric that is a single value representing the percentage of errors:

    • Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
      For more information, refer to the Authentication section.
    tip

    For more detailed metrics description, refer to the Available Metric Types section of the documentation.

  5. In step 3, define a Time Window for the SLO.

  • Rolling time windows are better for tracking the recent user experience of a service.

  • Calendar-aligned windows are best suited for SLOs that are intended to map to business metrics measured on a calendar-aligned basis, such as every calendar month or every quarter.

  1. In step 4, specify the Error Budget Calculation Method and your Objective(s).

    • Occurrences method counts good attempts against the count of total attempts.
    • Time Slicesmethod measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
    • You can define up to 12 objectives for an SLO.

    See the use case example and the SLO calculations guide for more information on the error budget calculation methods.

  2. In step 5, add the Display name, Name, and other settings for your SLO:

    • Create a composite SLO
    • Set notification on data, if this option is available for your data source.
      When activated, Nobl9 notifies you if your SLO hasn't received data or received incomplete data for more than 15 minutes.
    • Add alert policies, labels, and links, if required.
      You can add up to 20 links per SLO.

  3. Click Create SLO.

SLI values for good and total
When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
  • Must be positive.
  • While we recommend using integers, fractions are also acceptable.
    • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
  • Shouldn't be larger than 1e+20.

sloctl​

Here’s an example of ServiceNow Cloud Observability using rawMetric (threshold metric) with Metrics as the configuration type:

YAML definition for a metrics threshold SLO
- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: api-server-slo
    # Optional
    #displayName: API Server SLO
    project: default
    # Labels and annotations are optional
    #labels:
    #  area:
    #    - latency
    #    - slow-check
    #  env:
    #    - prod
    #    - dev
    #  region:
    #    - us
    #    - eu
    #  team:
    #    - green
    #    - sales
    #annotations:
    #  area: latency
    #  env: prod
    #  region: us
    #  team: sales
  spec:
    description: Example ServiceNow Cloud Observability SLO
    indicator:
      metricSource:
        name: lightstep
        project: default
        kind: Agent
    budgetingMethod: Occurrences
    objectives:
      - displayName: Good response (200)
        value: 200.0
        name: ok
        target: 0.95
        rawMetric:
          query:
            lightstep:
              typeOfData: metric
              uql: metric cpu.utilization | rate | group_by [], mean
        op: lte
        primary: true
    service: api-server
    timeWindows:
      - unit: Month
        count: 1
        isRolling: false
        calendar:
          startTime: 2022-12-01 00:00:00
          timeZone: UTC
    # Alert policies, attachments, and anomaly notifications are optional
    #alertPolicies:
    #  - fast-burn-5x-for-last-10m
    #attachments:
    #  - url: https://docs.nobl9.com
    #    displayName: Nobl9 Documentation
    #anomalyConfig:
    #  noData:
    #    alertMethods:
    #      - name: slack-notification
    #        project: default

When ServiceNow Cloud Observability is used as ratio (count) metric, then the field incremental under spec.objectives.countMetrics must be set to false.

Metric specification from ServiceNow Cloud Observability has three fields:

  • streamID – mandatory, string. For instructions on how to retrieve it, go to Authentication section.

  • typeOfData – accepts one of the following values: metric, latency, error_rate, good, total. For more detailed information, refer to the Scope of support section of the documentation.

    Description of values for typeOfData fields:

    • metric - metrics queries with which you can use ServiceNow Cloud Observability's Query Language (UQL) to retrieve and process your metric data by creating your query in the ServiceNow Cloud Observability UI and copying and pasting the query into Nobl9. This type can be used both for rawMetric and countMetrics SLO types.

    • latency – the n-th percentile (look at field percentile) of latency in milliseconds. This type can be used only as rawMetric. The value of value under spec.objective must also represent milliseconds.

    • error_rate – a single value representing the percentage of errors. This type can be used only as rawMetric. The value of value under spec.objectives must be between 0 and 1.

    • good – the number of successful events (operations). It is calculated as total operations minus the number of errors. This value is only allowed in the ratio (count) metric.

    • total – the number of all events (operations). This value is only allowed in ratio (count) metric.

  • percentile – number of percentiles of latency. The value must be greater than 0 and less or equal to 99.99. This field is mandatory when you use typeOfData: latency, and is forbidden otherwise.

ServiceNow Cloud Observability UQL​

You can use ServiceNow Cloud Observability Unified Query Language (UQL) to retrieve and process your metric data. Nobl9 supports the constant, metrics, and spans query types in the UQL.

Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9. Nobl9 then passes the query to the query_timeseries ServiceNow Cloud Observability API to retrieve the time series data.

constant queries​

constant fetches a gauge time series where all points have value literal-value.

To build a query of the constant type, specify the required value:

constant 100

metrics queries​

You can build the UQL queries using the following ServiceNow Cloud Observability metric kinds:

  • Gauge, an instantaneous measurement, for example,
    metric memory.utilized | latest | group_by [], sum,
    spans count | delta | group_by [], sum.

  • Delta, a measurement of the change in metrics from point to point. For the delta-kind queries, you must choose one of the operators or appropriate reducer to convert the distribution values into scalar values to build SLI on it, for example,
    metric request.size.bytes | delta | group_by [], sum | point dist_count(value)
    For more information, refer to the Using distributions in UQL | ServiceNow Cloud Observability documentation.

  • If you select a percentile as an operator in a query for the threshold metric SLI type, we recommend using the 100th percentile for best results as Nobl9 uses percentiles to display the data in the SLI chart. The following is an example metric query with Delta metric kind:
    metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0)

  • If you define many aggregation values, Nobl9 will fetch data for the first aggregation value defined in your query. For example, Nobl9 will fetch data for the 100th percentile in the below query:
    metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0), percentile(value, 99.9)

lightstep:
  typeOfData: metric
  uql: metric cpu.utilization | rate | group_by [], mean

spans queries​

Limitations:

ServiceNow Cloud Observability UQL spans queries supported in the public API must have retained data in ServiceNow Cloud Observability streams.

For example, when spans is not retained in a stream, the following query:

spans latency | delta | filter ((service == "adservice") || (service == "frontend")) | group_by [], sum | point percentile(value, 99.9)

will return the following error when querying the API:

"rpc error: code = InvalidArgument desc = public API only supports retained spans TQL queries at this time, please create a retained span query first"

However, when spans is retained in a stream, after creating a stream for a given filter, API starts returning a metric. For example, the following UQL query will return the metric:

spans latency | delta | filter (service == "frontend") | group_by [], sum | point percentile(value, 99.9)

if service IN ("frontend") is an existing stream.

tip

You can test your spans query whether it has retained data in the stream in the ServiceNow Cloud Observability API Reference documentation.

caution

Retention period:

  • for UQL spans queries retained in the stream, the retention period is set from 28 days, up to two years.

For more information, refer to the ServiceNow Cloud Observability documentation.

Metric YAML sample:

lightstep:
  typeOfData: metric
  uql: spans count | delta | group_by [], sum

SLOs explained​

Latency threshold​

The Latency threshold SLO configuration uses the threshold metric method under the hood with the SLI equal to the specific percentile value defined in SLO configuration.
Learn more about performance investigation in ServiceNow Cloud Observability.

Nobl9 retrieves percentile values from ServiceNow Cloud Observability API under data.attributes[].latencies[].

These values are represented in ServiceNow Cloud Observability on the following chart (the Latency section):

Metric YAML sample:

lightstep:
  streamID: DzpxcSRh
  typeOfData: latency
  percentile: 95

Error threshold​

The Error threshold SLO configuration uses the threshold metric method under the hood with the SLI equal to the percentage of errors for a given stream.

Nobl9 retrieves te ops-counts and error-counts values from ServiceNow Cloud Observability API and uses them to calculate the value:

value = error-counts / ops-counts

Such calculated values are used as an SLI for SLOs configured with this method.

They are represented in ServiceNow Cloud Observability on the following chart (the Err% section):

Metric YAML sample:

lightstep:
  streamID: DzpxcSRh
  typeOfData: error_rate

Error ratio​

This SLO configuration uses count (ratio) metric method under the hood. Each count metric SLO needs two data streams: good and total.

With this configuration, Nobl9 retrieves the error-counts and ops-counts values from ServiceNow Cloud Observability API and calculates those data streams as following:

Good = ops-counts - error-counts
Total = ops-counts

By default, ServiceNow Cloud Observability does not show these values on chart. It shows operations per second instead.

Nobl9 doesn’t use Rate to calculate error budgets for any SLO. Events counts are used instead (calculated from ops-counts and error-counts that are retrieved from the API).

Metric YAML sample:

countMetrics:
  incremental: false
  good:
    lightstep:
      streamID: DzpxcSRh
      typeOfData: good
  total:
    lightstep:
      streamID: DzpxcSRh
      typeOfData: total

Querying the ServiceNow Cloud Observability API​

The Nobl9 agent makes calls the ServiceNow Cloud Observability API once every 60 seconds.

API rate limits​

ServiceNow Cloud Observability has low rate limits for its Streams Timeseries API. For Community, Professional, and Enterprise licenses it’s 60, 200, 600 requests per hour respectively. The Nobl9 agent makes requests once every 60s, which allows for one ServiceNow Cloud Observability organization to use only 1, 3, or 10 unique metric specifications. For more information, refer to the Rate Limits | ServiceNow Cloud Observability documentation.

ServiceNow Cloud Observability users can request an increase of rate limits via ServiceNow Cloud Observability customer support.

For a more in-depth look, consult additional resources:
Monitor Metric DataServiceNow Cloud Observability docsInvestigate a Latency RegressionServiceNow Cloud Observability docsInvestigate an Error Rate IncreaseServiceNow Cloud Observability docsInvestigate a Metric DeviationServiceNow Cloud Observability docsCreate and Manage API KeysServiceNow Cloud Observability docsStreams TimeseriesServiceNow Cloud Observability docsQuery TimeseriesServiceNow Cloud Observability docsRate LimitsServiceNow Cloud Observability docsUnified Query Language (UQL)ServiceNow Cloud Observability docsMetric KindsServiceNow Cloud Observability docsUsing distributions in UQLServiceNow Cloud Observability docsAgent metricsNobl9 agentCreating SLOs via TerraformThe Nobl9 Terraform provider docsCreating agents via TerraformThe Nobl9 Terraform provider docs
Last updated on
On this page