Skip to main content

ServiceNow Cloud Observability (Formerly Lightstep)

Reading time: 0 minute(s) (0 words)

ServiceNow Cloud Observability (formerly Lightstep) enables distributed tracing, that can be used to rapidly pinpoint the causes of failures and poor performance across the deeply complex dependencies among services, teams, and workloads in modern production systems. Nobl9 integration with ServiceNow Cloud Observability enables organizations to establish service level objectives from performance data captured through distributed traces in the ServiceNow Cloud Observability platform.

Scope of Support​

Nobl9 supports reading Streams Timeseries and Query Timeseries ServiceNow Cloud Observability API.

You can configure Nobl9 SLOs with ServiceNow Cloud Observability by using one of the following metric types:

  • ServiceNow Cloud Observability Unified Query Language (UQL)

    • Nobl9 supports metrics and spans query types in the UQL for both, Threshold and Ratio metric types

      caution

      Nobl9 does not support creating SLOs with the following ServiceNow Cloud Observability UQL queries: constant, spans_sample, and assemble.

  • Latency Threshold for Threshold metric type

  • Error Threshold for Threshold metric type

  • Error Ratio for Ratio metric type

For more detailed information, refer to the Available Metric Types section of the documentation.

Authentication​

Before making an API call to ServiceNow Cloud Observability, Nobl9 needs to pass the following credentials:

  • Lightstep Organization ID: you can retrieve the organization ID from the Settings tab in ServiceNow Cloud Observability UI.

  • Lightstep Project Name. For details on how to create your ServiceNow Cloud Observability Projects, go here.

  • Lightstep Stream ID. It is an ID of a metric stream created in ServiceNow Cloud Observability. There are several ways to obtain the Stream ID (refer to How to Obtain Stream ID? for more details):

    • The Stream ID is described in the ServiceNow Cloud Observability documentation: How to Create Stream in Lightstep? | Lightstep Documentation

    • In Lightstream UI:

      1. Select a service.

      2. Go to the Streams tab.

      3. Open the selected stream.

      Your stream ID can be obtained from a URL of the "Stream Details" page:

      Image 1: Lightstream UI - stream ID example

ServiceNow Cloud Observability (Lightstep) API Token​

When deploying the Nobl9 agent, it is required to provide LS_APP_TOKEN environment variable for authentication with the ServiceNow Cloud Observability Streams Timeseries API. There is a placeholder for that value in the configuration obtained from installation instructions in the Nobl9 web app. For more details, go here.

When setting up the Direct connection to ServiceNow Cloud Observability, the LS_APP_TOKEN environment variable must be specified indirectly in the Lightstep App Token field.

Adding ServiceNow Cloud Observability as a Data Source in the UI​

To add ServiceNow Cloud Observability as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:

  1. Navigate to Integrations > Sources.
  2. Click the button.
  3. Click the relevant Source icon.
  4. Choose a relevant connection method (Agent or Direct), then configure the source as described below.

ServiceNow Cloud Observability Direct​

Direct Configuration in the UI​

Direct connection to ServiceNow Cloud Observability requires users to enter their credentials which Nobl9 stores safely. To set up this type of connection:

  1. Select one of the following Release Channels:
    • The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
    • The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel may be subject to change.

  2. Enter the name of the Lightstep Organization to connect to your data source (mandatory).
    For more details, refer to the Authentication section above.

  3. Enter a name in the Lightstep Project field (mandatory).
    For more details, refer to the Authentication section above.

  4. Enter the Lightstep App Token (mandatory).
    For details on how to create the app token, refer to the ServiceNow Cloud Observability API Token section above.

  1. Select a Project.
    Specifying a Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank then object is assigned to project default.
  2. Enter a Display Name.
    You can enter a friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example, my-project-name). This field is populated automatically when you enter a display name, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Customize the Query delay to set a customized delay for queries when pulling the data from the data source.
    • The default value in ServiceNow Cloud Observability integration for Query delay is 1 minute.
    info
    Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
  6. Enter a Maximum Period for Historical Data Retrieval.
    • This value defines how far back in the past your data will be retrieved.
    • The value for the Maximum period of data retrieval depends on the Source. Check the Replay documentation for details.
    • Entering a more extended Period might slow down the loading time when creating an SLO.
      • The value must be a positive integer.
  7. Enter a Default Period for Historical Data Retrieval.
    • It is the Period that will be used by the SLO connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. If you set it to >0, you will create an SLO with Replay.
  8. Click the Add Data Source button.
caution

ServiceNow Cloud Observability does not recognize the distinction between missing data and valid data with a 0 value in the stream. In such cases, ServiceNow Cloud Observability considers these values to be equal and returns the 0 value.

Direct Using CLI - YAML​

The YAML for setting up a Direct connection to ServiceNow Cloud Observability looks like this:

apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: snow-obs-direct
  displayName: Snow Obs direct
  project: snow-obs-direct
spec:
  description: direct integration with ServiceNow Cloud Observability
  sourceOf: # One or many values from this list are allowed: Metrics, Services
    - Metrics
    - Services
  queryDelay:
      unit: Minute # string, one of: Second || Minute
      value: 720 # numeric, must be a number less than 1440 minutes (24 hours)
  logCollectionEnabled: false # boolean, defaults to 'false'. Set to true if you'd like your source to collect logs. It's a beta functionality available only through direct release channel. Reach out to support@nobl9.com to activate it.
  lightstep:
    organization: ""
    project: ""
    appToken: "" # secret
  historicalDataRetrieval:
      maxDuration:
        value: 30 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day
      defaultDuration: # value must be less than or equal to value of maxDuration
        value: 0 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day

Important Notes:

ServiceNow Cloud Observability Agent​

Agent Configuration in the UI​

Follow the instructions below to set up an Agent connection. Refer to the section above for the descriptions of the fields.

  1. Select one of the following Release Channels:
    • The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
    • The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel may be subject to change.

  2. Enter the name of the Lightstep Organization to connect to your data source (mandatory).

  3. Enter a name in the Lightstep Project field (mandatory).

  1. Enter a Project.
  2. Enter a Name.
  3. Create a Description.
  4. In the Advanced Settings you can:
    1. Enter a Maximum Period for Historical Data Retrieval.
    2. Enter a Default Period for Historical Data Retrieval.
  5. Click the Add Data Source button.

Agent Using CLI - YAML​

The YAML for setting up an Agent connection to ServiceNow Cloud Observability looks like this:

apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: my-snow-obs-source
  displayName: ServiceNow Cloud Observability Agent # optional
  project: my-project
spec:
  description: Agent settings for ServiceNow Cloud Observability # optional
  sourceOf:
    - Metrics
    - Services
  queryDelay:
      unit: Minute # string, one of: Second || Minute
      value: 720 # numeric, must be a number less than 1440 minutes (24 hours)
  lightstep:
     organization: my-snow-obs-organization
     project: my-snow-obs-project
  historicalDataRetrieval:
      maxDuration:
        value: 30 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day
      defaultDuration: # value must be less than or equal to value of maxDuration
        value: 0 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day

Important Notes:

warning

You can deploy only one Agent in one YAML file by using the sloctl apply command.

Deploying ServiceNow Cloud Observability Agent​

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <LIGHTSTEP_APP_TOKEN> with your organization key).

note

The Nobl Agent by default uses the following API endpoint: https://api.lightstep.com. You can override it by specifying LS_API_URL environment variable during Agent start up.

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:

# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
  name: nobl9-agent-nobl9-dev-snow-obs-agent
  namespace: default
type: Opaque
stringData:
  ls_app_token: "<LIGHTSTEP_APP_TOKEN>"
  client_id: "unique_client_id"
  client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-nobl9-dev-default-hrun
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: "snowobsagent"
      nobl9-agent-project: "mysnowobs"
      nobl9-agent-organization: "nobl9-dev"
  template:
    metadata:
      labels:
        nobl9-agent-name: "snowobsagent"
        nobl9-agent-project: "mysnowobs"
        nobl9-agent-organization: "nobl9-dev"
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:latest
          resources:
            requests:
              memory: "350Mi"
              cpu: "0.1"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: nobl9-agent-nobl9-dev-default-hrun
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: nobl9-agent-nobl9-dev-default-hrun
            - name: LS_APP_TOKEN
              valueFrom:
                secretKeyRef:
                  key: ls_app_token
                  name: nobl9-agent-nobl9-dev-default-hrun
            # The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
            # The 9090 is the default value and can be changed.
            # If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
            - name: N9_METRICS_PORT
              value: "9090"

Creating SLOs with ServiceNow Cloud Observability​

Creating SLOs in the UI​

Follow the instructions below to create your SLOs with ServiceNow Cloud Observability in the UI:

  1. Navigate to Service Level Objectives.

  2. Click the button.

  3. In step 2, select ServiceNow Cloud Observability as the Data Source for your SLO.

  4. Specify the Metric. You can choose either a Threshold Metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).

    For the Threshold Metric, you can create SLO using one of the following metrics:

    UQL query:


    Latency Threshold metric that is the n-th percentile of latency in milliseconds:

    • Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
      For more information, refer to the Authentication section.
    • Select a Percentile.

    Error Threshold metric that is a single value representing the percentage of errors:

    • Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
      For more information, refer to the Authentication section.
    tip

    For more detailed metrics description, refer to the Available Metric Types section of the documentation.

  5. In step 3, define a Time Window for the SLO.

  6. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  7. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  8. When you’re done, click Create SLO.

Available Metric Types​

You can use one of the following metric types to configure Nobl9 SLOs with ServiceNow Cloud Observability:

ServiceNow Cloud Observability UQL​

This SLO configuration uses Threshold Metric method or the Ratio Metric under the hood.

You can use ServiceNow Cloud Observability Unified Query Language (UQL) to retrieve and process your metric data. Nobl9 supports the metrics and spans query types in the UQL.

Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9. Nobl9 then passes the query to the query_timeseries ServiceNow Cloud Observability API to retrieve the time series data.

You can build the UQL queries using the following ServiceNow Cloud Observability metric kinds:

  • Gauge, an instantaneous measurement, for example,
    metric memory.utilized | latest | group_by [], sum,
    spans count | delta | group_by [], sum.

  • Delta, a measurement of the change in metrics from point to point. For the delta-kind queries, you must choose one of the operators or appropriate reducer to convert the distribution values into scalar values to build SLI on it, for example,
    metric request.size.bytes | delta | group_by [], sum | point dist_count(value)
    For more information, refer to the Using Distributions in UQL | ServiceNow Cloud Observability Documentation.

    • If you select a percentile as an operator in a query for the Threshold Metric SLI type, we recommend using the 100th percentile for best results as Nobl9 uses percentiles to display the data in the SLI chart. The following is an example metric query with Delta metric kind:
      metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0)

    • If you define many aggregation values, Nobl9 will fetch data for the first aggregation value defined in your query. For example, Nobl9 will fetch data for the 100th percentile in the below query:
      metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0), percentile(value, 99.9)

Example Metric Configuration in YAML​
lightstep:
  typeOfData: metric
  uql: metric cpu.utilization | rate | group_by [], mean
spans Queries​
Limitations​

ServiceNow Cloud Observability UQL spans queries supported in the public API must have retained data in ServiceNow Cloud Observability streams.

For example, when spans is not retained in a stream, the following query:

spans latency | delta | filter ((service == "adservice") || (service == "frontend")) | group_by [], sum | point percentile(value, 99.9)

will return the following error when querying the API:

"rpc error: code = InvalidArgument desc = public API only supports retained spans TQL queries at this time, please create a retained span query first"

However, when spans is retained in a stream, after creating a stream for a given filter, API starts returning a metric. For example, the following UQL query will return the metric:

spans latency | delta | filter (service == "frontend") | group_by [], sum | point percentile(value, 99.9)

if service IN ("frontend") is an existing stream.

tip

You can test your spans query whether it has retained data in the stream in the ServiceNow Cloud Observability API Reference documentation.

caution

Retention period:

  • for UQL spans queries retained in the stream, the retention period is set from 28 days, up to 2 years.

For more information, refer to the ServiceNow Cloud Observability Documentation.

Example Metric Configuration in YAML​
lightstep:
  typeOfData: metric
  uql: spans count | delta | group_by [], sum

Latency Threshold​

This SLO configuration uses Threshold Metric method under the hood with the SLI equal to the specific percentile value defined when configuring the SLO. For more details, go here.

Nobl9 retrieves those values from ServiceNow Cloud Observability API under data.attributes[].latencies[].

These values are represented in ServiceNow Cloud Observability on the following chart (in Latency section).

Example Metric Configuration in YAML​
lightstep:
  streamID: DzpxcSRh
  typeOfData: latency
  percentile: 95

Error Threshold​

This SLO configuration uses Threshold Metric method under the hood with the SLI equal to the percentage of errors for a given Stream.

Nobl9 retrieves values ops-counts and error-counts from ServiceNow Cloud Observability API and uses them to calculate the value:

value = error-counts / ops-counts

Such calculated values are used as an SLI for SLOs configured with this method.

They are represented in ServiceNow Cloud Observability on the following chart (in Err% section).

Example Metric Configuration in YAML​
lightstep:
  streamID: DzpxcSRh
  typeOfData: error_rate

Error Ratio​

This SLO configuration uses Count (Ratio) Metric method under the hood. Each Count Metric SLO needs two data streams: Good and Total.

With this configuration, Nobl9 retrieves error-counts and ops-counts from ServiceNow Cloud Observability API and calculates those data streams as following:

Good = ops-counts - error-counts
Total = ops-counts

By default, ServiceNow Cloud Observability does not show these values on chart. It shows operations per second instead.

Nobl9 doesn’t use Rate to calculate Error Budgets for any SLO. Events counts are used instead (calculated from ops-counts and error-counts that are retrieved from the API).

Example Metric Configuration in YAML​
countMetrics:
  incremental: false
  good:
    lightstep:
      streamID: DzpxcSRh
      typeOfData: good
  total:
    lightstep:
      streamID: DzpxcSRh
      typeOfData: total

SLOs using ServiceNow Cloud Observability - YAML samples​

Here’s an example of ServiceNow Cloud Observability using rawMetric (Threshold metric) with Metrics as configuration type:

apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: get-cpu-utilization-rolling
  project: default
spec:
  service: default
  indicator:
    metricSource:
      name: servicenowobs
  timeWindows:
    - unit: Hour
      count: 1
      isRolling: true
  budgetingMethod: Occurrences
  objectives:
    - displayName: High
      op: gt
      rawMetric:
        query:
          lightstep:
            typeOfData: metric
            uql: metric cpu.utilization | rate | group_by [], mean
      value: 0.0005
      name: objective-1
      target: 0.97

When ServiceNow Cloud Observability is used as ratio (count) metric, then the field incremental under spec.objectives.countMetrics must be set to false.

Metric specification from ServiceNow Cloud Observability has 3 fields:

  • streamID – mandatory, string. For instructions on how to retrieve it, go to Authentication section.

  • typeOfData – accepts one of the following values: metric, latency, error_rate, good, total. For more detailed information, refer to the Scope of Support section of the documentation.

    Description of values for typeOfData fields:

    • metric - metrics queries with which you can use ServiceNow Cloud Observability's Query Language (UQL) to retrieve and process your metric data by creating your query in the ServiceNow Cloud Observability UI and copying and pasting the query into Nobl9. This type can be used both for rawMetric and countMetrics SLO types.

    • latency – the n-th percentile (look at field percentile) of latency in milliseconds. This type can be used only as rawMetric. The value of value under spec.objective must also represent milliseconds.

    • error_rate – a single value representing the percentage of errors. This type can be used only as rawMetric. The value of value under spec.objectives must be between 0 and 1.

    • good – the number of successful events (operations). It is calculated as total operations minus the number of errors. This value is only allowed in the ratio (count) metric.

    • total – the number of all events (operations). This value is only allowed in ratio (count) metric.

  • percentile – number of percentile of latency. The value must be greater than 0 and less or equal to 99.99. This field is mandatory when you use typeOfData: latency, and is forbidden otherwise.

Querying the ServiceNow Cloud Observability API​

The Nobl9 Agent makes calls the ServiceNow Cloud Observability API once every 60 seconds.

ServiceNow Cloud Observability API Rate Limits​

ServiceNow Cloud Observability has low rate limits for its Streams Timeseries API. For Community, Professional, and Enterprise licenses it’s 60, 200, 600 requests per hour respectively. The Nobl9 Agent makes requests once every 60s, which allows for one ServiceNow Cloud Observability organization to use only 1, 3, or 10 unique metric specifications. For more information, refer to the Rate Limits | ServiceNow Cloud Observability Documentation.

ServiceNow Cloud Observability users can request an increase of rate limits via ServiceNow Cloud Observability customer support.

Monitor Metric Data | ServiceNow Cloud Observability Documentation

Investigate a Latency Regression | ServiceNow Cloud Observability Documentation

Investigate an Error Rate Increase | ServiceNow Cloud Observability Documentation

Investigate a Metric Deviation | ServiceNow Cloud Observability Documentation

Create and Manage API Keys | ServiceNow Cloud Observability Documentation

Rate Limits | ServiceNow Cloud Observability Documentation

Unified Query Language (UQL | ServiceNow Cloud Observability Documentation

Metric Kinds | ServiceNow Cloud Observability Documentation

Using Distributions in UQL | ServiceNow Cloud Observability Documentation

Agent Metrics | Nobl9 Documentation

Creating SLOs via Terraform | Nobl9 Terraform Documentation

Creating Agents via Terraform | Nobl9 Terraform Documentation

Last updated on