ServiceNow Cloud Observability (formerly Lightstep)
ServiceNow Cloud Observability (formerly Lightstep) features distributed tracing that can be used to rapidly pinpoint the causes of failures and poor performance across the deeply complex dependencies among services, teams, and workloads in modern production systems. Nobl9 integration with ServiceNow Cloud Observability facilitates organizations to establish service level objectives from performance data captured through distributed traces in the ServiceNow Cloud Observability platform.
Scope of supportβ
ServiceNow Cloud Observability parameters and supported features in Nobl9
- General support:
- Release channel: Stable, Beta
- Connection method: Agent, Direct
- Replay and SLI Analyzer: Historical data limit 30 days
- Event logs: Supported
- Query checker: Not supported
- Query parameters retrieval: Supported
- Timestamp cache persistence: Supported
- Query parameters:
- Query interval: 1 min
- Query delay: 2 min
- Jitter: 15 sec
- Timeout: 30 sec
- Agent details and minimum required versions for supported features:
- Plugin name: n9lightstep
- Query delay environment variable: LS_QUERY_DELAY
- Replay and SLI Analyzer: 0.65.0
- Query parameters retrieval: 0.73.2
- Timestamp cache persistence: 0.65.0
You can configure Nobl9 SLOs with ServiceNow Cloud Observability as follows:
- Threshold metrics:
- UQL:
constant
,metrics
, andspans
query types - Latency Threshold
- Error Threshold
- UQL:
- Ratio metrics:
- UQL:
constant
,metrics
, andspans
query types - Error Ratio
- UQL:
No support for spans_sample
and assemble
UQL queries.
Creating SLOs with ServiceNow Cloud Observabilityβ
Nobl9 Webβ
Follow the instructions below to create your SLOs with ServiceNow Cloud Observability on the Nobl9 Web:
- Navigate to Service Level Objectives.
- Click
.
- Select a Service.
It will be the location for your SLO in Nobl9. - Select your ServiceNow Cloud Observability data source.
- Modify Period for Historical Data Retrieval, when necessary.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on ServiceNow Cloud Observability.
- A longer period can extend the data loading time for your SLO.
- Must be a positive whole number up to the maximum period value you've set when adding the ServiceNow Cloud Observability data source.
- Select the Metric type:
- Threshold metric: a single time series is evaluated against a threshold.
- Ratio metric: two-time series for comparison for good events and total events.
For ratio metrics, select the Data count method: incremental or non-incremental.
- Threshold metric
- Ratio metric
constant
, metrics
, and spans
data.constant
, metrics
, and spans
data.ServiceNow Cloud Observability does not recognize the distinction between missing data and valid data with a 0
value in the stream. In such cases, ServiceNow Cloud Observability considers these values equal and returns the 0
value.
- Define the Time window for your SLO:
- Rolling time windows constantly move forward as time passes. This type can help track the most recent events.
- Calendar-aligned time windows are usable for SLOs intended to map to business metrics measured on a calendar-aligned basis.
- Configure the Error budget calculation method and Objectives:
- Occurrences method counts good attempts against the count of total attempts.
- Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
- You can define up to 12 objectives for an SLO.
Similar threshold values for objectivesTo use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
For example, if you want to use threshold value1
for two objectives, set it to1.0000001
for the first objective and to1.0000002
for the second one.
Learn more about threshold value uniqueness. - Add the Display name, Name, and other settings for your SLO:
- Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
Use only lowercase letters, numbers, and dashes. - Create composite SLO: with this option selected, you create a composite SLO 1.0. Composite SLOs 1.0 are deprecated. They're fully operable; however, we encourage you to create new composite SLOs 2.0.
You can create composite SLOs 2.0 withsloctl
using the provided template. Alternatively, you can create a composite SLO 2.0 with Nobl9 Terraform provider. - Set Notifications on data. With it, Nobl9 will notify you in the cases when SLO won't be reporting data for more than 15 minutes.
- Add alert policies, labels, and links, if required.
Up to 20 items of each type per SLO is allowed.
- Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
- Click CREATE SLO
countMetrics
), keep in mind that the values ββresulting from that query for both good and total:- Must be positive.
- While we recommend using integers, fractions are also acceptable.
- If using fractions, we recommend them to be larger than
1e-4
=0.0001
. - Shouldn't be larger than
1e+20
.
sloctlβ
- Metrics threshold
- Latency threshold
- Error threshold
- Metrics ratio
- Error ratio
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Metrics as the configuration type:
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: api-server-slo
# Optional
#displayName: API Server SLO
project: default
# Labels and annotations are optional
#labels:
# area:
# - latency
# - slow-check
# env:
# - prod
# - dev
# region:
# - us
# - eu
# team:
# - green
# - sales
#annotations:
# area: latency
# env: prod
# region: us
# team: sales
spec:
description: Example ServiceNow Cloud Observability SLO
indicator:
metricSource:
name: lightstep
project: default
kind: Agent
budgetingMethod: Occurrences
objectives:
- displayName: Good response (200)
value: 200.0
name: ok
target: 0.95
rawMetric:
query:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
op: lte
primary: true
service: api-server
timeWindows:
- unit: Month
count: 1
isRolling: false
calendar:
startTime: 2022-12-01 00:00:00
timeZone: UTC
# Alert policies, attachments, and anomaly notifications are optional
#alertPolicies:
# - fast-burn-5x-for-last-10m
#attachments:
# - url: https://docs.nobl9.com
# displayName: Nobl9 Documentation
#anomalyConfig:
# noData:
# alertMethods:
# - name: slack-notification
# project: default
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Latency threshold as the configuration type:
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: api-server-slo
# Optional
#displayName: API Server SLO
project: default
# Labels and annotations are optional
#labels:
# area:
# - latency
# - slow-check
# env:
# - prod
# - dev
# region:
# - us
# - eu
# team:
# - green
# - sales
#annotations:
# area: latency
# env: prod
# region: us
# team: sales
spec:
description: Example ServiceNow Cloud Observability SLO
indicator:
metricSource:
name: lightstep
project: default
kind: Agent
budgetingMethod: Occurrences
objectives:
- displayName: Good response (200)
value: 200.0
name: ok
target: 0.95
rawMetric:
query:
lightstep:
streamId: DzpxcSRh
typeOfData: latency
percentile: 95.0
op: lte
primary: true
service: api-server
timeWindows:
- unit: Month
count: 1
isRolling: false
calendar:
startTime: 2022-12-01 00:00:00
timeZone: UTC
# Alert policies, attachments, and anomaly notifications are optional
#alertPolicies:
# - fast-burn-5x-for-last-10m
#attachments:
# - url: https://docs.nobl9.com
# displayName: Nobl9 Documentation
#anomalyConfig:
# noData:
# alertMethods:
# - name: slack-notification
# project: default
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Error threshold as the configuration type:
# Metric type: threshold
# Metric variant: error
# Budgeting method: Occurrences
# Time window type: Calendar
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: api-server-slo
# Optional
#displayName: API Server SLO
project: default
# Labels and annotations are optional
#labels:
# area:
# - latency
# - slow-check
# env:
# - prod
# - dev
# region:
# - us
# - eu
# team:
# - green
# - sales
#annotations:
# area: latency
# env: prod
# region: us
# team: sales
spec:
description: Example ServiceNow Cloud Observability SLO
indicator:
metricSource:
name: lightstep
project: default
kind: Agent
budgetingMethod: Occurrences
objectives:
- displayName: Good response (200)
value: 200.0
name: ok
target: 0.95
rawMetric:
query:
lightstep:
streamId: DzpxcSRh
typeOfData: error_rate
op: lte
primary: true
service: api-server
timeWindows:
- unit: Month
count: 1
isRolling: false
calendar:
startTime: 2022-12-01 00:00:00
timeZone: UTC
# Alert policies, attachments, and anomaly notifications are optional
#alertPolicies:
# - fast-burn-5x-for-last-10m
#attachments:
# - url: https://docs.nobl9.com
# displayName: Nobl9 Documentation
#anomalyConfig:
# noData:
# alertMethods:
# - name: slack-notification
# project: default
Hereβs an example of ServiceNow Cloud Observability using countMetrics
(ratio metric) with Metrics as the configuration type:
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: api-server-slo
# Optional
#displayName: API Server SLO
project: default
# Labels and annotations are optional
#labels:
# area:
# - latency
# - slow-check
# env:
# - prod
# - dev
# region:
# - us
# - eu
# team:
# - green
# - sales
#annotations:
# area: latency
# env: prod
# region: us
# team: sales
spec:
description: Example ServiceNow Cloud Observability SLO
indicator:
metricSource:
name: lightstep
project: default
kind: Agent
budgetingMethod: Occurrences
objectives:
- displayName: Good response (200)
value: 1.0
name: ok
target: 0.95
countMetrics:
incremental: false
good:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
total:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], max
primary: true
service: api-server
timeWindows:
- unit: Month
count: 1
isRolling: false
calendar:
startTime: 2022-12-01 00:00:00
timeZone: UTC
# Alert policies, attachments, and anomaly notifications are optional
#alertPolicies:
# - fast-burn-5x-for-last-10m
#attachments:
# - url: https://docs.nobl9.com
# displayName: Nobl9 Documentation
#anomalyConfig:
# noData:
# alertMethods:
# - name: slack-notification
# project: default
Hereβs an example of ServiceNow Cloud Observability using countMetrics
(ratio metric) with Error ratio as the configuration type:
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: api-server-slo
# Optional
#displayName: API Server SLO
project: default
# Labels and annotations are optional
#labels:
# area:
# - latency
# - slow-check
# env:
# - prod
# - dev
# region:
# - us
# - eu
# team:
# - green
# - sales
#annotations:
# area: latency
# env: prod
# region: us
# team: sales
spec:
description: Example ServiceNow Cloud Observability SLO
indicator:
metricSource:
name: lightstep
project: default
kind: Agent
budgetingMethod: Occurrences
objectives:
- displayName: Good response (200)
value: 1.0
name: ok
target: 0.95
countMetrics:
incremental: false
good:
lightstep:
streamId: DzpxcSRh
typeOfData: error_rate
total:
lightstep:
streamId: DzpxcSRh
typeOfData: error_rate
primary: true
service: api-server
timeWindows:
- unit: Month
count: 1
isRolling: false
calendar:
startTime: 2022-12-01 00:00:00
timeZone: UTC
# Alert policies, attachments, and anomaly notifications are optional
#alertPolicies:
# - fast-burn-5x-for-last-10m
#attachments:
# - url: https://docs.nobl9.com
# displayName: Nobl9 Documentation
#anomalyConfig:
# noData:
# alertMethods:
# - name: slack-notification
# project: default
When ServiceNow Cloud Observability is used as ratio (count) metric, then the field incremental
under spec.objectives.countMetrics
must be set to false.
Metric specification from ServiceNow Cloud Observability has three fields:
-
streamID
β mandatory, string. For instructions on how to retrieve it, go to Authentication section. -
typeOfData
β accepts one of the following values: metric, latency, error_rate, good, total. For more detailed information, refer to the Scope of support section of the documentation.Description of values for
typeOfData
fields:-
metric -
metrics
queries with which you can use ServiceNow Cloud Observability's Query Language (UQL)Β to retrieve and process your metric data by creating your query in the ServiceNow Cloud Observability UI and copying and pasting the query into Nobl9. This type can be used both forrawMetric
andcountMetrics
SLO types. -
latency β the n-th percentile (look at field percentile) of latency in milliseconds. This type can be used only as
rawMetric
. The value ofvalue
underspec.objective
must also represent milliseconds. -
error_rate β a single value representing the percentage of errors. This type can be used only as
rawMetric
. The value ofvalue
underspec.objectives
must be between0
and1
. -
good β the number of successful events (operations). It is calculated as total operations minus the number of errors. This value is only allowed in the ratio (count) metric.
-
total β the number of all events (operations). This value is only allowed in ratio (count) metric.
-
-
percentile
β number of percentiles of latency. The value must be greater than0
and less or equal to99.99
. This field is mandatory when you usetypeOfData: latency
, and is forbidden otherwise.
ServiceNow Cloud Observability UQLβ
You can use ServiceNow Cloud Observability Unified Query Language (UQL)
to retrieve and process your metric data.
Nobl9 supports the constant
, metrics
, and spans
query types in the UQL.
Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9. Nobl9 then passes the query to the query_timeseries
ServiceNow Cloud Observability API to retrieve the time series data.
constant
queriesβ
constant
fetches a gauge time series where all points have value literal-value
.
To build a query of the constant
type, specify the required value:
constant 100
metrics
queriesβ
You can build the UQL queries using the following ServiceNow Cloud Observability metric kinds:
-
Gauge, an instantaneous measurement, for example,
metric memory.utilized | latest | group_by [], sum
,
spans count | delta | group_by [], sum
. -
Delta, a measurement of the change in metrics from point to point. For the delta-kind queries, you must choose one of the operators or appropriate reducer to convert the distribution values into scalar values to build SLI on it, for example,
metric request.size.bytes | delta | group_by [], sum | point dist_count(value)
For more information, refer to the Using distributions in UQL | ServiceNow Cloud Observability documentation. -
If you select a percentile as an operator in a query for the threshold metric SLI type, we recommend using the 100th percentile for best results as Nobl9 uses percentiles to display the data in the SLI chart. The following is an example
metric
query with Delta metric kind:
metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0)
-
If you define many aggregation values, Nobl9 will fetch data for the first aggregation value defined in your query. For example, Nobl9 will fetch data for the 100th percentile in the below query:
metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0), percentile(value, 99.9)
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
spans
queriesβ
Limitations:
ServiceNow Cloud Observability UQL spans
queries supported in the public API must have retained data in ServiceNow Cloud Observability streams.
For example, when spans
is not retained in a stream, the following query:
spans latency | delta | filter ((service == "adservice") || (service == "frontend")) | group_by [], sum | point percentile(value, 99.9)
will return the following error when querying the API:
"rpc error: code = InvalidArgument desc = public API only supports retained spans TQL queries at this time, please create a retained span query first"
However, when spans
is retained in a stream, after creating a stream for a given filter, API starts returning a metric. For example, the following UQL query will return the metric:
spans latency | delta | filter (service == "frontend") | group_by [], sum | point percentile(value, 99.9)
if service IN ("frontend")
is an existing stream.
You can test your spans
query whether it has retained data in the stream in the ServiceNow Cloud Observability API Reference documentation.
Retention period:
- for UQL
spans
queries retained in the stream, the retention period is set from 28 days, up to two years.
For more information, refer to the ServiceNow Cloud Observability documentation.
Metric YAML sample:
lightstep:
typeOfData: metric
uql: spans count | delta | group_by [], sum
SLOs explainedβ
Latency thresholdβ
The Latency threshold SLO configuration uses the threshold metric method under the hood with the SLI equal to the specific percentile value defined
in SLO configuration.
Learn more about performance investigation in ServiceNow Cloud Observability.
Nobl9 retrieves percentile values from ServiceNow Cloud Observability API under data.attributes[].latencies[]
.
These values are represented in ServiceNow Cloud Observability on the following chart (the Latency section):

Metric YAML sample:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
Error thresholdβ
The Error threshold SLO configuration uses the threshold metric method under the hood with the SLI equal to the percentage of errors for a given stream.
Nobl9 retrieves te ops-counts
and error-counts
values from ServiceNow Cloud Observability API and uses them to calculate the value:
value = error-counts / ops-counts
Such calculated values are used as an SLI for SLOs configured with this method.
They are represented in ServiceNow Cloud Observability on the following chart (the Err% section):

Metric YAML sample:
lightstep:
streamID: DzpxcSRh
typeOfData: error_rate
Error ratioβ
This SLO configuration uses count (ratio) metric method under the hood. Each count metric SLO needs two data streams: good and total.
With this configuration, Nobl9 retrieves the error-counts
and ops-counts
values from ServiceNow Cloud Observability API and calculates those data streams as following:
Good = ops-counts - error-counts
Total = ops-counts
By default, ServiceNow Cloud Observability does not show these values on chart. It shows operations per second instead.

Nobl9 doesnβt use Rate to calculate error budgets for any SLO. Events counts are used instead (calculated from ops-counts
and error-counts
that are retrieved from the API).
Metric YAML sample:
countMetrics:
incremental: false
good:
lightstep:
streamID: DzpxcSRh
typeOfData: good
total:
lightstep:
streamID: DzpxcSRh
typeOfData: total
Querying the ServiceNow Cloud Observability APIβ
The Nobl9 agent makes calls the ServiceNow Cloud Observability API once every 60 seconds.
API rate limitsβ
ServiceNow Cloud Observability has low rate limits for its Streams Timeseries API. For Community, Professional, and Enterprise licenses itβs 60, 200, 600 requests per hour respectively. The Nobl9 agent makes requests once every 60s, which allows for one ServiceNow Cloud Observability organization to use only 1, 3, or 10 unique metric specifications. For more information, refer to the Rate Limits | ServiceNow Cloud Observability documentation.
ServiceNow Cloud Observability users can request an increase of rate limits via ServiceNow Cloud Observability customer support.