ServiceNow Cloud Observability (formerly Lightstep)
ServiceNow Cloud Observability (formerly Lightstep) features distributed tracing that can be used to rapidly pinpoint the causes of failures and poor performance across the deeply complex dependencies among services, teams, and workloads in modern production systems. Nobl9 integration with ServiceNow Cloud Observability facilitates organizations to establish service level objectives from performance data captured through distributed traces in the ServiceNow Cloud Observability platform.
Scope of supportβ
Nobl9 supports reading Streams Timeseries and Query Timeseries ServiceNow Cloud Observability API.
You can configure Nobl9 SLOs with ServiceNow Cloud Observability by using one of the following metric types:
-
ServiceNow Cloud Observability Unified Query Language (UQL)
-
Nobl9 supports
metrics
andspans
query types in the UQL for both, Threshold and Ratio metric typescautionNobl9 does not support creating SLOs with the following ServiceNow Cloud Observability UQL queries:
constant
,spans_sample
, andassemble
.
-
-
Latency Threshold for Threshold metric type
-
Error Threshold for Threshold metric type
-
Error Ratio for Ratio metric type
For more information, refer to the Available metric types section of the documentation.
The following Nobl9 features are supported:
- Query parameters retrieval with
sloctl
- SLI Analyzer
- Replay
- Event logs for direct connection method
Authenticationβ
Before making an API call to ServiceNow Cloud Observability, Nobl9 needs to pass the following credentials:
-
Lightstep Organization ID: you can retrieve the organization ID from the Settings tab in ServiceNow Cloud Observability UI.
-
Lightstep Project Name. For details on how to create your ServiceNow Cloud Observability projects, go here.
-
Lightstep Stream ID. It is an ID of a metric stream created in ServiceNow Cloud Observability. There are several ways to obtain the Stream ID (refer to How to Obtain Stream ID? for more details):
-
The Stream ID is described in the ServiceNow Cloud Observability documentation: How to Create Stream in Lightstep? | Lightstep documentation
-
In the Lightstream UI:
-
Select a service.
-
Go to the Streams tab.
-
Open the selected stream.
Your stream ID can be obtained from a URL of the stream details page:
Image 1: Lightstream UI - stream ID example -
-
ServiceNow Cloud Observability (Lightstep) API tokenβ
When deploying the Nobl9 agent,
provide the LS_APP_TOKEN
environment variable for authentication with the ServiceNow Cloud Observability Streams Timeseries API.
To obtain it,
create a ServiceNow Cloud Observability API key.
When setting up the direct connection to ServiceNow Cloud Observability, provide your created API key in the Lightstep App Token field.
Adding ServiceNow Cloud Observability as a data sourceβ
To ensure data transmission between Nobl9 and ServiceNow Cloud Observability, it may be necessary to list Nobl9 IP addresses as trusted.
- 18.159.114.21
- 18.158.132.186
- 3.64.154.26
You can add the ServiceNow Cloud Observability data source using the direct or agent connection methods.
Direct connection methodβ
Direct connection to ServiceNow Cloud Observability requires users to enter their credentials which Nobl9 stores safely.
Nobl9 Webβ
To set up this type of connection:
- Navigate to Integrations > Sources.
- Click
.
- Click the required Source icon.
- Choose Direct.
-
Select one of the following Release Channels:
- The
stable
channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of abeta
release. Use it to avoid crashes and other limitations. - The
beta
channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
- The
-
Enter the name of the Lightstep Organization to connect to your data source (mandatory).
More about the Lightstep organization. -
Enter a name in the Lightstep Project field (mandatory).
More about the Lightstep project. -
Enter the Lightstep App Token (mandatory).
More about ServiceNow Cloud Observability API token.
- Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses thedefault
project. - Enter a Display Name.
You can enter a user-friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example,my-project-1
). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in ServiceNow Cloud Observability integration for Query delay is
1 minute
.
infoChanging the Query delay may affect your SLI data. For more details, check the Query delay documentation. - The default value in ServiceNow Cloud Observability integration for Query delay is
- Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
- The maximum period value depends on the data source.
Find the maximum value for your data source. - A greater period can extend the loading time when creating an SLO.
- The value must be a positive integer.
- Enter a Default Period for Historical Data Retrieval.
- It is used by SLOs connected to this data source.
- The value must be a positive integer or
0
. - By default, this value is set to 0. When you set it to
>0
, you will create SLOs with Replay.
- Click Add Data Source
ServiceNow Cloud Observability does not recognize the distinction between missing data and valid data with a 0
value in the stream. In such cases, ServiceNow Cloud Observability considers these values to be equal and returns the 0
value.
sloctlβ
The YAML for setting up a direct connection to ServiceNow Cloud Observability looks like this:
apiVersion: n9/v1alpha
kind: Direct
metadata:
name: snow-obs-direct
displayName: Snow Obs direct
project: snow-obs-direct
spec:
description: Direct integration with ServiceNow Cloud Observability
sourceOf:
- Metrics
- Services
queryDelay:
unit: Minute
value: 720
logCollectionEnabled: false
lightstep:
organization: my-snow-obs-organization
project: my-snow-obs-project
appToken: <APP_TOKEN>
url: <SERVICENOW_CLOUD_OBSERVABILITY_API_URL>
historicalDataRetrieval:
maxDuration:
value: 30
unit: Day
defaultDuration:
value: 0
unit: Day
Field | Type | Description |
---|---|---|
queryDelay.unit mandatory | enum | Specifies the unit for the query delay. Possible values: Second | Minute . β’ Check query delay documentation for default unit of query delay for each source. |
queryDelay.value mandatory | numeric | Specifies the value for the query delay. β’ Must be a number less than 1440 minutes (24 hours). β’ Check query delay documentation for default unit of query delay for each source. |
logCollectionEnabled optional | boolean | Optional. Defaults to false . Set to true if you'd like your direct to collect event logs. Beta functionality available only through direct release channel. Reach out to support@nobl9.com to activate it. |
releaseChannel mandatory | enum | Specifies the release channel. Accepted values: beta | stable . |
Source-specific fields | ||
lightstep.organization mandatory | string | ServiceNow Cloud Observability requires the name of organization registered in ServiceNow Cloud Observability, the name of the project , and the appToken . See authentication for more details. |
lightstep.project mandatory | string, secret | See authentication section above for more details. |
lightstep.appToken mandatory | string, secret | See authentication section above for more details. |
lightstep.url optional | string | A URL of the ServiceNow Cloud Observability instance. When no value provided, the default https://api.lightstep.com is used. |
Replay-related fields | ||
historicalDataRetrieval optional | n/a | Optional structure related to configuration related to Replay. β Use only with supported sources. β’ If omitted, Nobl9 uses the default values of value: 0 and unit: Day for maxDuration and defaultDuration . |
maxDuration.value optional | numeric | Specifies the maximum duration for historical data retrieval. Must be integer β₯ 0 . See Replay documentation for values of max duration per data source. |
maxDuration.unit optional | enum | Specifies the unit for the maximum duration of historical data retrieval. Accepted values: Minute | Hour | Day . |
defaultDuration.value optional | numeric | Specifies the default duration for historical data retrieval. Must be integer β₯ 0 and β€ maxDuration . |
defaultDuration.unit optional | enum | Specifies the unit for the default duration of historical data retrieval. Accepted values: Minute | Hour | Day . |
Important notes:
-
ServiceNow Cloud Observability requires the name of
organization
registered in ServiceNow Cloud Observability, the name of theproject
, and theappToken
. Refer to the Authentication section for more details. -
spec.historicalDataRetrieval
- refer to Replay documentation for more details. -
You can update the
lightstep.url
withsloctl
or Nobl9 Terraform provider.
Agent connection methodβ
Nobl9 Webβ
Follow the instructions below to set up an agent connection.
- Navigate to Integrations > Sources.
- Click
.
- Click the required Source icon.
- Choose Agent.
-
Select one of the following Release Channels:
- The
stable
channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of abeta
release. Use it to avoid crashes and other limitations. - The
beta
channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
- The
-
Enter the name of the Lightstep Organization to connect to your data source (mandatory).
-
Enter a name in the Lightstep Project field (mandatory).
- Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses thedefault
project. - Enter a Display Name.
You can enter a user-friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example,my-project-1
). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in ServiceNow Cloud Observability integration for Query delay is
1 minute
.
infoChanging the Query delay may affect your SLI data. For more details, check the Query delay documentation. - The default value in ServiceNow Cloud Observability integration for Query delay is
- Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
- The maximum period value depends on the data source.
Find the maximum value for your data source. - A greater period can extend the loading time when creating an SLO.
- The value must be a positive integer.
- Enter a Default Period for Historical Data Retrieval.
- It is used by SLOs connected to this data source.
- The value must be a positive integer or
0
. - By default, this value is set to 0. When you set it to
>0
, you will create SLOs with Replay.
- Click Add Data Source
sloctlβ
The YAML for setting up an agent connection to ServiceNow Cloud Observability looks like this:
apiVersion: n9/v1alpha
kind: Agent
metadata:
name: my-snow-obs-source
displayName: ServiceNow Cloud Observability Agent
project: my-project
spec:
description: Agent settings for ServiceNow Cloud Observability
sourceOf:
- Metrics
- Services
queryDelay:
unit: Minute
value: 720
lightstep:
organization: my-snow-obs-organization
project: my-snow-obs-project
url: <SERVICENOW_CLOUD_OBSERVABILITY_API_URL>
historicalDataRetrieval:
maxDuration:
value: 30
unit: Day
defaultDuration:
value: 0
unit: Day
Field | Type | Description |
---|---|---|
queryDelay.unit mandatory | enum | Specifies the unit for the query delay. Possible values: Second | Minute . β’ Check query delay documentation for default unit of query delay for each source. |
queryDelay.value mandatory | numeric | Specifies the value for the query delay. β’ Must be a number less than 1440 minutes (24 hours). β’ Check query delay documentation for default unit of query delay for each source. |
releaseChannel mandatory | enum | Specifies the release channel. Accepted values: beta | stable . |
Source-specific fields | ||
lightstep.organization mandatory | string | ServiceNow Cloud Observability requires the name of organization registered in ServiceNow Cloud Observability, the name of the project , and the appToken . See authentication for more details. |
lightstep.project mandatory | string, secret | See authentication section above for more details. |
lightstep.url optional | string | A URL of the ServiceNow Cloud Observability instance. When no value provided, the default https://api.lightstep.com is used. |
Replay-related fields | ||
historicalDataRetrieval optional | n/a | Optional structure related to configuration related to Replay. β Use only with supported sources. β’ If omitted, Nobl9 uses the default values of value: 0 and unit: Day for maxDuration and defaultDuration . |
maxDuration.value optional | numeric | Specifies the maximum duration for historical data retrieval. Must be integer β₯ 0 . See Replay documentation for values of max duration per data source. |
maxDuration.unit optional | enum | Specifies the unit for the maximum duration of historical data retrieval. Accepted values: Minute | Hour | Day . |
defaultDuration.value optional | numeric | Specifies the default duration for historical data retrieval. Must be integer β₯ 0 and β€ maxDuration . |
defaultDuration.unit optional | enum | Specifies the unit for the default duration of historical data retrieval. Accepted values: Minute | Hour | Day . |
Important notes:
-
ServiceNow Cloud Observability requires the name of
organization
registered in ServiceNow Cloud Observability, the name of theproject
, and theappToken
. Refer to the Authentication section for more details. -
spec.historicalDataRetrieval
- refer to Replay documentation for more details. -
You can update the
lightstep.url
withsloctl
or Nobl9 Terraform provider.
You can deploy only one agent in one YAML file by using the sloctl apply
command.
Agent deploymentβ
When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <LIGHTSTEP_APP_TOKEN>
with your organization key).
The Nobl9 agent by default uses the following API endpoint: https://api.lightstep.com
. You can override it by specifying LS_API_URL
environment variable during agent start up.
- Kubernetes
- Docker
If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the agent. It will look something like this:
# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.
apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-snow-obs-agent
namespace: default
type: Opaque
stringData:
ls_app_token: "<LIGHTSTEP_APP_TOKEN>"
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-default-hrun
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "snowobsagent"
nobl9-agent-project: "mysnowobs"
nobl9-agent-organization: "nobl9-dev"
template:
metadata:
labels:
nobl9-agent-name: "snowobsagent"
nobl9-agent-project: "mysnowobs"
nobl9-agent-organization: "nobl9-dev"
spec:
containers:
- name: agent-container
image: nobl9/agent:0.80.0
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-default-hrun
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-default-hrun
- name: LS_APP_TOKEN
valueFrom:
secretKeyRef:
key: ls_app_token
name: nobl9-agent-nobl9-dev-default-hrun
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you donβt want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
- name: N9_METRICS_PORT
value: "9090"
If you use Docker, you can run the Docker command to deploy the agent. It will look something like this:
# DISCLAIMER: This Docker command contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply command, and you will need to replace the placeholder values with your own values.
docker run -d --restart on-failure \
--name nobl9-agent-nobl9-dev-mysnowobs-snowobsagent \
-e N9_CLIENT_ID="unique_client_id" \
-e N9_CLIENT_SECRET="unique_client_secret" \
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you donβt want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
-e N9_METRICS_PORT=9090 \
-e LS_APP_TOKEN="<LIGHTSTEP_APP_TOKEN>" \
nobl9/agent:0.80.0
Creating SLOs with ServiceNow Cloud Observabilityβ
Nobl9 Webβ
Follow the instructions below to create your SLOs with ServiceNow Cloud Observability in the UI:
-
Navigate to Service Level Objectives.
-
Click
.
-
In step 2, select ServiceNow Cloud Observability as the data source for your SLO.
-
Specify the Metric. You can choose either a Threshold metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).
- Threshold metric
- Ratio metric
For the threshold metric, you can create SLO using one of the following metrics:
UQL query:
- Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9 using Unified Query Language (UQL) to retrieve and process your
metrics
orspans
data.
For more information, refer to the Available Metric Types -ServiceNow Cloud Observability UQL section of the documentation.
Latency Threshold metric that is the n-th percentile of latency in milliseconds:
- Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
For more information, refer to the Authentication section. - Select a Percentile.
Error Threshold metric that is a single value representing the percentage of errors:
- Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
For more information, refer to the Authentication section.
For the ratio metric, you can create SLO using one of the following metrics:
UQL query:
- Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9 using Unified Query Language (UQL) to retrieve and process your
metrics
orspans
data.
For more information, refer to the Available Metric Types - ServiceNow Cloud Observability UQL section of the documentation.
Error Ratio metric that allows you to enter two time series to compare (for example, a count of good events and total events).
- Enter a Stream ID, that is, an ID of a metric stream created in ServiceNow Cloud Observability.
For more information, refer to the Authentication section.
tipFor more detailed metrics description, refer to the Available Metric Types section of the documentation.
-
In step 3, define a Time Window for the SLO.
-
Rolling time windows are better for tracking the recent user experience of a service.
-
Calendar-aligned windows are best suited for SLOs that are intended to map to business metrics measured on a calendar-aligned basis, such as every calendar month or every quarter.
-
In step 4, specify the Error Budget Calculation Method and your Objective(s).
- Occurrences method counts good attempts against the count of total attempts.
- Time Slicesmethod measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
- You can define up to 12 objectives for an SLO.
See the use case example and the SLO calculations guide for more information on the error budget calculation methods.
-
In step 5, add the Display name, Name, and other settings for your SLO:
- Create a composite SLO
- Set notification on data, if this option is available for your data source.
When activated, Nobl9 notifies you if your SLO hasn't received data or received incomplete data for more than 15 minutes. - Add alert policies, labels, and links, if required.
You can add up to 20 links per SLO.
-
Click Create SLO.
countMetrics
), keep in mind that the values ββresulting from that query for both good and total:- Must be positive.
- While we recommend using integers, fractions are also acceptable.
- If using fractions, we recommend them to be larger than
1e-4
=0.0001
. - Shouldn't be larger than
1e+20
.
sloctlβ
- Metrics threshold
- Latency threshold
- Error threshold
- Metrics ratio
- Error ratio
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Metrics as the configuration type:
apiVersion: n9/v1alpha
kind: SLO
metadata:
name: get-cpu-utilization-rolling
project: default
spec:
service: default
indicator:
metricSource:
name: servicenowobs
timeWindows:
- unit: Hour
count: 1
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: High
op: gt
rawMetric:
query:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
value: 0.0005
name: objective-1
target: 0.97
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Latency threshold as the configuration type:
apiVersion: n9/v1alpha
kind: SLO
metadata:
name: get-store-p95-latency-rolling
project: servicenowobs-raw
spec:
service: android-service
indicator:
metricSource:
name: servicenowobs
timeWindows:
- unit: Day
count: 7
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: Good
op: lte
rawMetric:
query:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
value: 150
target: 0.95
Hereβs an example of ServiceNow Cloud Observability using rawMetric
(threshold metric) with Error threshold as the configuration type:
apiVersion: n9/v1alpha
kind: SLO
metadata:
name: get-store-p95-latency-rolling
project: snow-obs-raw
spec:
service: android-service
indicator:
metricSource:
name: servicenowobs
timeWindows:
- unit: Day
count: 7
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: Good
op: lte
rawMetric:
query:
lightstep:
streamID: DzpxcSRh
typeOfData: error_rate
value: 0.05
target: 0.95
Hereβs an example of ServiceNow Cloud Observability using countMetrics
(ratio metric) with Metrics as the configuration type:
apiVersion: n9/v1alpha
kind: SLO
metadata:
name: avg-to-max-cpu-utilization-calendar
project: default
spec:
service: default
indicator:
metricSource:
name: servicenowobs
timeWindows:
- unit: Day
count: 7
calendar:
startTime: 2022-12-12 00:00:00
timeZone: Europe/Warsaw
budgetingMethod: Occurrences
objectives:
- displayName: Good
target: 0.95
name: objective-1
countMetrics:
incremental: false
good:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
total:
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], max
Hereβs an example of ServiceNow Cloud Observability using countMetrics
(ratio metric) with Error ratio as the configuration type:
apiVersion: n9/v1alpha
kind: SLO
metadata:
name: my-slo-name
project: my-project
spec:
service: my-service
indicator:
metricSource:
name: servicenowobs
timeWindows:
- unit: Day
count: 7
calendar:
startTime: 2020-03-09 00:00:00
timeZone: Europe/Warsaw
budgetingMethod: Occurrences
objectives:
- displayName: Good
target: 0.95
countMetrics:
incremental: false
good:
lightstep:
streamID: DzpxcSRh
typeOfData: good
total:
lightstep:
streamID: DzpxcSRh
typeOfData: total
When ServiceNow Cloud Observability is used as ratio (count) metric, then the field incremental
under spec.objectives.countMetrics
must be set to false.
Metric specification from ServiceNow Cloud Observability has three fields:
-
streamID
β mandatory, string. For instructions on how to retrieve it, go to Authentication section. -
typeOfData
β accepts one of the following values: metric, latency, error_rate, good, total. For more detailed information, refer to the Scope of support section of the documentation.Description of values for
typeOfData
fields:-
metric -
metrics
queries with which you can use ServiceNow Cloud Observability's Query Language (UQL)Β to retrieve and process your metric data by creating your query in the ServiceNow Cloud Observability UI and copying and pasting the query into Nobl9. This type can be used both forrawMetric
andcountMetrics
SLO types. -
latency β the n-th percentile (look at field percentile) of latency in milliseconds. This type can be used only as
rawMetric
. The value ofvalue
underspec.objective
must also represent milliseconds. -
error_rate β a single value representing the percentage of errors. This type can be used only as
rawMetric
. The value ofvalue
underspec.objectives
must be between0
and1
. -
good β the number of successful events (operations). It is calculated as total operations minus the number of errors. This value is only allowed in the ratio (count) metric.
-
total β the number of all events (operations). This value is only allowed in ratio (count) metric.
-
-
percentile
β number of percentiles of latency. The value must be greater than0
and less or equal to99.99
. This field is mandatory when you usetypeOfData: latency
, and is forbidden otherwise.
Available metric typesβ
You can use one of the following metric types to configure Nobl9 SLOs with ServiceNow Cloud Observability:
ServiceNow Cloud Observability UQLβ
This SLO configuration uses threshold metric method or the ratio metric under the hood.
You can use ServiceNow Cloud Observability Unified Query Language (UQL) to retrieve and process your metric data. Nobl9 supports the metrics
and spans
query types in the UQL.
Create your query in the ServiceNow Cloud Observability UI and copy and paste the query into Nobl9. Nobl9 then passes the query to the query_timeseries
ServiceNow Cloud Observability API to retrieve the time series data.
You can build the UQL queries using the following ServiceNow Cloud Observability metric kinds:
-
Gauge, an instantaneous measurement, for example,
metric memory.utilized | latest | group_by [], sum
,
spans count | delta | group_by [], sum
. -
Delta, a measurement of the change in metrics from point to point. For the delta-kind queries, you must choose one of the operators or appropriate reducer to convert the distribution values into scalar values to build SLI on it, for example,
metric request.size.bytes | delta | group_by [], sum | point dist_count(value)
For more information, refer to the Using distributions in UQL | ServiceNow Cloud Observability documentation. -
If you select a percentile as an operator in a query for the threshold metric SLI type, we recommend using the 100th percentile for best results as Nobl9 uses percentiles to display the data in the SLI chart. The following is an example
metric
query with Delta metric kind:
metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0)
-
If you define many aggregation values, Nobl9 will fetch data for the first aggregation value defined in your query. For example, Nobl9 will fetch data for the 100th percentile in the below query:
metric request.size.bytes | delta | group_by [], sum | point percentile(value, 100.0), percentile(value, 99.9)
lightstep:
typeOfData: metric
uql: metric cpu.utilization | rate | group_by [], mean
spans
queriesβ
Limitations:
ServiceNow Cloud Observability UQL spans
queries supported in the public API must have retained data in ServiceNow Cloud Observability streams.
For example, when spans
is not retained in a stream, the following query:
spans latency | delta | filter ((service == "adservice") || (service == "frontend")) | group_by [], sum | point percentile(value, 99.9)
will return the following error when querying the API:
"rpc error: code = InvalidArgument desc = public API only supports retained spans TQL queries at this time, please create a retained span query first"
However, when spans
is retained in a stream, after creating a stream for a given filter, API starts returning a metric. For example, the following UQL query will return the metric:
spans latency | delta | filter (service == "frontend") | group_by [], sum | point percentile(value, 99.9)
if service IN ("frontend")
is an existing stream.
You can test your spans
query whether it has retained data in the stream in the ServiceNow Cloud Observability API Reference documentation.
Retention period:
- for UQL
spans
queries retained in the stream, the retention period is set from 28 days, up to two years.
For more information, refer to the ServiceNow Cloud Observability documentation.
Metric YAML sample:
lightstep:
typeOfData: metric
uql: spans count | delta | group_by [], sum
Latency thresholdβ
This SLO configuration uses the threshold metric method under the hood with the SLI equal to the specific percentile value defined when configuring the SLO. For more details, go here.
Nobl9 retrieves those values from ServiceNow Cloud Observability API
under data.attributes[].latencies[]
.
These values are represented in ServiceNow Cloud Observability on the following chart (in Latency section).
![](/img/lightstep-latency.png)
Metric YAML sample:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
Error thresholdβ
This SLO configuration uses the threshold metric method under the hood with the SLI equal to the percentage of errors for a given stream.
Nobl9 retrieves values ops-counts and error-counts from ServiceNow Cloud Observability API and uses them to calculate the value:
value = error-counts / ops-counts
Such calculated values are used as an SLI for SLOs configured with this method.
They are represented in ServiceNow Cloud Observability on the following chart (in Err% section).
![](/img/lightstep-err.png)
Metric YAML sample:
lightstep:
streamID: DzpxcSRh
typeOfData: error_rate
Error ratioβ
This SLO configuration uses count (ratio) metric method under the hood. Each count metric SLO needs two data streams: good and total.
With this configuration, Nobl9 retrieves error-counts
and ops-counts
from ServiceNow Cloud Observability API and calculates those data streams as following:
Good = ops-counts - error-counts
Total = ops-counts
By default, ServiceNow Cloud Observability does not show these values on chart. It shows operations per second instead.
![](/img/lightstep-rate.png)
Nobl9 doesnβt use Rate to calculate error budgets for any SLO. Events counts are used instead (calculated from ops-counts
and error-counts
that are retrieved from the API).
Metric YAML sample:
countMetrics:
incremental: false
good:
lightstep:
streamID: DzpxcSRh
typeOfData: good
total:
lightstep:
streamID: DzpxcSRh
typeOfData: total
Querying the ServiceNow Cloud Observability APIβ
The Nobl9 agent makes calls the ServiceNow Cloud Observability API once every 60 seconds.
ServiceNow Cloud Observability API rate limitsβ
ServiceNow Cloud Observability has low rate limits for its Streams Timeseries API. For Community, Professional, and Enterprise licenses itβs 60, 200, 600 requests per hour respectively. The Nobl9 agent makes requests once every 60s, which allows for one ServiceNow Cloud Observability organization to use only 1, 3, or 10 unique metric specifications. For more information, refer to the Rate Limits | ServiceNow Cloud Observability documentation.
ServiceNow Cloud Observability users can request an increase of rate limits via ServiceNow Cloud Observability customer support.