Skip to main content

Lightstep

Lightstep is an observability platform that enables distributed tracing, that can be used to rapidly pinpoint the causes of failures and poor performance across the deeply complex dependencies among services, teams, and workloads in modern production systems. Nobl9 integration with Lightstep enables organizations to establish service level objectives from performance data captured through distributed traces in the Lightstep platform.

Scope of Support

Lightstep in Nobl9 collects three types of metrics:

  • Latency Threshold - the n-th percentile of latency in milliseconds. For more details, go here.

  • Error Threshold - a single value representing the total number of reported errors against a threshold.

  • Error Ratio - comparing two time series to compare (for example, a count of good requests and total requests).

    • good
    • total

Authentication

Before making an API call Lightstep, Nobl9 needs to pass the following credentials:

  • Lightstep Organization ID: you can retrieve the organization ID from the Settings tab in Lightstep UI.

  • Lightstep Project Name. For details on how to create your Lightstep Projects, go here.

  • Lightstep Stream ID. It is an ID of a metric stream created in Lightstep. There are several ways to obtain the Stream ID (refer to How to Obtain Stream ID? for more details):

Lightstep API Token

When deploying the Nobl9 agent, it is required to provide LS_APP_TOKEN environment variable for authentication with the Lightstep Streams Timeseries API. There is a placeholder for that value in the configuration obtained from installation instructions in the Nobl9 web app. For more details, go here.

When setting up the Direct connection to Lightstep, the LS_APP_TOKEN environment variable must be specified indirectly in the Lightstep App Token field.

Adding Lightstep as a Data Source in the UI

To add Lightstep as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:

  1. Navigate to Integrations > Sources.

  2. Click the plus button button.

  3. Click the Lightstep icon.

  4. Choose a configuration method (Direct or Agent), then configure the source as described below.

Lightstep Direct

Direct Configuration in the UI

Direct connection to Lightstep requires users to enter their credentials which Nobl9 stores safely. To set up this type of connection:

  1. Enter the name of the Lightstep Organization to connect to your data source (mandatory).
    For more details, refer to the Authentication section above.

  2. Enter a name in the Lightstep Project field (mandatory).
    For more details, refer to the Authentication section above.

  3. Enter the Lightstep App Token (mandatory).
    For details on how to create the app token, refer to the Lightstep API Token section above.

  4. Select a Project (mandatory).
    Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, a default value appears.

  5. Enter a Display name (optional).
    You can enter a friendly name with spaces in this field.

  6. Enter a Name (mandatory).
    The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example, my-project-name). This field is populated automatically when you enter a display name, but you can edit the result.

  7. Enter a Description (optional).
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.

  8. Click the Add Data Source button.

Lightstep Agent

Agent Configuration in the UI

Follow the instructions below to set up an Agent connection. Refer to the section above for the descriptions of the fields.

  1. Enter the name of the Lightstep Organization to connect to your data source (mandatory).

  2. Enter a name in the Lightstep Project field (mandatory).

  3. Select a Project (mandatory).

  4. Enter a Display name (optional).

  5. Enter a Name (mandatory).

  6. Enter a Description (optional).

  7. Click the Add Data Source button.

Deploying Lightstep Agent

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <LIGHTSTEP_APP_TOKEN> with your organization key).

note

The Nobl Agent by default uses the following API endpoint: https://api.lightstep.com. You can override it by specifying LS_API_URL environment variable during Agent start up.

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:

# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-mylightstep-lightstepagent
namespace: default
type: Opaque
stringData:
ls_app_token: "<LIGHTSTEP_APP_TOKEN>"
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-default-hrun
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "lightstepagent"
nobl9-agent-project: "mylightstep"
nobl9-agent-organization: "nobl9-dev"
template:
metadata:
labels:
nobl9-agent-name: "lightstepagent"
nobl9-agent-project: "mylightstep"
nobl9-agent-organization: "nobl9-dev"
spec:
containers:
- name: agent-container
image: nobl9/agent:latest
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-default-hrun
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-default-hrun
- name: LS_APP_TOKEN
valueFrom:
secretKeyRef:
key: ls_app_token
name: nobl9-agent-nobl9-dev-default-hrun

Creating SLOs with Lightstep

Creating SLOs in the UI

Follow the instructions below to create your SLOs with Lightstep in the UI:

  1. Navigate to Service Level Objectives.

  2. Click the plus button button.

  3. In step 2, select Lightstep as the Data Source for your SLO, then specify the Metric. You can choose either:

    • Latency Threshold metric that is the n-th percentile of latency in milliseconds.
    • Error Threshold metric that is a single value representing the total number of reported errors against a threshold.
    • Error Ratio metric that allows you to enter two time series to compare (for example, a count of good requests and total requests).
  4. In step 3, define a Time Window for the SLO.

  5. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  6. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  7. When you’re done, click Create SLO.

SLOs using Lightstep - YAML samples

Here’s an example of Lightstep using a rawMetric (Threshold Metric):

apiVersion: n9/v1alpha
kind: SLO
metadata:
name: get-store-p95-latency-rolling
project: lightstep-raw
spec:
service: android-service
indicator:
metricSource:
name: lightstep
timeWindows:
- unit: Day
count: 7
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: Good
op: lte
rawMetric:
query:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
value: 150
target: 0.50
- displayName: Moderate
op: lte
rawMetric:
query:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
value: 175
target: 0.75
- displayName: Annoying
op: lte
rawMetric:
query:
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
value: 200
target: 0.95

When Lightstep is used as ratio (count) metric, then the field incremental under spec.objectives.countMetrics must be set to false.

Metric specification from Lightstep has 3 fields:

  • streamID – mandatory, string. For instructions on how to retrieve it, go to Authentication section.

  • typeOfData – accepts one of the following values: latency, error_rate, good, total

    Description of values for typeOfData fields:

    • latency – the n-th percentile (look at field percentile) of latency in milliseconds. This means that when used Lightstep is used as a rawMetric the value of value under spec.objective must also represent milliseconds. This value is only allowed in typeOfData for Lightstep.

    • error_rate – a single value representing the total number of reported errors against a threshold, corresponds to Error Threshold in UI. This means that when used Lightstep is used as a rawMetric, the value of value is under spec.objectives must be between 0 and 1.

    • good – the number of successful requests. It is calculated as total requests minus the number of errors. This value is only allowed in the ratio (count) metric.

    • total – the number of all requests. This value is only allowed in ratio (count) metric.

  • percentile – number of percentile of latency. The value must be greater than 0 and less or equal to 99.99. This field is mandatory when typeOfData: latency and is forbidden otherwise.

Query Examples

The following are query example for each Lightstep metric types:

  • Latency
lightstep:
streamID: DzpxcSRh
typeOfData: latency
percentile: 95
  • Error_rate (Error Threshold in UI)
lightstep:
streamID: DzpxcSRh
typeOfData: error_rate
  • Good
lightstep:
streamID: DzpxcSRh
typeOfData: good
  • Total
lightstep:
streamID: DzpxcSRh
typeOfData: total

Querying the Lightstep Server

The Nobl9 Agent makes calls the Lightstep server once every 60 seconds.

Lightstep API Rate Limits

Lightstep has low rate limits for its Streams Timeseries API. For Community, Professional, and Enterprise licenses it’s 60, 200, 600 requests per hour respectively. The Nobl9 Agent makes requests once every 60s, which allows for one Lightstep organization to use only 1, 3, or 10 unique metric specifications. For more information, refer to the Rate Limits | Lightstep Documentation.

Lightstep users can request an increase of rate limits via Lightstep’s customer support.

Monitor Metric Data | Lightstep Documentation

Investigate a Latency Regression | Lightstep Documentation

Investigate an Error Rate Increase | Lightstep Documentation

Investigate a Metric Deviation | Lightstep Documentation

Create and Manage API Keys | Lightstep Documentation

Rate Limits | Lightstep Documentation