Splunk Observability
Splunk Observability allows users to search, monitor, and analyze machine-generated big data. Splunk Observability enables collecting and monitoring metrics, logs, and traces from common data sources. Data collection and monitoring in one place enables full-stack, end-to-end observability of the entire infrastructure.
Splunk Observability is different from the Splunk Core that powers Splunk Cloud / Enterprise and is the traditional log management solution from Splunk. Nobl9 also integrates to that through a different set of APIs.
Authentication
SplunkObservability is SaaS but the URL which indicates the realm (region) needs to be provided. For more details, refer to Realms in Endpoints | Splunk Observability Documentation.
When deploying the N9 agent for SplunkObservability, it is required to provide
SPLUNK_OBSERVABILITY_ACCESS_TOKEN
as an environment variable for authentication with organization API Access Token (see Create an Access Token | Splunk Observability Documentation). There is a placeholder for that value in configuration obtained from installation instructions in the Nobl9 UI (refer to the Agent Configuration in the UI section).
Adding Splunk Observability Realm
Splunk Observability connection also requires entering your organization’s Realm. Follow the below instructions to get your API endpoint for the Realm in Splunk:
In your Splunk account, go to Settings > Profile.
Go to the Endpoints section
Choose the
URL
from theAPI field
.

Access tokens are valid for 30 days.
Customers could use Org tokens which are valid for 5 years. Org tokens can also be used to generate session tokens
- Sample access token for Splunk Observability:
t4QJpMY1XLcECzm1c5Jb0A
- Sample access token for Splunk Observability:
Adding Splunk Observability as a Data Source in the UI
To add Splunk Observability as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:
- Navigate to Integrations > Sources.
- Click the
button.
- Click the relevant Source icon.
- Choose a relevant connection method (Agent or Direct), then configure the source as described below.
Splunk Observability Direct
Direct Configuration in the UI
Direct connection to Splunk Observability requires users to enter their credentials which Nobl9 stores safely. To set up this type of connection:
Enter your organization's Realm to connect your data source.
Refer to the Authentication section above for more details.Enter the Access Token environment variable for authentication with the organization API Access Token.
Refer to the Authentication section above for more details.
- Select a Project.
Specifying a Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank then object is assigned to projectdefault
. - Enter a Display Name.
You can enter a friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example,my-project-name
). This field is populated automatically when you enter a display name, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Click the Add Data Source
Splunk Observability Agent
Agent Configuration in the UI
Follow the instructions below to configure your Splunk Observability Agent. Refer to the section above for the description of the fields.
- Enter your organization's Realm to connect your data source.
- Enter a Project.
- Enter a Display Name.
- Enter a Name.
- Create a Description.
- Click the Add Data Source
Agent Using CLI - YAML
The YAML for setting up an Agent connection to Splunk Observability looks like this:
apiVersion: n9/v1alpha
kind: Agent
metadata:
name: splunk-observability
displayName: Splunk Observability # optional
project: splunk-observability
spec:
description: Agent settings for Splunk Observability # optional
sourceOf:
- Metrics
- Services
splunkObservability:
realm: us1
Important notes:
- The
realm
is required. See Realms in Endpoints | Splunk Observability Documentation.
You can deploy only one Agent in one YAML file by using the sloctl apply
command.
Deploying Splunk Observability Agent
When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace the <SPLUNK_OBSERVABILITY_ACCESS_TOKEN>
with your organization key).
- Kubernetes
- Docker
If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:
# DISCLAIMER: This Deployment description is containing only the necessary fields for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description and the client_id as well as the client_secret are only exemplary values.
apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-dwq-ble
namespace: default
type: Opaque
stringData:
splunk_observability_access_token: "<SPLUNK_OBSERVABILITY_ACCESS_TOKEN>"
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-splunkobs-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "splunkobs"
nobl9-agent-project: "deployment"
nobl9-agent-organization: "nobl9-dev"
template:
metadata:
labels:
nobl9-agent-name: "splunkobs"
nobl9-agent-project: "deployment"
nobl9-agent-organization: "nobl9-dev"
spec:
containers:
- name: agent-container
image: nobl9/agent:latest
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-splunkobs-deployment
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-dwq-ble
- name: SPLUNK_OBSERVABILITY_ACCESS_TOKEN
valueFrom:
secretKeyRef:
key: splunk_observability_access_token
name: nobl9-agent-nobl9-dev-dwq-ble
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
- name: N9_METRICS_PORT
value: "9090"
If you use Docker, you can run the Docker command to deploy the Agent. It will look something like this:
# DISCLAIMER: This Docker command contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply command, and you will need to replace the placeholder values with your own values.
docker run -d --restart on-failure \
--name nobl9-agent-nobl9-dev-splunkobs_deployment \
-e N9_CLIENT_ID="unique_client_id" \
-e N9_CLIENT_SECRET="unique_client_secret" \
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
-e N9_METRICS_PORT=9090 \
-e SPLUNK_OBSERVABILITY_ACCESS_TOKEN="<SPLUNK_OBSERVABILITY_ACCESS_TOKEN>" \
nobl9/agent:latest
Enabling Log Sampling for the Splunk Observability Agent
With the Agent release 0.50.0, we introduced a separate logging mechanism for Splunk Observability Agent to handle burstable log loads. This mechanism only works for redundant points dropping information - other logs are logged normally.
You can decide whether you want to use log sampling or not by setting SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG environment variable. It's a JSON object with the following fields:
- JSON
- Kubernetes - logs
- Alternative config
{
"burst": int, // how many messages?
"period": int, // how often? (in seconds)
"enabled": bool,
}
Here's an example of Kubernetes deployment YAML with enabled burst logs:
# DISCLAIMER: This Deployment description is containing only the necessary fields for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description and the client_id as well as the client_secret are only exemplary values.
apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-dwq-ble
namespace: default
type: Opaque
stringData:
splunk_observability_access_token: "<SPLUNK_OBSERVABILITY_ACCESS_TOKEN>"
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-splunkobs-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "splunkobs"
nobl9-agent-project: "deployment"
nobl9-agent-organization: "nobl9-dev"
template:
metadata:
labels:
nobl9-agent-name: "splunkobs"
nobl9-agent-project: "deployment"
nobl9-agent-organization: "nobl9-dev"
spec:
containers:
- name: agent-container
image: nobl9/agent:latest
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-splunkobs-deployment
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-dwq-ble
- name: SPLUNK_OBSERVABILITY_ACCESS_TOKEN
valueFrom:
secretKeyRef:
key: splunk_observability_access_token
name: nobl9-agent-nobl9-dev-dwq-ble
- name: N9_METRICS_PORT
value: "9090"
- name: SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG
value: '{ \"burst\": 3, \"period\": 120, \"enabled\": true}'
Here's an alternative way to enable burst logs via a YAML config:
- name: SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG
value: '{"enabled":true}'
The above YAMLs default .enabled
to false
so that Agents by default don't use it.
If only the .enabled
variable is set to true, it defaults .burst
to 1
, and .period
to 900
, which is an equivalent to log 1 message each 15 minutes per organization
.
Here's an example of configuration that allows to log 3 messages per 120 seconds per organization:
"{ \"burst\": 3, \"period\": 120, \"enabled\": true}"
Creating SLOs with Splunk Observability
Creating SLOs in the UI
Follow the instructions below to create your SLOs with Splunk Observability in the UI:
Navigate to Service Level Objectives.
- Click the
button.
In step 2, select Splunk Observability as the Data Source for your SLO, then specify the Metric. You can choose either a Threshold Metric, where a single time series is evaluated against a threshold, or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).
noteFor the Ratio Metric, you can choose the Data Count Method:
- For the Non-incremental method, we expect it to be the components of the sum.
- For the Incremental method, we expect the value of a metric to be the current sum of some numerator.
For more information, refer to the SLO Calculations Guide.
Enter a Program (for the Threshold metric), or Program for good counter, and Program for total counter (for the Count metric). The following are program examples:
Threshold metric for Splunk Observability:
A = data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish(label='A', enable=False);
B = data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').stddev().publish(label='B', enable=False);
C = (B/A).publish(label='C');Ratio metric for Splunk Observability:
Program for good counter:
data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'),rollup='rate').stddev().publish()
Program for total counter:
data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish()
In step 3, define a Time Window for the SLO.
In step 4, specify the Error Budget Calculation Method and your Objective(s).
In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.
When you’re done, click Create SLO.
SLOs using Splunk Observability - YAML samples
- rawMetric
- countMetric
Here’s an example of Splunk Observability using a rawMetric
(Threshold metric):
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: tokyo-server-4-latency
displayName: Server4 Latency [Tokyo]
project: splunk-observability
spec:
description: Latency of Server4 in Tokyo ragion
service: splunk-observability-demo-service
indicator:
metricSource:
name: splunk-observability
timeWindows:
- unit: Day
count: 1
calendar:
startTime: 2020-01-21 12:30:00
timeZone: America/New_York
budgetingMethod: Occurrences
objectives:
- displayName: Excellent
op: lte
rawMetric:
query:
splunkObservability:
program: 'data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish()'
value: 200
target: 0.8
- displayName: Good
op: lte
rawMetric:
query:
splunkObservability:
program: 'data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish()'
value: 250
target: 0.9
- displayName: Poor
op: lte
rawMetric:
query:
splunkObservability:
program: 'data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish()'
value: 300
target: 0.99
Here’s an example of Splunk Observability using a countMetric
(Ratio metric):
- apiVersion: n9/v1alpha
kind: SLO
metadata:
displayName: Splunk Observability demo
name: splunk-obs-demo
project: splunk-observability
spec:
budgetingMethod: Occurrences
indicator:
metricSource:
kind: Agent
name: splunk-observability
project: splunk-observability
objectives:
- countMetrics:
incremental: false
good:
splunkObservability:
program: data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'),rollup='rate').stddev().publish()
total:
splunkObservability:
program: data('demo.trans.count', filter=filter('demo_datacenter', 'Tokyo'), rollup='rate').mean().publish()
displayName: Enough
target: 0.5
value: 1
service: splunk-observability-demo-service
timeWindows:
- count: 1
isRolling: true
period:
begin: "2021-05-05T10:39:55Z"
end: "2021-05-05T11:39:55Z"
unit: Hour
Important notes:
Metric specification from SplunkObservability has 1 field:
program
– it is a SignalFlow analytics program and is mandatory (string). Search criteria that return exactly one time series. Program needs to return only one key in the data map (one time series). For more details, see the Query Examples section.
Query Examples
For details on Splunk Observability queries syntax, check Signalflow | Splunk Observability Documentation.
Querying the Splunk Observability Server
Nobl9 queries Splunk observability 4 data points every minute, resulting in a 15-second resolution.
Splunk Observability API Rate Limits
You can control your resource usage using org token (Access Tokens) limits. For more information, refer to the Org token limits | Splunk Observability Documentation and the System limits for Splunk Infrastructure Monitoring | Splunk Observability Documentation.
Useful Links
Splunk Observability Cloud Documentation | Splunk Observability Documentation
Create an Access Token | Splunk Observability Documentation
Realms in Endpoints | Splunk Observability Documentation
Signalflow | Splunk Observability Documentation
Org token limits | Splunk Observability Documentation
System limits for Splunk Infrastructure Monitoring | Splunk Observability Documentation
Agent Metrics | Nobl9 Documentation
Creating SLOs via Terraform | Nobl9 Terraform Documentation
Creating Agents via Terraform | Nobl9 Terraform Documentation