Splunk
Splunk provides software for searching, monitoring, and analyzing machine-generated data via a Web-style interface. Splunk integration with Nobl9 allows users to enter their metrics using the Splunk Processing Language (SPL).
Scope of Support
Nobl9 does not support a self-signed Splunk Enterprise. Nobl9 Agent requires that if Splunk Enterprise is configured to use TLS then it must successfully pass certificate validation which self-signed certificates do not.
Authentication
Splunk configuration for the Agent only accepts a single parameter: url
. The url
has to point to the base API URL of the Splunk Search app. It will usually have a form of {SPLUNK_BASE_URL}:{PORT_NUMBER}/services
where:
SPLUNK_BASE_URL
- for Splunk Enterprise, the base URL is configured during the deployment of Splunk software.PORT_NUMBER
- Assuming the API is using the default port is8089
. It is recommended that you contact your Splunk Admin to get your API Token and to verify the correct URL to connect.
When deploying the Agent for Splunk, you can use one of the following authentication methods:
SAML
: provideSPLUNK_APP_TOKEN
as environment variables for authentication with the Splunk Search App REST API.basic
: to use basic type, the Agent requiresSPLUNK_USER
andSPLUNK_PASSWORD
passed as an environment variables during agent startup.
For more details, refer to the How to Obtain Value for SPLUNK_APP_TOKEN | Splunk Documentation.
Alternatively, you can pass the token using a local config file with key application_token
under the n9splunk
section. You can give your username and password the same way, and the keys are app_user
and app_password
.
Adding Splunk as a Data Source in the UI
To add Splunk as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:
- Navigate to Integrations > Sources.
- Click the
button.
- Click the relevant Source icon.
- Choose a relevant connection method (Agent or Direct), then configure the source as described below.
Splunk Direct
Direct Configuration in the UI
Direct configuration for Splunk requires users to enter their credentials, which Nobl9 stores safely. Follow these steps to set up a direct configuration:
To access your Splunk Cloud Platform deployment using the Splunk REST API, you must submit a case requesting access using the Splunk Support Portal Splunk Support opens port 8089
for REST access. You can specify a range of IP addresses to control who can access the REST API.
You will need to allow the following IPs:
18.159.114.21
18.158.132.186
3.64.154.26
Enter an API Endpoint URL to connect to your data source (mandatory).
Refer to the Authentication section above for more details.Enter the Access Token generated from your Splunk instance (mandatory).
Refer to the Authentication section above for more details.
- Select a Project.
Specifying a Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank then object is assigned to projectdefault
. - Enter a Display Name.
You can enter a friendly name with spaces in this field. - Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example,my-project-name
). This field is populated automatically when you enter a display name, but you can edit the result. - Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it. - Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved.
- The value for the Maximum period of data retrieval depends on the Source. Check the Replay documentation for details.
- Entering a more extended Period might slow down the loading time when creating an SLO.
- The value must be a positive integer.
- Enter a Default Period for Historical Data Retrieval.
- It is the Period that will be used by the SLO connected to this data source.
- The value must be a positive integer or
0
. - By default, this value is set to 0. If you set it to
>0
, you will create an SLO with Replay.
- Click the Add Data Source
Splunk Agent
Agent Configuration in the UI
Follow the instructions below to configure your Splunk Agent. Refer to the section above for the description of the fields.
Add the URL to connect to your data source.
Example URL:https://splunk.example.com:8089/services
. It is recommended that your contact your Splunk Admin to get your API Token and to verify the correct URL to connect.Enter a Project (mandatory).
Enter a Display name (optional).
Enter a Name (mandatory).
Enter a Description (optional).
In the Advanced settings, you can:
Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved.
- The maximum Period of retrieving the data for Splunk cannot exceed 30 days.
- Entering a more extended Period might slow down the loading time when creating an SLO.
- The value must be a positive integer.
Enter a Default Period for Historical Data Retrieval.
- It is the Period that will be used by the SLO connected to this data source.
- The value must be a positive integer.
- Click the Add Data Source
Agent Using CLI - YAML
The YAML for setting up an Agent connection to Splunk looks like this:
apiVersion: n9/v1alpha
kind: Agent
metadata:
name: splunk
project: splunk
spec:
sourceOf:
- Metrics
- Services
splunk:
url: https://splunk.example.com
historicalDataRetrieval:
maxDuration:
value: 30 # integer greater than or equal to 0
unit: Day # accepted values: Minute, Hour, Day
defaultDuration: # value must be less than or equal to value of maxDuration
value: 0 # integer greater than or equal to 0
unit: Day # accepted values: Minute, Hour, Day
Important notes:
url
- base API URL of the Splunk Search app. For more details, refer to the Authentication section above.spec[n].historicalDataRetrieval
- refer to Replay Documentation | Nobl9 Documentation for more details.
You can deploy only one Agent in one YAML file by using the sloctl apply
command.
Deploying Splunk Agent
When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <SPLUNK_APP_TOKEN>
with your organization key).
- Kubernetes
- Docker
If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:
# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.
apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-dev-default-splunkagent
namespace: default
type: Opaque
stringData:
splunk_app_token: "<SPLUNK_APP_TOKEN>"
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-dev-default-splunkagent
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "splunkagent"
nobl9-agent-project: "default"
nobl9-agent-organization: "nobl9-dev"
template:
metadata:
labels:
nobl9-agent-name: "splunkagent"
nobl9-agent-project: "default"
nobl9-agent-organization: "nobl9-dev"
spec:
containers:
- name: agent-container
image: nobl9/agent:latest
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-dev-default-splunkagent
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-dev-default-splunkagent
- name: SPLUNK_APP_TOKEN
valueFrom:
secretKeyRef:
key: splunk_app_token
name: nobl9-agent-nobl9-dev-default-splunkagent
- name: SPLUNK_USER
valueFrom:
secretKeyRef:
key: splunk_user
name: nobl9-agent-nobl9-dev-default-splunkagent
- name: SPLUNK_PASSWORD
valueFrom:
secretKeyRef:
key: splunk_password
name: nobl9-agent-nobl9-dev-default-splunkagent
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
- name: N9_METRICS_PORT
value: "9090"
If you use Docker, you can run the Docker command to deploy the Agent. It will look something like this:
# DISCLAIMER: This Docker command contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply command, and you will need to replace the placeholder values with your own values.
docker run -d --restart on-failure \
--name nobl9-agent-nobl9-dev-default-splunkagent \
-e N9_CLIENT_ID="unique_client_id" \
-e N9_CLIENT_SECRET="unique_client_secret" \
# The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
# The 9090 is the default value and can be changed.
# If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
-e N9_METRICS_PORT=9090 \
-e SPLUNK_APP_TOKEN="<SPLUNK_APP_TOKEN>" \
-e SPLUNK_USER="<SPLUNK_USERNAME>" \
-e SPLUNK_PASSWORD="<SPLUNK_PASSWORD>" \
nobl9/agent:latest
Creating SLOs with Splunk
Creating SLOs in the UI
Navigate to Service Level Objectives.
- Click the
button.
In step 1 of the SLO wizard, select the Service the SLO will be associated with.
In step 2, select Splunk as the Data Source for your SLO, then specify the Metric. You can choose either a Threshold Metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).
noteFor the Ratio Metric, you can choose the Data Count Method:
- For the Non-incremental method, we expect it to be the components of the sum.
- For the Incremental method, we expect the value of a metric to be the current sum of some numerator.
For more information, refer to the SLO Calculations Guide.
Enter a Query, Good Query, or Total Query for the metric you selected:
Every query must return
n9time
andn9value
fields.- The
n9time
field must be a Unix timestamp, andn9value
field must be a float value.
- The
Nobl9 validates the query provided by the user against 3 rules:
The query contains
index=
with a value.The query contains the
n9time
value.The query contains an
n9value
value.
Every time range of the dataset is segmented into 15-second chunks and aggregated. The aggregation is as follows:
Raw metric: calculates the average.
Count metric incremental: takes the max value.
Count metric non-incremental: the sum of values.
In step 3, define a Time Window for the SLO.
In step 4, specify the Error Budget Calculation Method and your Objective(s).
In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.
When you’re done, click Create SLO.
SLOs using Splunk - YAML samples
- rawMetric
- countMetric
Here’s an example of Splunk using a rawMetric
(Threshold metric):
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: splunk-raw-rolling
project: splunk
spec:
service: splunk-service
indicator:
metricSource:
name: splunk
timeWindows:
- unit: Day
count: 7
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: Good
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 0.25
target: 0.50
- displayName: Moderate
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 0.5
target: 0.75
- displayName: Annoying
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 1.0
target: 0.95
---
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: splunk-raw-calendar
project: splunk
spec:
service: splunk-service
indicator:
metricSource:
name: splunk
timeWindows:
- unit: Day
count: 7
calendar:
startTime: 2020-03-09 00:00:00
timeZone: Europe/Warsaw
budgetingMethod: Occurrences
objectives:
- displayName: Good
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 0.25
target: 0.50
- displayName: Moderate
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 0.5
target: 0.75
- displayName: Annoying
op: lte
rawMetric:
query:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
value: 1.0
target: 0.95
Here’s an example of Splunk using a countMetric
(Ratio metric):
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: splunk-counts-rolling
project: splunk
spec:
service: splunk-service
indicator:
metricSource:
kind: Agent
name: splunk
project: splunk
timeWindows:
- unit: Hour
count: 1
isRolling: true
budgetingMethod: Occurrences
objectives:
- displayName: Poor
target: 0.50
countMetrics:
incremental: false
good:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
total:
splunk:
query: index=* source=udp:5072 sourcetype=syslog | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
- apiVersion: n9/v1alpha
kind: SLO
metadata:
name: splunk-counts-calendar
project: splunk
spec:
service: splunk-service
indicator:
metricSource:
kind: Agent
name: splunk
project: splunk
timeWindows:
- unit: Day
count: 1
calendar:
startTime: 2021-04-09 00:00:00
timeZone: Europe/Warsaw
budgetingMethod: Occurrences
objectives:
- displayName: So so
target: 0.80
countMetrics:
incremental: false
good:
splunk:
query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
total:
splunk:
query: index=* source=udp:5072 sourcetype=syslog | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
Splunk queries require:
Defining an
index
attribute ("index=index_name"
) to avoid long-running queries.The query can retrieve data from both, the Events and Metrics indexes.
You can retrieve Metrics data by using the
| mstats
command.To retrieve data from the Events and Metrics indexes, you must enter SPL query and select a proper index:
index=_metrics
orindex=_events
, where_metrics
is the name of the metrics index, and_events
is the name of the events index. For more information on the SPL query, refer to the About the Search Language | Splunk Documentation.Query example for Events index:
search index=_events sourcetype=syslog status<400
| bucket _time span=1m
| stats count as n9value by _time
| rename _time as n9time
| fields n9time n9valueQuery example for Metrics index:
| mstats avg("my.metric") as n9value WHERE index=_metrics span=15s
| rename _time as n9time
| fields n9time n9value
A return value for
n9time
andn9value
.
Use Splunk field extractions to return values using those exact names. Then9time
is the actual time, and then9value
is the metric value. Then9time
must be a Unix timestamp and then9value
must be a float value.Example:
index=myserver-events source=udp:5072 sourcetype=syslog response_time>0
| rename _time as n9time, response_time as n9value
| fields n9time n9valueTypically, you will rename
_time
ton9time
and then rename the field containing the metric value (response_time
in the previous example) to then9value
. The following is the appendage to your normal query that handles this.| rename _time as n9time, response_time as n9value
| fields n9time n9value- The Splunk query will be executed once every minute, returning the values found in the fields
n9time
andn9value
. Ensure your hardware can support the query frequency.
- The Splunk query will be executed once every minute, returning the values found in the fields
Querying Splunk Server
The Nobl9 agent leverages Splunk Enterprise API parameters. It pulls data at a per-minute interval from the Splunk server.
API Rate Limits for the Nobl9 Agent
Splunk Enterprise API rate limits are configured by its administrators. Rate limits have to be high enough to accommodate searches from Nobl9 Agent. The Nobl9 agent makes one query per minute per unique query
.
Number of events returned from Splunk queries
Supported search SPL command searches within indexed events. The total number of events can be large, and a query without specific conditions, such as search sourcetype=*
, returns all indexed events. A large number of data points sent to Nobl9 could disrupt the system’s performance. Therefore, there is a hard limit of 4 events per minute.
Known Limitations
Queries limitations:
- Within search command Time Range Modifiers | Splunk Documentation
earliest
andlatest
are not allowed.
Useful Links
REST Access | Splunk Documentation
Create Auth Tokens | Splunk Documentation
Agent Metrics | Nobl9 Documentation
Creating SLOs via Terraform | Nobl9 Terraform Documentation
Creating Agents via Terraform | Nobl9 Terraform Documentation