Splunk

Reading time: 0 minute(s) (0 words)

Splunk provides software for searching, monitoring, and analyzing machine-generated data via a Web-style interface. Splunk-Nobl9 integration allows users to enter their metrics using the Splunk Processing Language (SPL).

Splunk parameters and supported features in Nobl9

General support:: Release channel: Stable, Beta; Connection method: Agent, Direct; Event logs: Supported
Query parameters:: Query interval: 1 min; Query delay: 5 min; Jitter: 20 sec; Timeout: 30 sec
Agent details and minimum required versions for supported features:: Environment variable: SPLUNK_QUERY_DELAY; Plugin name: n9splunk; Replay and SLI Analyzer: 0.55.0; Maximum historical data retrieval period: 30 days; Query parameters retrieval: 0.73.2; Timestamp cache persistence: 0.56.1
Additional notes:: Self-signed Splunk Enterprise certificates are not supported. Nobl9 requires successful certificate validation for any Splunk Enterprise instance using TLS, and self-signed certificates cannot pass this validation

Requirements

Splunk API Endpoint URL

To connect to the required Splunk instance, both direct and agent connection methods require API Endpoint URL to contain the following:

SPLUNK_BASE_URL the base URL configured during the deployment of Splunk software, for Splunk Enterprise.
PORT_NUMBER: 8089, if the API is using the default port.
Ask your Splunk administrator for the API Token and correct URL for connecting.

This URL must point to the base API URL of the Splunk Search app.

Usually, the format is {SPLUNK_BASE_URL}:{PORT_NUMBER}/services/.
So, for example, your resulting API Endpoint URL can be https://splunk.my-instance.com:8089/services/.

Typos can happen with manual entry

Here's a quick checklist to avoid request failures:

Splunk base URL: confirm it's correct with your Splunk administrator
Port: 8089 by default, or your specific port
/services/: ensure it's exactly like this

Authentication

Splunk agent deployment requires authentication. You can authenticate in either way:

With Splunk Search App REST API, using SAML.
For this, pass your Splunk App Token with the SPLUNK_APP_TOKEN environment variable.
Passing your token with a local config file under the n9splunk section.
For example
Create the cfg.toml file and specify your token as the n9splunk value:
[n9splunk] application_token="YOUR_TOKEN"
Likewise, you can use your username and password with the app_user and app_password keys.
Using the basic authentication method.
This requires passing your user credentials with the SPLUNK_USER and SPLUNK_PASSWORD environment variables at the agent startup.

Minimum required permissions

Ensure the following permissions are set for the Nobl9 agent:

The search capability
Access to index

Alternatively, you can use a wildcard:

Adding Splunk as a data source

To ensure data transmission between Nobl9 and your data source, it may be necessary to list Nobl9 IP addresses as trusted.

IP addresses to add to your allowlist:

18.159.114.21
18.158.132.186
3.64.154.26

⚠ Applies to app.nobl9.com only. In all other cases, contact Nobl9 support.

You can add the Splunk data source using the direct or agent connection methods.

Direct connection method

Direct configuration for Splunk requires users to enter their credentials, which Nobl9 stores safely.

Nobl9 Web

Follow these steps to set up a direct configuration:

Navigate to Integrations > Sources.
Click .
Click the required Source icon.
Choose Direct.

Select one of the following Release Channels:
- The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
- The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
Specify API Endpoint URL to connect to your required Splunk instance.
Example URL: https://splunk.example.com:8089/services/. Make sure it doesn't contain any typos.
Enter the Access Token generated from your Splunk instance (mandatory).

Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
Enter a Display Name.
You can enter a user-friendly name with spaces in this field.
Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in Splunk integration for Query delay is 5 minutes.
info
Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
- The maximum period value depends on the data source.
  Find the maximum value for your data source.
- A greater period can extend the loading time when creating an SLO.
Enter a Default Period for Historical Data Retrieval.
- It is used by SLOs connected to this data source.
- The value must be a positive integer or 0.
- By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
Click Add Data Source.

sloctl

To connect the Splunk data source using the direct method, apply the following configuration using the sloctl apply -f command:

apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: my-splunk-data-source
  displayName: My Splunk data source
  project: my-project
spec:
  description: Sample Splunk data source use direct connection
  releaseChannel: stable
  splunk:
    url: https://{splunk.my-org}.com/services
    accessToken: "[secret]"
  logsCollectionEnabled: true
  historicalDataRetrieval:
    maxDuration:
      value: 30
      unit: Day
    defaultDuration:
      value: 15
      unit: Day
  queryDelay:
    value: 6
    unit: Minute

Field	Type	Description
`queryDelay.unit` mandatory	enum	Specifies the unit for the query delay. Possible values: `Second` \| `Minute`. • Check query delay documentation for default unit of query delay for each source.
`queryDelay.value` mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
`logCollectionEnabled` optional	boolean	Optional. Defaults to `false`. Set to `true` if you'd like your direct to collect event logs. Beta functionality available only through direct release channel. Reach out to support@nobl9.com to activate it.
`releaseChannel` mandatory	enum	Specifies the release channel. Accepted values: `beta` \| `stable`.
Source-specific fields
`splunk.accessToken` mandatory	string, secret	Environment variable used for authentication with the Splunk Search App REST API. See authentication for more details.
`splunk.URL` mandatory	string	Base API URL of the Splunk Search app. See authentication for more details.
Replay-related fields
`historicalDataRetrieval` optional	n/a	Optional structure related to configuration related to Replay. ❗ Use only with supported sources. • If omitted, Nobl9 uses the default values of `value: 0` and `unit: Day` for `maxDuration` and `defaultDuration`.
`maxDuration.value` optional	numeric	Specifies the maximum duration for historical data retrieval. Must be integer `≥ 0`. See Replay documentation for values of max duration per data source.
`maxDuration.unit` optional	enum	Specifies the unit for the maximum duration of historical data retrieval. Accepted values: `Minute` \| `Hour` \| `Day`.
`defaultDuration.value` optional	numeric	Specifies the default duration for historical data retrieval. Must be integer `≥ 0` and `≤` `maxDuration`.
`defaultDuration.unit` optional	enum	Specifies the unit for the default duration of historical data retrieval. Accepted values: `Minute` \| `Hour` \| `Day`.

Agent connection method

Nobl9 Web

Follow the instructions below to configure your Splunk agent.

Navigate to Integrations > Sources.
Click .
Click the required Source icon.
Choose Agent.

Select one of the following Release Channels:
- The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
- The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
Specify API Endpoint URL to connect to your required Splunk instance.
Example URL: https://splunk.example.com:8089/services/. Make sure it doesn't contain any typos.

Select a Project.
Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
Enter a Display Name.
You can enter a user-friendly name with spaces in this field.
Enter a Name.
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
Enter a Description.
Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in Splunk integration for Query delay is 5 minutes.
info
Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
- The maximum period value depends on the data source.
  Find the maximum value for your data source.
- A greater period can extend the loading time when creating an SLO.
Enter a Default Period for Historical Data Retrieval.
- It is used by SLOs connected to this data source.
- The value must be a positive integer or 0.
- By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
Click Add Data Source.

sloctl

To connect the Splunk data source using the agent method, apply the following configuration using the sloctl apply -f command:

apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: my-splunk-data-source
  displayName: My Splunk data source
  project: my-project
spec:
  description: Sample Splunk data source use agent connection
  releaseChannel: stable
  splunk:
    url: https://{splunk.my-org}.com/services
  historicalDataRetrieval:
    maxDuration:
      value: 30
      unit: Day
    defaultDuration:
      value: 15
      unit: Day
  queryDelay:
    value: 6
    unit: Minute

Field	Type	Description
`queryDelay.unit` mandatory	enum	Specifies the unit for the query delay. Possible values: `Second` \| `Minute`. • Check query delay documentation for default unit of query delay for each source.
`queryDelay.value` mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
`releaseChannel` mandatory	enum	Specifies the release channel. Accepted values: `beta` \| `stable`.
Source-specific fields
`splunk.URL` mandatory	string	Base API URL of the Splunk Search app. See authentication section above for more details.
Replay-related fields
`historicalDataRetrieval` optional	n/a	Optional structure related to configuration related to Replay. ❗ Use only with supported sources. • If omitted, Nobl9 uses the default values of `value: 0` and `unit: Day` for `maxDuration` and `defaultDuration`.
`maxDuration.value` optional	numeric	Specifies the maximum duration for historical data retrieval. Must be integer `≥ 0`. See Replay documentation for values of max duration per data source.
`maxDuration.unit` optional	enum	Specifies the unit for the maximum duration of historical data retrieval. Accepted values: `Minute` \| `Hour` \| `Day`.
`defaultDuration.value` optional	numeric	Specifies the default duration for historical data retrieval. Must be integer `≥ 0` and `≤` `maxDuration`.
`defaultDuration.unit` optional	enum	Specifies the unit for the default duration of historical data retrieval. Accepted values: `Minute` \| `Hour` \| `Day`.

warning

You can deploy only one agent in one YAML file by using the sloctl apply command.

Agent deployment

When you add a data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to deploy the agent. Both configurations are available on the Nobl9 Web under the data source details > Agent configuration tab.

Agent deployment requires client_id and client_secret. To retrieve their values, run sloctl get agents -p <YOUR_PROJECT_NAME> -k.

Replace the placeholders provided in the generated configuration files with your actual Splunk values.

Minimum agent versions for single-query ratio metrics

Feature support: 0.80.0 or 0.80.0-beta
Replay and SLI Analyzer support: 0.82.2 or 0.82.0-beta

Kubernetes
Docker

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the agent. It will look something like this:

apiVersion: v1
kind: Secret
metadata:
  name: splunk-agent-data-source
  namespace: my-namespace
type: Opaque
stringData:
  splunk_app_token: "<SPLUNK_APP_TOKEN>"
  splunk_user: "<SPLUNK_USERNAME>"
  splunk_password: "<SPLUNK_PASSWORD>"
  client_id: "<AGENT_CLIENT_ID>"
  client_secret: "<AGENT_CLIENT_SECRET>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk-agent-data-source
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: "splunk"
      nobl9-agent-project: "my-project"
      nobl9-agent-organization: "my-organization"
  template:
    metadata:
      labels:
        nobl9-agent-name: "splunk"
        nobl9-agent-project: "my-project"
        nobl9-agent-organization: "nobl9-dev-stable"
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:0.82.7-beta
          resources:
            requests:
              memory: "700Mi"
              cpu: "0.2"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: splunk-agent-data-source
            - name: SPLUNK_APP_TOKEN
              valueFrom:
                secretKeyRef:
                  key: splunk_app_token
                  name: splunk-agent-data-source
            - name: SPLUNK_USER
              valueFrom:
                secretKeyRef:
                  key: splunk_user
                  name: splunk-agent-data-source
            - name: SPLUNK_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: splunk_password
                  name: splunk-agent-data-source
            - name: N9_INTAKE_URL
              value: "<YOUR_VALUE>"
            - name: N9_QUERYENGINE_URL
              value: "<YOUR_VALUE>"
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: splunk-agent-data-source
            - name: N9_AUTH_SERVER
              value: "<YOUR_VALUE>"
            - name: N9_OKTA_ORG_URL
              value: "<YOUR_VALUE>"
            - name: N9_METRICS_PORT
              value: "9090"
            - name: N9_NATS_URL
              # A wss:// URL
              value: "<YOUR_VALUE>"

If you use Docker, you can run the Docker command to deploy the agent. It will look something like this:

docker run -d --restart on-failure \
  --name splunk-agent-data-source \
  -e N9_CLIENT_ID="unique_client_id" \
  -e N9_CLIENT_SECRET="unique_client_secret" \
  # The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
  # The 9090 is the default value and can be changed.
  # If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
  -e N9_METRICS_PORT=9090 \
  -e SPLUNK_APP_TOKEN="<SPLUNK_APP_TOKEN>" \
  -e SPLUNK_USER="<SPLUNK_USERNAME>" \
  -e SPLUNK_PASSWORD="<SPLUNK_PASSWORD>" \
  nobl9/agent:0.82.7-beta

Since agent version 0.65.3, the following environment variables are available for the Splunk integration:

- name: N9_SPLUNK_COLLECTION_JITTER
  value: "15s" # Deafult: 15s
- name: N9_SPLUNK_QUERY_INTERVAL
  value: "1m" # Default: 1m
- name: N9_SPLUNK_HTTP_CLIENT_TIMEOUT
  value: "15s" # Default: 15s

Learn more about Query customization.

Creating SLOs with Splunk

Nobl9 Web

Navigate to Service Level Objectives.
Click .
Select a Service.
It will be the location for your SLO in Nobl9.
Select Splunk as the data source for your SLO.
Modify Period for Historical Data Retrieval, when necessary.
- This value will be used to replay your SLO and defines how far back in the past your data will be retrieved.
- A longer period can extend the data loading time for your SLO.
- Must be a positive whole number up to the maximum period value you've set when adding the Splunk data source.
Select the Metric type:
- Threshold Metric, where a single time series is evaluated against a threshold.
- Ratio Metric, where you enter two time series for a good and total counter.
- Ratio Metric, where you can select the query structure: Single query or Two queries.
  - Using the Single query option, you enter one query to compare both time series: for the good and total counters.
  - The Two queries option allows you to enter two time series to compare—a count of good requests and total requests.
  For the ratio metrics, select the Data Count Method:
  - Non-incremental: counts incoming metric values one-by-one. So the resulting SLO graph is pike-shaped.
  - Incremental: counts the incoming metric values incrementally, adding every next value to previous values. It results in a constantly increasing SLO graph.

Define the Query. It must be in the Splunk Search Processing Language and meet the following requirements:

Every query must contain n9 fields, and Splunk must return their values in the dataset:

The required n9 values per metric type

Threshold	Two-query ratio	Single-query ratio	Description
`n9time`	`n9time`	`n9time`	A Unix timestamp
`n9value`	`n9value`	not required	Float number
not required	not required	`n9good`	Float number
not required	not required	`n9total`	Float number

The query can retrieve data from either Events or Metrics Splunk indexes.
Nobl9 validates the queries to contain the respective n9 field along with index= with the value.
Dataset time ranges are segmented into 15-second chunks and aggregated as shown in the table:

Dataset aggregation

Threshold Ratio incremental Ratio non-incremental
Average Maximum value Sum of values

Threshold	Ratio incremental	Ratio non-incremental
Average	Maximum value	Sum of values

SLI values for good and total

When choosing the query for the ratio SLI (countMetrics), keep in mind that the values resulting from that query for both good and total:

Must be positive.
While we recommend using integers, fractions are also acceptable.

If using fractions, we recommend them to be larger than 1e-4 = 0.0001.

Shouldn't be larger than 1e+20.

Define the Time Window for your SLO:
- Rolling time windows constantly move forward as time passes. This type can help track the most recent events.
- Calendar-aligned time windows are usable for SLOs intended to map to business metrics measured on a calendar-aligned basis.
Configure the Error budget calculation method and Objectives:
- Occurrences method counts good attempts against the count of total attempts.
- Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
- You can define up to 12 objectives for an SLO.
Add the Display name, Name, and other settings for your SLO:
- Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
  Use only lowercase letters, numbers, and dashes.
- Create Composite SLO: with this option selected, you create a composite SLO 1.0. Composite SLOs 1.0 are deprecated. They're fully operable; however, we encourage you to create new composite SLOs 2.0.
  You can create composite SLOs 2.0 with sloctl using the provided template. Alternatively, you can create a composite SLO 2.0 with Nobl9 Terraform provider.
- Set Notifications on data. With it, Nobl9 will notify you in the cases when SLO won't be reporting data or report incomplete data for more than 15 minutes.
- Add alert policies, labels, and links, if required.
  Up to 20 items of each type per SLO is allowed.
Click CREATE SLO.

SLO configuration use case

Check the SLO configuration use case for a real-life SLO example.

sloctl

rawMetric
countMetric good over total
countMetric single query

Here’s an example of Splunk using a rawMetric (threshold metric):

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-splunk-threshold-slo
    project: my-project
    #displayName: My Splunk threshold SLO
    #labels:
    #  area:
    #    - latency
    #    - slow-check
    #  team:
    #    - green
    #    - sales
    #annotations:
    #  area: latency
    #  team: sales
  spec:
    #description: Sample Splunk threshold SLO
    indicator:
      metricSource:
        name: splunk
        project: my-project
        kind: Agent
    budgetingMethod: Occurrences
    objectives:
      - name: my-objective
        #displayName: My objective (200)
        value: 200.0
        target: 0.95
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        op: lte
        primary: true
    service: my-service
    timeWindows:
      - unit: Month
        count: 1
        isRolling: false
        calendar:
          startTime: 2022-12-01 00:00:00
          timeZone: UTC
    #alertPolicies:
    #  - my-alert-policy
    #attachments:
    #  - url: https://my-url.com
    #    displayName: My URL
    #anomalyConfig:
    #  noData:
    #    alertMethods:
    #      - name: my-alert-method
    #        project: my-project

Here’s an example of Splunk using a countMetric (good over total ratio metric):

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-splunk-ratio-good-over-total-slo
    project: my-project
    #displayName: My Splunk ratio SLO good over total
    #labels:
    #  area:
    #    - latency
    #    - slow-check
    #  team:
    #    - green
    #    - sales
    #annotations:
    #  area: latency
    #  team: sales
  spec:
    #description: Sample Splunk ratio good over total SLO
    indicator:
      metricSource:
        name: splunk
        project: my-project
        kind: Agent
    budgetingMethod: Timeslices
    objectives:
      - name: my-objective
        #displayName: My objective (200)
        value: 1.0
        target: 0.95
        timeSliceTarget: 0.9
        countMetrics:
          incremental: true
          good:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
          total:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog | bucket _time span=1m | stats count as n9value by _time | rename _time as n9time | fields n9time n9value
        primary: true
    service: my-service
    timeWindows:
      - unit: Month
        count: 1
        isRolling: false
        calendar:
          startTime: 2022-12-01 00:00:00
          timeZone: UTC
    #alertPolicies:
    #  - my-alert-policy
    #attachments:
    #  - url: https://my-url.com
    #    displayName: My URL
    #anomalyConfig:
    #  noData:
    #    alertMethods:
    #      - name: my-alert-method
    #        project: my-project

Here’s an example of Splunk using a countMetric (single query ratio metric):

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: my-splunk-ratio-single-query-slo
    project: my-project
    #displayName: My Splunk ratio SLO single query
    #labels:
    #  area:
    #    - latency
    #    - slow-check
    #  team:
    #    - green
    #    - sales
    #annotations:
    #  area: latency
    #  team: sales
  spec:
    #description: Sample Splunk ratio SLO single query
    indicator:
      metricSource:
        name: splunk
        project: my-project
        kind: Agent
    budgetingMethod: Timeslices
    objectives:
      - name: my-objective
        #displayName: My objective (200)
        value: 1.0
        target: 0.95
        timeSliceTarget: 0.9
        countMetrics:
          incremental: true
          goodTotal:
            splunk:
              query: |-
                | mstats avg("spl.intr.resource_usage.IOWait.data.avg_cpu_pct") as n9good WHERE index="_metrics" span=15s
                | join type=left _time [
                | mstats avg("spl.intr.resource_usage.IOWait.data.max_cpus_pct") as n9total WHERE index="_metrics" span=15s]
                | rename _time as n9time 
                | fields n9time n9good n9total
        primary: true
    service: my-service
    timeWindows:
      - unit: Hour
        count: 1
        isRolling: true
    #alertPolicies:
    #  - my-alert-policy
    #attachments:
    #  - url: https://my-url.com
    #    displayName: My URL
    #anomalyConfig:
    #  noData:
    #    alertMethods:
    #      - name: my-alert-method
    #        project: my-project

Query tips, requirements, and samples

For the best experience, keep query run time short enough to return results within 1–2 minutes. For example, when you query data for the last 15 minutes, the response should return within less than one minute.
You can check query duration using Splunk's Search Job Inspector or learn quick tips for query optimization.
Agent version requirements for single-query ratio metrics

Name Stable Beta
Single-query ratio metric 0.80.0 0.80.0-beta
Replay and SLI Analyzer 0.82.2 0.82.0-beta

Name	Stable	Beta
Single-query ratio metric	`0.80.0`	`0.80.0-beta`
Replay and SLI Analyzer	`0.82.2`	`0.82.0-beta`

Every query must contain n9 fields, and Splunk must return their values in the dataset. The n9 fields are as follows:

The required n9 values per metric type

Threshold	Two-query ratio	Single-query ratio	Description
`n9time`	`n9time`	`n9time`	A Unix timestamp
`n9value`	`n9value`	not required	Float number
not required	not required	`n9good`	Float number
not required	not required	`n9total`	Float number

Use Splunk field extractions to return values with the exact names. The n9time is the actual time, and the n9value, n9good, and n9total are metric values.

Typically, you rename _time to n9time and the field containing metric values (for example, response_time)—to the n9value, n9good, or n9total.

For example,

index=myserver-events source=udp:5072 sourcetype=syslog response_time>0
| rename _time as n9time, response_time as n9value
| fields n9time n9value

Query frequency

The Splunk query is by default executed once per minute, returning the values found in the fields n9time and n9value. Ensure your hardware can support the query frequency or consider increasing the Query Interval.

The query can retrieve data from either Events or Metrics Splunk indexes.
- Nobl9 validates the queries to contain the respective n9 field along with index= with the value.
- Dataset time ranges are segmented into 15-second chunks and aggregated. The aggregation is as follows:
  
  Dataset aggregation
  
  Threshold Ratio incremental Ratio non-incremental
  Average Maximum value Sum of values
The index attribute ("index=index_name") lets avoiding long-running queries.
- The query can retrieve data from both the Events and Metrics indexes.
- To retrieve Metrics data, use the | mstats command.
- To retrieve data from the Events and Metrics indexes, enter the SPL query and select a proper index: index=_metrics or index=_events, where _metrics is the name of the metrics index, and _events is the name of the events index.
Sample query for the Events index:

Threshold	Ratio incremental	Ratio non-incremental
Average	Maximum value	Sum of values

search index=_events sourcetype=syslog status<400
| bucket _time span=1m
| stats count as n9value by _time
| rename _time as n9time
| fields n9time n9value

Sample query for the Metrics index:

| mstats avg("my.metric") as n9value WHERE index=_metrics span=15s
| rename _time as n9time
| fields n9time n9value

Sample single query for the Metrics index:

| mstats avg("spl.intr.resource_usage.IOWait.data.avg_cpu_pct") as n9good WHERE index="_metrics" span=15s
| join type=left _time [
| mstats avg("spl.intr.resource_usage.IOWait.data.max_cpus_pct") as n9total WHERE index="_metrics" span=15s]
| rename _time as n9time
| fields n9time n9good n9total

Querying Splunk server

The Nobl9 agent leverages Splunk Enterprise API parameters. It pulls data at a per-minute interval from the Splunk server.

API rate limits for the Nobl9 agent

Splunk Enterprise API rate limits are configured by its administrators. Rate limits must be high enough to accommodate searches from the Nobl9 agent. The Nobl9 agent makes one query per minute per unique query.

Concurrent searches

For the best results, the number of concurrent searches must be about the same as the number of SLIs you have for this data source.

Number of events returned from Splunk queries

Supported search SPL command searches within indexed events. The total number of events can be large, and a query without specific conditions, such as search sourcetype=*, returns all indexed events. A large number of data points sent to Nobl9 could disrupt the system’s performance. Therefore, there is a hard limit of 4 events per minute.

File-based queries and Splunk disk quota

If you’re using file-based queries (the inputlookup function) instead of index-based queries, your query might not work as expected. Due to the difference in jitter configuration between Splunk and Nobl9, you might need to increase your Splunk disk quota for the inputlookup function to work properly.

To determine the appropriate disk quota size for your Splunk account, we recommend the following steps:

Go to the Splunk UI and navigate to Activity > Jobs.
Filter the logs by the user you currently use in Nobl9 App.
Execute requests at 29-second intervals to gather all logs from the corresponding cycle.
Sum the sizes of all requests from the list to determine the minimum disk quota. It is important to add a buffer to this number for safety.
Once you have created more SLOs, adjust the disk quota accordingly.

We suggest increasing the quota to 2GB to resolve the issue. However, it’s important to note that the final disk quota size will depend on the data being queried.

Known limitations

Query limitations:

earliest and latest are not allowed in the Time Range Modifiers search command.

Useful links

For a more in-depth look, consult additional resources:

REST accessSplunk docs

Create auth tokensSplunk docs

Agent metricsAgent

Creating SLOs via TerraformTerraform

Creating agents via TerraformTerraform

Requirements​

Splunk API Endpoint URL​

Authentication​

Minimum required permissions​

Adding Splunk as a data source​

Direct connection method​

Nobl9 Web​

sloctl​

Agent connection method​

Nobl9 Web​

sloctl​

Agent deployment​

Creating SLOs with Splunk​

Nobl9 Web​

sloctl​

Query tips, requirements, and samples​

Querying Splunk server​

API rate limits for the Nobl9 agent​

Number of events returned from Splunk queries​

File-based queries and Splunk disk quota​

Known limitations​

Useful links​

Requirements

Splunk API Endpoint URL

Authentication

Minimum required permissions

Adding Splunk as a data source

Direct connection method

Nobl9 Web

sloctl

Agent connection method

Nobl9 Web

sloctl

Agent deployment

Creating SLOs with Splunk

Nobl9 Web

sloctl

Query tips, requirements, and samples

Querying Splunk server

API rate limits for the Nobl9 agent

Number of events returned from Splunk queries

File-based queries and Splunk disk quota

Known limitations

Useful links