Splunk

Reading time: 0 minute(s) (0 words)

Splunk provides software for searching, monitoring, and analyzing machine-generated data via a Web-style interface. Splunk-Nobl9 integration allows users to enter their metrics using the Splunk Processing Language (SPL).

Splunk parameters and supported features in Nobl9

General support:

Release channel: Stable, Beta

Connection method: Agent, Direct

Replay and SLI Analyzer: Historical data limit 30 days

Event logs: Supported

Query checker: Not supported

Query parameters retrieval: Supported

Timestamp cache persistence: Supported

---

Query parameters:

Query interval: 1 min

Query delay: 5 min

Jitter: 20 sec

Timeout: 30 sec

---

Agent details and minimum required versions for supported features:

Plugin name: n9splunk

Query delay environment variable: SPLUNK_QUERY_DELAY

Replay and SLI Analyzer: 0.65.0

Query parameters retrieval: 0.73.2

Timestamp cache persistence: 0.65.0

Custom HTTP headers: 0.89.1-beta

---

Additional notes:

Self-signed Splunk Enterprise certificates are not supported. Nobl9 requires successful certificate validation for any Splunk Enterprise instance using TLS, and self-signed certificates cannot pass this validation

Requirements

Splunk API Endpoint URL

To connect to the required Splunk instance, both direct and agent connection methods require API Endpoint URL to contain the following:

• SPLUNK_BASE_URL the base URL configured during the deployment of Splunk software, for Splunk Enterprise.
• PORT_NUMBER: 8089, if the API is using the default port.
  Ask your Splunk administrator for the API Token and correct URL for connecting.

This URL must point to the base API URL of the Splunk Search app.

Usually, the format is {SPLUNK_BASE_URL}:{PORT_NUMBER}/services/.
So, for example, your resulting API Endpoint URL can be https://splunk.my-instance.com:8089/services/.

Typos can happen with manual entry

Here's a quick checklist to avoid request failures:

• Splunk base URL: confirm it's correct with your Splunk administrator
• Port: 8089 by default, or your specific port
• /services/: ensure it's exactly like this

Authentication

Splunk agent deployment requires authentication. You can authenticate in either way:

• With Splunk Search App REST API, using SAML.
  For this, pass your Splunk App Token with the SPLUNK_APP_TOKEN environment variable.

• Passing your token with a local config file under the n9splunk section.

  For example

  Create the cfg.toml file and specify your token as the n9splunk value:

  [n9splunk]
    application_token="YOUR_TOKEN"

  Likewise, you can use your username and password with the app_user and app_password keys.

• Using the basic authentication method.
  This requires passing your user credentials with the SPLUNK_USER and SPLUNK_PASSWORD environment variables at the agent startup.

Minimum required permissions

Ensure the following permissions are set for the Nobl9 agent:

• The search capability
• Access to index

Alternatively, you can use a wildcard:

Adding Splunk as a data source

To ensure data transmission between Nobl9 and your data source, it may be necessary to list Nobl9 IP addresses as trusted.

💻ip allowlist

IP addresses to include in your allowlist for secure access:

If you're using app.nobl9.com instance:

• 18.159.114.21
• 18.158.132.186
• 3.64.154.26

If you're using us1.nobl9.com instance:

• 34.121.54.120
• 34.123.193.191
• 34.134.71.10
• 35.192.105.150
• 35.225.248.37
• 35.226.78.175
• 104.198.44.161

You can add the Splunk data source using the direct or agent connection methods.

Direct connection method

Direct configuration for Splunk requires users to enter their credentials, which Nobl9 stores safely.

Nobl9 Web

Follow these steps to set up a direct configuration:

1. Navigate to Integrations > Sources.
2. Click .
3. Click the required Source button.
4. Choose Direct.

5. Select one of the following Release Channels:

   • The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
   • The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.

6. Specify API Endpoint URL to connect to your required Splunk instance.
   Example URL: https://splunk.example.com:8089/services/. Make sure it doesn't contain any typos.

7. Enter the Access Token generated from your Splunk instance (mandatory).

8. Select a Project.
   Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
9. Enter a Display Name.
   You can enter a user-friendly name with spaces in this field.
10. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
11. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
12. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.

    • The default value in Splunk integration for Query delay is 5 minutes.

    info

    Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
13. Enter a Maximum Period for Historical Data Retrieval.

    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
    • The maximum period value depends on the data source.
      Find the maximum value for your data source.
    • A greater period can extend the loading time when creating an SLO.
    • The value must be a positive integer.
14. Enter a Default Period for Historical Data Retrieval.

    • It is used by SLOs connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
15. Click Add Data Source.

sloctl

1. Create a YAML definition to set up a direct connection with Splunk. For this, refer to the following example:

YAML definition for the direct connection method

apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: splunk
  displayName: Splunk Direct
  project: default
spec:
  description: Example Splunk Direct
  releaseChannel: stable
  splunk:
    url: https://splunk.my-org.com/services
    accessToken: "[secret]"
  historicalDataRetrieval:
    maxDuration:
      value: 30
      unit: Day
    defaultDuration:
      value: 15
      unit: Day
    triggeredBySloCreation:
      value: 15
      unit: Day
    triggeredBySloEdit:
      value: 15
      unit: Day
  queryDelay:
    value: 6
    unit: Minute

Field	Type	Description
queryDelay.unit mandatory	enum	Specifies the unit for the query delay. Possible values: Second | Minute. • Check query delay documentation for default unit of query delay for each source.
queryDelay.value mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
logCollectionEnabled optional	boolean	Optional. Defaults to false. Set to true if you'd like your direct to collect event logs. Contact us to activate it.
releaseChannel mandatory	enum	Specifies the release channel. Accepted values: beta | stable.
Source-specific fields
splunk.accessToken mandatory	string, secret	Environment variable used for authentication with the Splunk Search App REST API. See authentication for more details.
splunk.URL mandatory	string	Base API URL of the Splunk Search app. See authentication for more details.
Replay-related fields
historicalDataRetrieval optional	n/a	Optional structure related to configuration related to Replay. ❗ Use only with supported sources. • If omitted, Nobl9 uses the default values of value: 0 and unit: Day for maxDuration and defaultDuration.
maxDuration.value optional	numeric	Specifies the maximum duration for historical data retrieval. Must be integer ≥ 0. See Replay documentation for values of max duration per data source.
maxDuration.unit optional	enum	Specifies the unit for the maximum duration of historical data retrieval. Accepted values: Minute | Hour | Day.
defaultDuration.value optional	numeric	Specifies the default duration for historical data retrieval. Must be integer ≥ 0 and ≤ maxDuration.
defaultDuration.unit optional	enum	Specifies the unit for the default duration of historical data retrieval. Accepted values: Minute | Hour | Day.

2. Apply your YAML definition using the sloctl apply command.

Agent connection method

Nobl9 Web

Follow the instructions below to configure your Splunk agent.

1. Navigate to Integrations > Sources.
2. Click .
3. Click the required Source button.
4. Choose Agent.

5. Select one of the following Release Channels:

   • The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
   • The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.

6. Specify API Endpoint URL to connect to your required Splunk instance.
   Example URL: https://splunk.example.com:8089/services/. Make sure it doesn't contain any typos.

7. Select a Project.
   Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
8. Enter a Display Name.
   You can enter a user-friendly name with spaces in this field.
9. Enter a Name.
   The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
10. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
11. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.

    • The default value in Splunk integration for Query delay is 5 minutes.

    info

    Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
12. Enter a Maximum Period for Historical Data Retrieval.

    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
    • The maximum period value depends on the data source.
      Find the maximum value for your data source.
    • A greater period can extend the loading time when creating an SLO.
    • The value must be a positive integer.
13. Enter a Default Period for Historical Data Retrieval.

    • It is used by SLOs connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
14. Click Add Data Source.
Deploy your agent in a Kubernetes cluster or Docker container.

sloctl

1. Create a YAML definition to set up an agent connection with Splunk. For this, refer to the following example:

YAML definition for the agent connection method

apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: splunk
  displayName: Splunk Agent
  project: default
spec:
  description: Example Splunk Agent
  releaseChannel: stable
  splunk:
    url: https://splunk.my-org.com/services
  historicalDataRetrieval:
    maxDuration:
      value: 30
      unit: Day
    defaultDuration:
      value: 15
      unit: Day
  queryDelay:
    value: 6
    unit: Minute

Field	Type	Description
queryDelay.unit mandatory	enum	Specifies the unit for the query delay. Possible values: Second | Minute. • Check query delay documentation for default unit of query delay for each source.
queryDelay.value mandatory	numeric	Specifies the value for the query delay. • Must be a number less than 1440 minutes (24 hours). • Check query delay documentation for default unit of query delay for each source.
releaseChannel mandatory	enum	Specifies the release channel. Accepted values: beta | stable.
Source-specific fields
splunk.URL mandatory	string	Base API URL of the Splunk Search app. See authentication section above for more details.
Replay-related fields
historicalDataRetrieval optional	n/a	Optional structure related to configuration related to Replay. ❗ Use only with supported sources. • If omitted, Nobl9 uses the default values of value: 0 and unit: Day for maxDuration and defaultDuration.
maxDuration.value optional	numeric	Specifies the maximum duration for historical data retrieval. Must be integer ≥ 0. See Replay documentation for values of max duration per data source.
maxDuration.unit optional	enum	Specifies the unit for the maximum duration of historical data retrieval. Accepted values: Minute | Hour | Day.
defaultDuration.value optional	numeric	Specifies the default duration for historical data retrieval. Must be integer ≥ 0 and ≤ maxDuration.
defaultDuration.unit optional	enum	Specifies the unit for the default duration of historical data retrieval. Accepted values: Minute | Hour | Day.

1. Apply your YAML definition using the sloctl apply command.
2. Deploy your agent in a Kubernetes cluster or Docker container.

Useful links

For a more in-depth look, consult additional resources:

REST accessSplunk docsCreate auth tokensSplunk docsAgent metricsAgentCreating SLOs via TerraformTerraformCreating agents via TerraformTerraform