Skip to main content

Splunk Observability
On demand

Reading time: 0 minute(s) (0 words)

Splunk Observability allows users to search, monitor, and analyze machine-generated big data. Splunk Observability facilitates collecting and monitoring metrics, logs, and traces from common data sources. Data collection and monitoring in one place ensure full-stack, end-to-end observability of the entire infrastructure.

Splunk Observability is different from the Splunk Core that powers Splunk Cloud / Enterprise and is the traditional log management solution from Splunk. Nobl9 also integrates to that through a different set of APIs.

Splunk Observability parameters and supported features in Nobl9
General support:
Release channel: Agent connection method: Stable, Beta. Direct connection method: Alpha
Connection method: Agent, Direct
Replay and SLI Analyzer: Not supported
Event logs: Not supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 5 min
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9splunk_observability
Query delay environment variable: N9SPLUNK_OBSERVABILITY_QUERYDELAY
Timestamp cache persistence: 0.65.0
Additional notes:
Available on demand
Maximum query delay for Splunk Observability is 15 minutes

On-demand feature

The Splunk Observability integration with Nobl9 is available on demand. Fill in the request form to access it.

Authentication​

The Nobl9 integration with Splunk Observability requires authentication using an access token and Splunk Observability realm.

Access token:

You can create an access token in your Splunk Observability profile. Provide your SPLUNK_OBSERVABILITY_ACCESS_TOKEN as an environment variable when deploying your Splunk Observability agent.

Access Token details
  • Access tokens are valid for 30 days.
  • Org tokens are valid for 5 years and can be used to generate session tokens.

Realm:

Your Splunk Observability realm identifies your region. You can find your realm URL in the Endpoints section of your Splunk Observability profile.
For more information on realms, visit Realms in Endpoints.

Adding Splunk Observability as a data source​

You can add the Splunk Observability data source using the direct or agent connection methods.

Direct connection method​

Direct connection to Splunk Observability requires users to enter their credentials which Nobl9 stores safely.

Nobl9 Web​

To set up this type of connection:

  1. Navigate to Integrations > Sources.
  2. Click .
  3. Click the required Source button.
  4. Choose Direct.
Release channel

Nobl9 integration with Splunk Observability is available in the Alpha release channel only.

  1. Enter your organization's Realm to connect your data source.
    Refer to the Authentication section above for more details.

  2. Enter the Access Token environment variable for authentication with the organization API Access Token.
    Refer to the Authentication section above for more details.

  1. Select a Project.
    Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
  2. Enter a Display Name.
    You can enter a user-friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
    • The default value in Splunk Observability integration for Query delay is 5 minutes.
    Changing the query delay
    Changing the query delay can affect your SLI data.
    Learn more about query delay and its impact.
  6. Click Add Data Source.

sloctl​

  1. Create a YAML definition to set up a direct connection with Splunk Observability. For this, refer to the following example:
YAML definition for the direct connection method
apiVersion: n9/v1alpha
kind: Direct
metadata:
  name: splunk-observability
  displayName: Splunk Observability Direct
  project: default
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Splunk Observability Direct
  releaseChannel: alpha
  splunkObservability:
    realm: us1
    accessToken: "[secret]"
  queryDelay:
    value: 6
    unit: Minute
FieldTypeDescription
queryDelay.unit
mandatory
enum

Specifies the unit for the query delay. Possible values: Second, Minute.
• Check query delay documentation for default unit of query delay for each source.

queryDelay.value
mandatory
numeric

Specifies the value for the query delay.
• Must be a number less than 1440 minutes (24 hours).
• Check query delay documentation for default unit of query delay for each source.

releaseChannel
mandatory
enum

Specifies the release channel. Accepted value: alpha.

Source-specific fields
splunkObservability.realm
mandatory
string

See realms in endpoints for more details.

splunkObservability.accessToken
mandatory
string

Environment variable used for authentication with the organization API Access Token. See authentication section above for more details.

  1. Apply your YAML definition using the sloctl apply command.

Agent connection method​

Nobl9 Web​

Follow the instructions below to configure your Splunk Observability agent.

  1. Navigate to Integrations > Sources.
  2. Click .
  3. Click the required Source button.
  4. Choose Agent.
  1. Select one of the following Release Channels:
    • The stable channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of a beta release. Use it to avoid crashes and other limitations.
    • The beta channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.

  2. Enter your organization's Realm to connect your data source.

  1. Select a Project.
    Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
  2. Enter a Display Name.
    You can enter a user-friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
    • The default value in Splunk Observability integration for Query delay is 5 minutes.
    Changing the query delay
    Changing the query delay can affect your SLI data.
    Learn more about query delay and its impact.
  6. Click Add Data Source.
  7. Deploy your agent in a Kubernetes cluster or Docker container.

sloctl​

  1. Create a YAML definition to set up an agent connection with Splunk Observability. For this, refer to the following example:
YAML definition for the agent connection method
apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: splunk-observability
  displayName: Splunk Observability Agent
  project: default
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Splunk Observability Agent
  releaseChannel: stable
  splunkObservability:
    realm: us1
  queryDelay:
    value: 6
    unit: Minute
FieldTypeDescription
queryDelay.unit
mandatory
enum

Specifies the unit for the query delay. Possible values: Second, Minute.
• Check query delay documentation for default unit of query delay for each source.

queryDelay.value
mandatory
numeric

Specifies the value for the query delay.
• Must be a number less than 1440 minutes (24 hours).
• Check query delay documentation for default unit of query delay for each source.

releaseChannel
mandatory
enum

Specifies the release channel. Accepted values: stable, beta.

Source-specific fields
splunkObservability.realm
mandatory
string

See realms in endpoints for more details.

  1. Apply your YAML definition using the sloctl apply command.
  2. Deploy your agent in a Kubernetes cluster or Docker container.

Log sampling for the Splunk Observability agent​

The Splunk Observability agent features a logging mechanism to handle burstable log loads. It applies only to redundant points dropping information. Other logs are logged normally.

You can decide whether you want to use log sampling or not by setting SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG environment variable. It's a JSON object with the following fields:

Here's an example of Kubernetes deployment YAML with activated burst logs:

# DISCLAIMER: This Deployment description is containing only the necessary fields for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description and the client_id as well as the client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
  name: nobl9-agent-nobl9-dev-dwq-ble
  namespace: default
type: Opaque
stringData:
  splunk_observability_access_token: "<SPLUNK_OBSERVABILITY_ACCESS_TOKEN>"
  client_id: "<CLIENT_ID>"
  client_secret: "<CLIENT_SECRET>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-nobl9-dev-splunkobs-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: "splunkobs"
      nobl9-agent-project: "deployment"
      nobl9-agent-organization: "nobl9-dev"
  template:
    metadata:
      labels:
        nobl9-agent-name: "splunkobs"
        nobl9-agent-project: "deployment"
        nobl9-agent-organization: "nobl9-dev"
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:0.91.0
          resources:
            requests:
              memory: "350Mi"
              cpu: "0.1"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: nobl9-agent-nobl9-dev-splunkobs-deployment
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: nobl9-agent-nobl9-dev-dwq-ble
            - name: SPLUNK_OBSERVABILITY_ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  key: splunk_observability_access_token
                  name: nobl9-agent-nobl9-dev-dwq-ble
            - name: N9_METRICS_PORT
              value: "9090"
            - name: SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG
              value: '{ \"burst\": 3, \"period\": 120, \"enabled\": true}'

Here's an alternative way to activate burst logs via a YAML config:

            - name: SPLUNK_OBSERVABILITY_DATA_POINT_LOG_SAMPLING_CONFIG
              value: '{"enabled":true}'

The above YAMLs default .enabled to false so that agents by default don't use it.

If only the .enabled variable is set to true, it defaults .burst to 1, and .period to 900, which is an equivalent to log 1 message each 15 minutes per organization.

Here's an example of configuration that allows to log 3 messages per 120 seconds per organization:

"{ \"burst\": 3, \"period\": 120, \"enabled\": true}"
For a more in-depth look, consult additional resources:
Create SLOs with Splunk ObservabilityCreating SLOsAgent metricsAgentCreating agents via TerraformTerraformSplunk Observability cloud documentationExternal docsCreate an Access TokenExternal docsRealms in endpointsExternal docsOrg token limitsExternal docsSystem limits for Splunk infrastructure monitoringExternal docs
On this page