Skip to main content

Splunk

Splunk provides software for searching, monitoring, and analyzing machine-generated data via a Web-style interface. Splunk integration with Nobl9 allows users to enter their metrics using the Splunk Processing Language (SPL).

Scope of Support​

Nobl9 does not support a self-signed Splunk Enterprise. Nobl9 Agent requires that if Splunk Enterprise is configured to use TLS then it must successfully pass certificate validation which self-signed certificates do not.

Authentication​

Splunk configuration for the Agent only accepts a single parameter: url. The url has to point to the base API URL of the Splunk Search app. It will usually have a form of {SPLUNK_BASE_URL}:{PORT_NUMBER}/services where:

  • SPLUNK_BASE_URL - for Splunk Enterprise, the base URL is configured during the deployment of Splunk software.

  • PORT_NUMBER - Assuming the API is using the default port is 8089. It is recommended that you contact your Splunk Admin to get your API Token and to verify the correct URL to connect.

When deploying the Agent for Splunk, you can use one of the following authentication methods:

  • SAML: provide SPLUNK_APP_TOKEN as environment variables for authentication with the Splunk Search App REST API.
  • basic: to use basic type, the Agent requires SPLUNK_USER and SPLUNK_PASSWORD passed as an environment variables during agent startup.

For more details, refer to the How to Obtain Value for SPLUNK_APP_TOKEN | Splunk Documentation.

tip

Alternatively, you can pass the token using a local config file with key application_token under the n9splunk section. You can give your username and password the same way, and the keys are app_user and app_password.

Adding Splunk as a Data Source in the UI​

To add Splunk as a data source in Nobl9 using the Agent or Direct connection method, follow these steps:

  1. Navigate to Integrations > Sources.
  2. Click the button.
  3. Click the relevant Source icon.
  4. Choose a relevant connection method (Agent or Direct), then configure the source as described below.

Splunk Direct​

Direct Configuration in the UI​

Direct configuration for Splunk requires users to enter their credentials, which Nobl9 stores safely. Follow these steps to set up a direct configuration:

caution

To access your Splunk Cloud Platform deployment using the Splunk REST API, you must submit a case requesting access using the Splunk Support Portal Splunk Support opens port 8089 for REST access. You can specify a range of IP addresses to control who can access the REST API.

You will need to allow the following IPs:

18.159.114.21 18.158.132.186 3.64.154.26

  1. Enter an API Endpoint URL to connect to your data source (mandatory).
    Refer to the Authentication section above for more details.

  2. Enter the Access Token generated from your Splunk instance (mandatory).
    Refer to the Authentication section above for more details.

  1. Select a Project.
    Specifying a Project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank then object is assigned to project default.
  2. Enter a Display Name.
    You can enter a friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters and dashes (for example, my-project-name). This field is populated automatically when you enter a display name, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Enter a Maximum Period for Historical Data Retrieval.
    • This value defines how far back in the past your data will be retrieved.
    • The value for the Maximum period of data retrieval depends on the Source. Check the Replay documentation for details.
    • Entering a more extended Period might slow down the loading time when creating an SLO.
      • The value must be a positive integer.
  6. Enter a Default Period for Historical Data Retrieval.
    • It is the Period that will be used by the SLO connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. If you set it to >0, you will create an SLO with Replay.
  7. Click the Add Data Source button.

Splunk Agent​

Agent Configuration in the UI​

Follow the instructions below to configure your Splunk Agent. Refer to the section above for the description of the fields.

  1. Add the URL to connect to your data source.
    Example URL: https://splunk.example.com:8089/services. It is recommended that your contact your Splunk Admin to get your API Token and to verify the correct URL to connect.

  2. Enter a Project (mandatory).

  3. Enter a Display name (optional).

  4. Enter a Name (mandatory).

  5. Enter a Description (optional).

  6. In the Advanced settings, you can:

    1. Enter a Maximum Period for Historical Data Retrieval.

      • This value defines how far back in the past your data will be retrieved.
      • The maximum Period of retrieving the data for Splunk cannot exceed 30 days.
      • Entering a more extended Period might slow down the loading time when creating an SLO.
      • The value must be a positive integer.

    2. Enter a Default Period for Historical Data Retrieval.

      • It is the Period that will be used by the SLO connected to this data source.
      • The value must be a positive integer.
  7. Click the Add Data Source button.

Agent Using CLI - YAML​

The YAML for setting up an Agent connection to Splunk looks like this:

apiVersion: n9/v1alpha
kind: Agent
metadata:
  name: splunk
  project: splunk
spec:
  sourceOf:
    - Metrics
    - Services
  splunk:
    url: https://splunk.example.com
  historicalDataRetrieval:
      maxDuration:
        value: 30 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day
      defaultDuration: # value must be less than or equal to value of maxDuration
        value: 0 # integer greater than or equal to 0
        unit: Day # accepted values: Minute, Hour, Day

Important notes:

warning

You can deploy only one Agent in one YAML file by using the sloctl apply command.

Deploying Splunk Agent​

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the Agent. Both of these are available in the web UI, under the Agent Configuration section. Be sure to swap in your credentials (e.g., replace <SPLUNK_APP_TOKEN> with your organization key).

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the Agent. It will look something like this:

# DISCLAIMER: This deployment description contains only the fields necessary for the purpose of this demo.
# It is not a ready-to-apply k8s deployment description, and the client_id and client_secret are only exemplary values.

apiVersion: v1
kind: Secret
metadata:
  name: nobl9-agent-nobl9-dev-default-splunkagent
  namespace: default
type: Opaque
stringData:
  splunk_app_token: "<SPLUNK_APP_TOKEN>"
  client_id: "unique_client_id"
  client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-nobl9-dev-default-splunkagent
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: "splunkagent"
      nobl9-agent-project: "default"
      nobl9-agent-organization: "nobl9-dev"
  template:
    metadata:
      labels:
        nobl9-agent-name: "splunkagent"
        nobl9-agent-project: "default"
        nobl9-agent-organization: "nobl9-dev"
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:latest
          resources:
            requests:
              memory: "350Mi"
              cpu: "0.1"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: nobl9-agent-nobl9-dev-default-splunkagent
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: nobl9-agent-nobl9-dev-default-splunkagent
            - name: SPLUNK_APP_TOKEN
              valueFrom:
                secretKeyRef:
                  key: splunk_app_token
                  name: nobl9-agent-nobl9-dev-default-splunkagent
            - name: SPLUNK_USER
              valueFrom:
                secretKeyRef:
                  key: splunk_user
                  name: nobl9-agent-nobl9-dev-default-splunkagent
            - name: SPLUNK_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: splunk_password
                  name: nobl9-agent-nobl9-dev-default-splunkagent
            # The N9_METRICS_PORT is a variable specifying the port to which the /metrics and /health endpoints are exposed.
            # The 9090 is the default value and can be changed.
            # If you don’t want the metrics to be exposed, comment out or delete the N9_METRICS_PORT variable.
            - name: N9_METRICS_PORT
              value: "9090"

Creating SLOs with Splunk​

Creating SLOs in the UI​

  1. Navigate to Service Level Objectives.

  2. Click the button.

  3. In step 1 of the SLO wizard, select the Service the SLO will be associated with.

  4. In step 2, select Splunk as the Data Source for your SLO, then specify the Metric. You can choose either a Threshold Metric, where a single time series is evaluated against a threshold or a Ratio Metric, which allows you to enter two time series to compare (for example, a count of good requests and total requests).

    note

    For the Ratio Metric, you can choose the Data Count Method:

    • For the Non-incremental method, we expect it to be the components of the sum.
    • For the Incremental method, we expect the value of a metric to be the current sum of some numerator.

    For more information, refer to the SLO Calculations Guide.

  5. Enter a Query, Good Query, or Total Query for the metric you selected:

    • Every query must return n9time and n9value fields.

      • The n9time field must be a Unix timestamp, and n9value field must be a float value.

    • Nobl9 validates the query provided by the user against 3 rules:

      • The query contains index= with a value.

      • The query contains the n9time value.

      • The query contains an n9value value.

    • Every time range of the dataset is segmented into 15-second chunks and aggregated. The aggregation is as follows:

      • Raw metric: calculates the average.

      • Count metric incremental: takes the max value.

      • Count metric non-incremental: the sum of values.

  6. In step 3, define a Time Window for the SLO.

  7. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  8. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  9. When you’re done, click Create SLO.

SLOs using Splunk - YAML samples​

Here’s an example of Splunk using a rawMetric (Threshold metric):

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: splunk-raw-rolling
    project: splunk
  spec:
    service: splunk-service
    indicator:
      metricSource:
        name: splunk
    timeWindows:
      - unit: Day
        count: 7
        isRolling: true
    budgetingMethod: Occurrences
    objectives:
      - displayName: Good
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 0.25
        target: 0.50
      - displayName: Moderate
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 0.5
        target: 0.75
      - displayName: Annoying
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 1.0
        target: 0.95
  ---
  - apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: splunk-raw-calendar
    project: splunk
  spec:
    service: splunk-service
    indicator:
      metricSource:
        name: splunk
    timeWindows:
      - unit: Day
        count: 7
        calendar:
          startTime: 2020-03-09 00:00:00
          timeZone: Europe/Warsaw
    budgetingMethod: Occurrences
    objectives:
      - displayName: Good
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 0.25
        target: 0.50
      - displayName: Moderate
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 0.5
        target: 0.75
      - displayName: Annoying
        op: lte
        rawMetric:
          query:
            splunk:
              query: index=* source=udp:5072 sourcetype=syslog status<400 | bucket _time span=1m | stats avg(response_time) as n9value by _time | rename _time as n9time | fields n9time n9value
        value: 1.0
        target: 0.95

Splunk queries require:

  • Defining an index attribute ("index=index_name") to avoid long-running queries.

    • The query can retrieve data from both, the Events and Metrics indexes.

    • You can retrieve Metrics data by using the | mstats command.

    • To retrieve data from the Events and Metrics indexes, you must enter SPL query and select a proper index: index=_metrics or index=_events, where _metrics is the name of the metrics index, and _events is the name of the events index. For more information on the SPL query, refer to the About the Search Language | Splunk Documentation.

    • Query example for Events index:

      search index=_events sourcetype=syslog status<400
      | bucket _time span=1m
      | stats count as n9value by _time
      | rename _time as n9time
      | fields n9time n9value

    • Query example for Metrics index:

      | mstats avg("my.metric") as n9value WHERE index=_metrics span=15s
      | rename _time as n9time
      | fields n9time n9value

  • A return value for n9time and n9value.
    Use Splunk field extractions to return values using those exact names. The n9time is the actual time, and the n9value is the metric value. The n9time must be a Unix timestamp and the n9value must be a float value.

    • Example:

      index=myserver-events source=udp:5072 sourcetype=syslog response_time>0
      | rename _time as n9time, response_time as n9value
      | fields n9time n9value

      Typically, you will rename _time to n9time and then rename the field containing the metric value (response_time in the previous example) to the n9value. The following is the appendage to your normal query that handles this.

      | rename _time as n9time, response_time as n9value
      | fields n9time n9value
      • The Splunk query will be executed once every minute, returning the values found in the fields n9time and n9value. Ensure your hardware can support the query frequency.

Querying Splunk Server​

The Nobl9 agent leverages Splunk Enterprise API parameters. It pulls data at a per-minute interval from the Splunk server.

API Rate Limits for the Nobl9 Agent​

Splunk Enterprise API rate limits are configured by its administrators. Rate limits have to be high enough to accommodate searches from Nobl9 Agent. The Nobl9 agent makes one query per minute per unique query.

Number of events returned from Splunk queries​

Supported search SPL command searches within indexed events. The total number of events can be large, and a query without specific conditions, such as search sourcetype=*, returns all indexed events. A large number of data points sent to Nobl9 could disrupt the system’s performance. Therefore, there is a hard limit of 4 events per minute.

Known Limitations​

Queries limitations:

REST Access | Splunk Documentation

Create Auth Tokens | Splunk Documentation

Agent Metrics | Nobl9 Documentation

Creating SLOs via Terraform | Nobl9 Terraform Documentation

Creating Agents via Terraform | Nobl9 Terraform Documentation

Last updated on
Submit Feedback