Skip to main content

Splunk

Reading time: 0 minute(s) (0 words)

Splunk provides software for searching, monitoring, and analyzing machine-generated data via a Web-style interface. Splunk-Nobl9 integration allows users to enter their metrics using the Splunk Processing Language (SPL).

Splunk parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent, Direct
Replay and SLI Analyzer: Historical data limit 30 days
Event logs: Supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 5 min
Jitter: 20 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9splunk
Query delay environment variable: SPLUNK_QUERY_DELAY
Replay and SLI Analyzer: 0.82.2
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0
Custom HTTP headers: 0.90.0 / 0.89.1-beta
Additional notes:
Self-signed Splunk Enterprise certificates are not supported. Nobl9 requires successful certificate validation for any Splunk Enterprise instance using TLS, and self-signed certificates cannot pass this validation

Creating SLOs with Splunk​

Nobl9 Web​

  1. Navigate to Service Level Objectives.
  2. Click .
  3. Select a Service.
    It will be the location for your SLO in Nobl9.
  4. Select your Splunk data source.
  5. Modify Period for Historical Data Retrieval, if necessary.
    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on Splunk.
    • A longer period can extend the data loading time for your SLO.
    • Must be a positive whole number up to the maximum period value you've set when adding the Splunk data source.
    • For your Splunk data source, you can extend this period up to 30 days.

    Non-editable Replay period
    Non-editable Replay period indicates that the maximum period for historical data retrieval set for your Splunk data source is set to zero.
    Adjust the data source settings to create the SLO with Replay.
  6. Metric refers to the way you calculate and interpret data from your data source.
    • Threshold metric is defined by a single numerical value (the threshold) that separates satisfactory performance from unsatisfactory performance. It's represented by a single time series evaluated against the threshold.
    • Ratio metric expresses the performance as a fraction or proportion, typically by dividing the number of successful events by the total number of potential events (successes + failures). For your Splunk SLO, you can configure a two-query or single-query ratio to compare good and total events.
      Select the Data count method for your ratio metric.

      SLI values for good and total
      When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
      • Must be positive.
      • While we recommend using integers, fractions are also acceptable.
        • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
      • Shouldn't be larger than 1e+20.

  1. Define the Query using the Splunk Search Processing Language. The query must meet the following requirements:

    • Every query must contain the n9 fields. Splunk must return their values in the dataset:
    The required n9 values per metric type
    ThresholdTwo-query ratioSingle-query ratioDescription
    n9timen9timen9timeA Unix timestamp
    n9valuen9valuenot requiredFloat number
    not requirednot requiredn9goodFloat number
    not requirednot requiredn9totalFloat number

    • The query can retrieve data from either Events or Metrics Splunk indexes.

    • Nobl9 verifies whether the query contains the required n9 fields and the index= field with an index value.

    • Dataset time ranges are segmented into 15-second chunks. Dataset aggregation depends on the metric type.

      Dataset aggregation
      Metric typeAggregation method
      ThresholdAverage
      Ratio incrementalMaximum value
      Ratio non-incrementalSum of values
    Click to open query samples
    Querying for Events
    search index=_events sourcetype=syslog status<400
    | bucket _time span=1m
    | stats count as n9value by _time
    | rename _time as n9time
    | fields n9time n9value
    Querying for Metrics
    | mstats avg("my.metric") as n9value WHERE index=_metrics span=15s
    | rename _time as n9time
    | fields n9time n9value
    Ratio single query, Metrics index
    | mstats avg("spl.intr.resource_usage.IOWait.data.avg_cpu_pct") as n9good WHERE index="_metrics" span=15s
    | join type=left _time [
    | mstats avg("spl.intr.resource_usage.IOWait.data.max_cpus_pct") as n9total WHERE index="_metrics" span=15s]
    | rename _time as n9time
    | fields n9time n9good n9total
  1. Define the Time window for your SLO:
  2. Configure the Error budget calculation method and Objectives:
    • Occurrences method counts good attempts against the count of total attempts.
    • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
    • You can define up to 12 objectives for an SLO.

      • Similar threshold values for objectives
      To use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
      For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one.
  3. Add the Display name, Name, and other settings for your SLO:
    • Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
      Use only lowercase letters, numbers, and dashes.
    • Select No data anomaly alert to receive notifications when your SLO stops reporting data for a specified period:
      • Choose up to five supported Alert methods.
      • Specify the delay period before Nobl9 sends an alert about the missing data.
        From 5 minutes to 31 days. Default: 15 minutes
    • Add alert policies, labels, and links, if required.
      Limits per SLO: 20 alert policies or links, 30 labels.
  4. Click CREATE SLO.

    5. SLO configuration use case
    Check the SLO configuration use case for a real-life SLO example.

YAML​

Sample Splunk threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Splunk SLO
  indicator:
    metricSource:
      name: splunk
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          splunk:
            query: >-
              index=* source=udp:5072 sourcetype=syslog status<400 | bucket
              _time span=1m | stats avg(response_time) as n9value by _time |
              rename _time as n9time | fields n9time n9value
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: '2022-12-01 00:00:00'
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default
      alertAfter: 1h
Click to open field reference
FieldTypeDescription
apiVersion
mandatory
stringAPI version. Use n9/v1alpha
kind
mandatory
stringThe resource type. Use SLO
Metadata
metadata.name
mandatory
stringName identifier for the SLO. Use only lowercase alphanumeric characters
metadata.displayNamestringUser-friendly SLO name
metadata.project
mandatory
stringThe name identifier of the project where you need to host your SLO
metadata.labelsobject (map: string[])Grouping labels for filtering or viewing
metadata.annotationsobject (map: string)Flat string annotations
Spec
spec.descriptionstringSLO description
spec.indicator.metricSource.name
mandatory
stringData source name
spec.indicator.metricSource.project
mandatory
stringProject containing the data source
spec.indicator.metricSource.kind
mandatory
stringData source connection method. Can be Agent or Direct
spec.budgetingMethod
mandatory
enumError budget calculation method. Can be Occurrences or Time slices
spec.objectives
mandatory
arrayYour SLO objective definition, up to 12 objectives per SLO.
spec.objectives[].displayNamestringUser-friendly objective name
spec.objectives[].value
mandatory
numberData point values that is considered "good" (e.g., 200.0).
In SLOs with two or more objectives, keep each objective's value unique.
In ratio (count) metrics, value is retained for legacy purposes.
spec.objectives[].name
mandatory
stringName identifier for this objective
spec.objectives[].op
mandatory
string (enum)Operator for objective. One of:
lte (less than or equal to)
lt (less than)
gte (greater than or equal to)
gt (greater than)
spec.objectives[].target
mandatory
floatThe percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is 0.95, the good performance is expected to be observed in at least 95% of the time window)
spec.objectives[].rawMetric/.countMetric
mandatory
objectThe metric type indicator. Set:
rawMetric for a threshold metric
countMetric for a ratio metric.
A ratio metric requires the additional fields:
countMetric.incremental (boolean) the data count method
countMetric.good/.bad and countMetric.total a numerator and denominator queries
spec.objectives[].countMetric.incremental
mandatory
booleanThe data count method for a ratio (countMetric) metric type
spec.objectives[].primarybooleanThe indicator of a primary SLO objective
spec.service
mandatory
stringThe name identifier of a service to host this SLO. The service must exist in the project specified in metadata.project
spec.timeWindows
mandatory
arrayDefines SLO time window for error budget calculation. Set:
isRolling: true for the rolling time window type
isRolling: false for the calendar-aligned type
spec.timeWindows.unit
mandatory
integerThe time window units. One of:
Day | Hour | Minute for the rolling time window
Year | Quarter | Month | Week | Day for the calendar-aligned time window
spec.timeWindows.count
mandatory
integerThe number of units in a time window
spec.timeWindows.startTimestringMandatory for calendar-aligned time windows. Date and time in the format YYYY-MM-DDTHH:mm:ss
spec.timeWindows.timeZonestringMandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
spec.timeWindows.isRolling
mandatory
boolean
true for the rolling time window type
false for the calendar-aligned type
spec.alertPoliciesarrayThe name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
spec.attachmentsarrayLinks to any additional attributes of this SLO
spec.anomalyConfigobjectSettings for a manual no data anomaly detection rule
spec.noData.alertMethodsarrayList of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the name and project fields
spec.noData.alertAfterstringWaiting time before sending a no-data notification. Must be 5m to 31d.
Default: 15m
Source-specific fields
splunk.query
mandatory
stringYour Splunk query

Query tips, requirements, and samples​

  • For the best experience, keep query run time short enough to return results within 1–2 minutes. For example, when you query data for the last 15 minutes, the response should return within less than one minute.
    You can check query duration using Splunk's Search Job Inspector or learn quick tips for query optimization.

  • Agent version requirements for single-query ratio metrics

    NameStableBeta
    Single-query ratio metric0.80.00.80.0-beta
    Replay and SLI Analyzer0.82.20.82.0-beta

  • Every query must contain n9 fields, and Splunk must return their values in the dataset. The n9 fields are as follows:

    The required n9 values per metric type
    ThresholdTwo-query ratioSingle-query ratioDescription
    n9timen9timen9timeA Unix timestamp
    n9valuen9valuenot requiredFloat number
    not requirednot requiredn9goodFloat number
    not requirednot requiredn9totalFloat number

    Use Splunk field extractions to return values with the exact names. The n9time is the actual time, and the n9value, n9good, and n9total are metric values.

    Typically, you rename _time to n9time and the field containing metric values (for example, response_time)—to the n9value, n9good, or n9total.

    For example,

    index=myserver-events source=udp:5072 sourcetype=syslog response_time>0
    | rename _time as n9time, response_time as n9value
    | fields n9time n9value
    Query frequency

    The Splunk query is by default executed once per minute, returning the values found in the fields n9time and n9value. Ensure your hardware can support the query frequency or consider increasing the query interval.

  • The query can retrieve data from either Events or Metrics Splunk indexes.

    • Nobl9 validates the queries to contain the respective n9 field along with index= with the value.

    • Dataset time ranges are segmented into 15-second chunks and aggregated. The aggregation is as follows:

      Dataset aggregation
      Metric typeAggregation
      ThresholdAverage
      Ratio incrementalMaximum value
      Ratio non-incrementalSum of values

  • The index attribute ("index=index_name") lets avoiding long-running queries.

    • The query can retrieve data from both the Events and Metrics indexes.
    • To retrieve Metrics data, use the | mstats command.
    • To retrieve data from the Events and Metrics indexes, enter the SPL query and select a proper index: index=_metrics or index=_events, where _metrics is the name of the metrics index, and _events is the name of the events index.

Querying Splunk server​

The Nobl9 agent leverages Splunk Enterprise API parameters. It pulls data at a per-minute interval from the Splunk server.

API rate limits for the Nobl9 agent​

Splunk Enterprise API rate limits are configured by its administrators. Rate limits must be high enough to accommodate searches from the Nobl9 agent. The Nobl9 agent makes one query per minute per unique query.

Read more in Maximum and actual search concurrency calculations | Splunk community.

Concurrent searches

For the best results, the number of concurrent searches must be about the same as the number of SLIs you have for this data source.

Number of events returned from Splunk queries​

Supported search SPL command searches within indexed events. The total number of events can be large, and a query without specific conditions, such as search sourcetype=*, returns all indexed events. A large number of data points sent to Nobl9 could disrupt the system’s performance. Therefore, there is a hard limit of 4 events per minute.

File-based queries and Splunk disk quota​

If you’re using file-based queries (the inputlookup function) instead of index-based queries, your query might not work as expected. Due to the difference in jitter configuration between Splunk and Nobl9, you might need to increase your Splunk disk quota for the inputlookup function to work properly.

To determine the appropriate disk quota size for your Splunk account, we recommend the following steps:

  1. Go to the Splunk UI and navigate to Activity > Jobs.
  2. Filter the logs by the user you currently use in Nobl9 App.
  3. Execute requests at 29-second intervals to gather all logs from the corresponding cycle.
  4. Sum the sizes of all requests from the list to determine the minimum disk quota. It is important to add a buffer to this number for safety.
  5. Once you have created more SLOs, adjust the disk quota accordingly.

We suggest increasing the quota to 2GB to resolve the issue. However, it’s important to note that the final disk quota size will depend on the data being queried.

Known limitations​

Query limitations:

Check out these related guides and references:
Add Splunk as a data sourceAdding data sourcesSplunk Search LanguageSplunk documentationTime Range ModifiersSplunk documentationMaximum and actual search concurrency calculationSplunk documentationCreating SLOs via TerraformTerraform
On this page