Skip to main content

Elasticsearch

Reading time: 0 minute(s) (0 words)

Elasticsearch is a distributed search and storage solution used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases. This integration supports histogram aggregate queries that return either a single value or a single pair stored in n9-val field, any filtering or matches can be applied as long as the output follows the mentioned format.

Elasticsearch parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent
Replay and SLI Analyzer: Historical data limit 30 days. Replay only
Event logs: Not supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 1 min
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9elasticsearch
Query delay environment variable: ES_QUERY_DELAY, ELASTICSEARCH_CUSTOM_AUTHORIZATION_HEADER, N9_ELASTIC_SEARCH_MAX_BUCKETS
Replay and SLI Analyzer: 0.85.0-beta
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0
Additional notes:
Support for Elasticsearch v7.9.1
Support for AWS OpenSearch with agent v0.89.0-beta

Creating SLOs with Elasticsearch​

Nobl9 Web​

Follow the instructions below to create your SLOs with Elasticsearch in the UI:

  1. Navigate to Service Level Objectives.
  2. Click .
  3. Select a Service.
    It will be the location for your SLO in Nobl9.
  4. Select your Elasticsearch data source.
  5. Modify Period for Historical Data Retrieval, when necessary.
    • This value defines how far back in the past your data will be retrieved when replaying your SLO based on Elasticsearch.
    • A longer period can extend the data loading time for your SLO.
    • Must be a positive whole number up to the maximum period value you've set when adding the Elasticsearch data source.
  6. Select the Metric type:
    • Threshold metric: a single time series is evaluated against a threshold.
    • Ratio metric: two-time series for comparison for good events and total events.
      For ratio metrics, select the Data count method: incremental or non-incremental.

  1. Enter a Query or Query for good counter and Query for total counter for the metric you selected.
    For examples of queries, refer to the section below.

    For details on Elasticsearch queries, refer to the Scope of support for Elasticsearch Queries section.

    SLI values for good and total
    When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:
    • Must be positive.
    • While we recommend using integers, fractions are also acceptable.
      • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
    • Shouldn't be larger than 1e+20.
  1. Define the Time window for your SLO:
  2. Configure the Error budget calculation method and Objectives:
    • Occurrences method counts good attempts against the count of total attempts.
    • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
    • You can define up to 12 objectives for an SLO.

      • Similar threshold values for objectives
      To use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
      For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one.
      Learn more about threshold value uniqueness.
  3. Add the Display name, Name, and other settings for your SLO:
    • Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
      Use only lowercase letters, numbers, and dashes.
    • Create composite SLO: with this option selected, you create a composite SLO 1.0. Composite SLOs 1.0 are deprecated. They're fully operable; however, we encourage you to create new composite SLOs 2.0.
      You can create composite SLOs 2.0 with sloctl using the provided template. Alternatively, you can create a composite SLO 2.0 with Nobl9 Terraform provider.
    • Set Notifications on data. With it, Nobl9 will notify you in the cases when SLO won't be reporting data for more than 15 minutes.
    • Add alert policies, labels, and links, if required.
      Up to 20 items of each type per SLO is allowed.
  4. Click CREATE SLO.

    5. SLO configuration use case
    Check the SLO configuration use case for a real-life SLO example.

sloctl​

Sample Elasticsearch threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Elasticsearch SLO
  indicator:
    metricSource:
      name: elasticsearch
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          elasticsearch:
            index: apm-7.13.3-transaction
            query: |-
              {
                "query": {
                  "bool": {
                    "must": [
                      {
                        "match": {
                          "service.name": "api-server"
                        }
                      },
                      {
                        "match": {
                          "transaction.result": "HTTP 2xx"
                        }
                      }
                    ],
                    "filter": [
                      {
                        "range": {
                          "@timestamp": {
                            "gte": "{{.BeginTime}}",
                            "lte": "{{.EndTime}}"
                          }
                        }
                      }
                    ]
                  }
                },
                "size": 0,
                "aggs": {
                  "resolution": {
                    "date_histogram": {
                      "field": "@timestamp",
                      "fixed_interval": "{{.Resolution}}",
                      "min_doc_count": 0,
                      "extended_bounds": {
                        "min": "{{.BeginTime}}",
                        "max": "{{.EndTime}}"
                      }
                    },
                    "aggs": {
                      "n9-val": {
                        "avg": {
                          "field": "transaction.duration.us"
                        }
                      }
                    }
                  }
                }
              }
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01T00:00:00.000Z
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default

Elasticsearch queries scope of support​

When you use data from Elastic APM, the document timestamp can be provided either with "field": "@timestamp" or other, depending on the schema used.

Learn more about query requirements and options in Elasticsearch:

For the Nobl9 agent, queries must meet the following requirements:

  • The search results must be a time series
  • The aggs.resolution object must hold the following:
    • date_histogram to provide the timestamps
    • aggs.n9-val to provide the value(s)
Sample Elasticsearch query with the resolution, date_histogram, and n9-val objects
{
  "aggs": {
    "resolution": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "{{.Resolution}}",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "{{.BeginTime}}",
          "max": "{{.EndTime}}"
        }
      },
      "aggs": {
        "n9-val": {
          "avg": {
            "field": "transaction.duration.us"
          }
        }
      }
    }
  }
}

  1. Date Histogram Aggregation

    • We recommend using date_histogram with fixed_interval and passing the {{.Resolution}} placeholder as the interval value. With it, the Nobl9 agent can control data resolution.
    • fixed_interval in the query must be no longer than one minute: Nobl9 agent queries your data source every one minute for a 1-minute time range.

  2. Date Histogram Aggregation Fixed Intervals

    • "field": "@timestamp" must match the field used in the filter query.
    • Use extended_bounds with the "{{.BeginTime}}", "{{.EndTime}}" placeholders as a filter query.
    Placeholder usage

    {{.BeginTime}} and {{.EndTime}} are mandatory placeholders and must be included in the query. If you use filter and aggregations parameters in your query, {{.BeginTime}} and {{.EndTime}} are required for both parameters.

    The Nobl9 agent replaces these placeholders with the correct time range values.

  3. Metrics Aggregations

    • n9-val must be a metric aggregation.

    • single value metric aggregation is used as the value of the time series.

    • multi-value metric aggregation first returns a non-null value and is used as the value of the time series. In the following example, the null values are skipped.

      "aggs": {
          "n9-val": {
              ...
          }
      }

  4. elasticsearch.index is the index name when the query completes.

Querying the Elasticsearch server​

Nobl9 calls Elasticsearch Get API every minute and retrieves data points from the previous minute to the present time point. The number of data points is dependent on how much data the customer has stored.

Elasticsearch API rate limits​

For aggregation and analysis, Elasticsearch uses buckets. Each bucket collects a set of documents that match a given criterion.

The default API rate limit in Elasticsearch is 65,536 buckets for aggregate queries; however, the actual number of buckets depends on the target cluster configuration.

To configure the API rate limits within the Nobl9 agent, provide the N9_ELASTIC_SEARCH_MAX_BUCKETS environment variable. The Nobl9 agent uses its value to determine the resolution duration when registering data while replaying SLOs. The higher the value, the more data Replay will request.

To configure rate limits at the level of your Elasticsearch cluster, set the required value for the search.max_buckets parameter.

Learn more about search settings in Elasticsearch.

For a more in-depth look, consult additional resources:
Sending more data points than expectedTroubleshootingElasticsearch authenticationElasticsearch docsElasticsearch get APIElasticsearch docsElasticsearch APMElasticsearch docsBoolean queryElasticsearch docsQuery and filter contextElasticsearch docsFilter aggregationElasticsearch docsRange queryElasticsearch docsElasticsearch indexElasticsearch docsSearch SettingsElasticsearch docsAgent metricsAgentCreating SLOs via TerraformTerraformCreating agents via TerraformTerraform
Last updated on
On this page