Skip to main content

Prometheus

Reading time: 0 minute(s) (0 words)

Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible query language and real-time alerting.

Prometheus parameters and supported features in Nobl9
General support:
Release channel: Stable, Beta
Connection method: Agent
Replay and SLI Analyzer: Historical data limit 30 days
Event logs: Not supported
Query checker: Not supported
Query parameters retrieval: Supported
Timestamp cache persistence: Supported
Query parameters:
Query interval: 1 min
Query delay: 0
Jitter: 15 sec
Timeout: 30 sec
Agent details and minimum required versions for supported features:
Plugin name: n9prometheus
Query delay environment variable: PROM_QUERY_DELAY
Replay and SLI Analyzer: 0.65.0
Query parameters retrieval: 0.73.2
Timestamp cache persistence: 0.65.0
Custom HTTP headers: 0.83.0-beta

Creating SLOs with Prometheus​

Nobl9 Web​

Follow the instructions below to create your SLOs with Prometheus on the Nobl9 Web:

  1. Navigate to Service Level Objectives.

  2. Click .

Step 1: Select the service the SLO will be associated with.

Step 2:

  1. Select your Prometheus data source.
  2. Specify Metric and enter the query:

The threshold metric evaluates a single time series against a threshold value you set.

3. Enter the query. For example: myapp_server_requestMsec{host="*",job="nginx"}

Step 3: define a Time Window for your SLO.

  • Rolling time windows are better for tracking the recent user experience of a service.
  • Calendar-aligned windows are best suited for SLOs that are intended to map to business metrics measured on a calendar-aligned basis, such as every calendar month or every quarter.

Step 4: specify the Error Budget Calculation Method and your Objective(s).

  • Occurrences method counts good attempts against the count of total attempts.
  • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
  • You can define up to 12 objectives for an SLO.

See the use case example and the SLO calculations guide for more information on the error budget calculation methods.

Step 5: add the Display name, Name, and other settings for your SLO:

  • Set notification on data, if this option is available for your data source.
    When activated, Nobl9 notifies you if your SLO hasn't received data or received incomplete data for more than 15 minutes.
  • Add alert policies, labels, and links, if required.
    You can add up to 20 links per SLO.

Click Create SLO.

sloctl​

Sample Prometheus threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Prometheus SLO
  indicator:
    metricSource:
      name: prometheus
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          prometheus:
            promql: api_server_requestMsec{host="*",job="nginx"}
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: 2022-12-01T00:00:00.000Z
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default

Specification for metric from Prometheus always has one mandatory field:

  • promql – a Prometheus query in the language called PromQL | Prometheus documentation (Prometheus Query Language) that lets the user select and aggregate time series data in real time.

Querying the Prometheus server​

The Nobl9 agent leverages the Prometheus API parameters. It pulls data at a per-minute interval from the Prometheus server.

Cortex support with Nobl9 Prometheus agent​

Cortex | Cortex documentation is a database based on Prometheus with compatible API. Therefore, it is possible to use Cortex with the Nobl9 Prometheus agent.

Cortex cluster setup is out of the scope of this document and is described in the Cortex documentation. Cortex deployment can be simplified with the official Helm chart.

As described in Cortex Architecture | Cortex documentation, Prometheus API is exposed by the Nginx under default address <http://cortex-nginx/prometheus.> This address can be used as Prometheus URL in the agent configuration panel. The default Prometheus endpoint can be changed according to the API documentation | Cortex documentation. In that case, the agent needs to access the /api/v1/query_range endpoint.

If you want the Nobl9 agent to support a multi-tenancy deployment mode in Cortex, use the following environment variable while deploying Nobl9 agent in Kubernetes (see section above):

          env:
            - name: PROMETHEUS_X_SCOPE_ORG_ID
              value: <X-Scope-OrgID>

of in your Docker deployment:

docker run -d --restart on-failure \
  --name nobl9-agent-nobl9-dev-stable-prometheus \
  -e PROMETHEUS_X_SCOPE_ORG_ID="<X-Scope-OrgID>"

Grafana Cloud support with Nobl9 Prometheus agent​

Grafana Cloud is an observability platform that leverages Prometheus by directly interacting with the Prometheus HTTP API | Prometheus documentation. Therefore, it is possible to use Grafana Cloud solution with the Nobl9 Prometheus agent.

To use Grafana Cloud with Prometheus, you must authenticate your Prometheus agent with the basic_auth proxy. Refer to the section above for more details.

As described in Analyzing metrics usage with the Prometheus API | Grafana Cloud documentation, Prometheus API is exposed through the /api/prom/api/v1/query_range endpoint which is accessed by the Nobl9 agent.

To use Grafana Cloud with Nobl9, you need to append /api/prom/ to the end of the URL you configure your Grafana Source in the Data source wizard for a regular Prometheus data integration. Thus, instead of http://HOST/, you need to enter http://HOST/api/prom/ in the Data source wizard.

For more details, check Grafana Cloud documentation.

Thanos direct with Nobl9 Prometheus agent​

Thanos is High Availability Prometheus setup and can be used with Nobl9 Prometheus agent.

Thanos cluster setup is out of the scope of this document and is described in the Thanos Components documentation.

Thanos exposes Prometheus API using Querier. Querier address must be used as Prometheus URL in Nobl9 agent configuration.

Other​

For a more in-depth look, consult additional resources:
Setting up a prometheus SLO with Nobl9 (video)VideoAgent metricsAgentCreating SLOs via TerraformTerraformCreating agents via TerraformTerraform
Last updated on
On this page