Prometheus

Reading time: 0 minute(s) (0 words)

Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible query language and real-time alerting.

Prometheus parameters and supported features in Nobl9

General support:: Release channel: Stable, Beta; Connection method: Agent; Replay and SLI Analyzer: Historical data limit 30 days; Event logs: Not supported; Query checker: Not supported; Query parameters retrieval: Supported; Timestamp cache persistence: Supported
Query parameters:: Query interval: 1 min; Query delay: 0 sec; Jitter: 15 sec; Timeout: 30 sec
Agent details and minimum required versions for supported features:: Plugin name: n9prometheus; Query delay environment variable: PROM_QUERY_DELAY; Replay and SLI Analyzer: 0.65.0; Query parameters retrieval: 0.73.2; Timestamp cache persistence: 0.65.0; Custom HTTP headers: 0.88.0 / 0.83.0-beta

Creating SLOs with Prometheus

Nobl9 Web

Follow the instructions below to create your SLOs with Prometheus on the Nobl9 Web:

Navigate to Service Level Objectives.
Click .
Select a Service.
It will be the location for your SLO in Nobl9.
Select your Prometheus data source.
Modify Period for Historical Data Retrieval, if necessary.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on Prometheus.
- A longer period can extend the data loading time for your SLO.
- Must be a positive whole number up to the maximum period value you've set when adding the Prometheus data source.
Non-editable Replay period
Non-editable Replay period indicates that the maximum period for historical data retrieval set for your Prometheus data source is set to zero.
Adjust the data source settings to create the SLO with Replay.
Metric refers to the way of calculating and interpreting calculate and interpret data from your data source.
- Threshold metric is defined by a single numerical value (the threshold) that separates satisfactory performance from unsatisfactory performance. It's represented by a single time series evaluated against the threshold.
- Ratio metric expresses the performance as a fraction or proportion, typically by dividing the number of successful events by the total number of potential events (successes + failures). It's represented by two-time series for comparison for good events and total events.
  For ratio metrics, select the Data count method.
  SLI values for good and total
  When choosing the query for the ratio SLI (countMetrics), keep in mind that the values resulting from that query for both good and total:
  Must be positive.
  While we recommend using integers, fractions are also acceptable.
  If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
  Shouldn't be larger than 1e+20.

Enter the query.

Click to open query samples

Threshold metric
myapp_server_requestMsec{host="*",job="nginx"}

Ratio metric, numerator
sum(production_http_response_time_seconds_hist_bucket{method=~"GET|POST",status=~"2..|3..",le="1"})

Ratio metric, denominator
sum(production_http_response_time_seconds_hist_bucket{method=~"GET|POST",le="+Inf"})

Define the Time window for your SLO:
- Rolling time windows constantly move forward as time passes. This type can help track the most recent events.
- Calendar-aligned time windows are usable for SLOs intended to map to business metrics measured on a calendar-aligned basis.
Configure the Error budget calculation method and Objectives:
- Occurrences method counts good attempts against the count of total attempts.
- Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
- You can define up to 12 objectives for an SLO.
Add the Display name, Name, and other settings for your SLO:
- Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
  Use only lowercase letters, numbers, and dashes.
- Select No data anomaly alert to receive notifications when your SLO stops reporting data for a specified period:
  - Choose up to five supported Alert methods.
  - Specify the delay period before Nobl9 sends an alert about the missing data.
    From 5 minutes to 31 days. Default: 15 minutes
- Add alert policies, labels, and links, if required.
  Limits per SLO: 20 alert policies or links, 30 labels.
Click CREATE SLO.

SLO configuration use case

Check the SLO configuration use case for a real-life SLO example.

YAML

Threshold (rawMetric)
Ratio (countMetric)

Sample Prometheus threshold SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Prometheus SLO
  indicator:
    metricSource:
      name: prometheus
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 200
      name: ok
      target: 0.95
      rawMetric:
        query:
          prometheus:
            promql: api_server_requestMsec{host="*",job="nginx"}
      op: lte
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: '2022-12-01 00:00:00'
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default
      alertAfter: 1h

Sample Prometheus ratio SLO
apiVersion: n9/v1alpha
kind: SLO
metadata:
  name: api-server-slo
  displayName: API Server SLO
  project: default
  labels:
    area:
      - latency
      - slow-check
    env:
      - prod
      - dev
    region:
      - us
      - eu
    team:
      - green
      - sales
  annotations:
    area: latency
    env: prod
    region: us
    team: sales
spec:
  description: Example Prometheus SLO
  indicator:
    metricSource:
      name: prometheus
      project: default
      kind: Agent
  budgetingMethod: Occurrences
  objectives:
    - displayName: Good response (200)
      value: 1
      name: ok
      target: 0.95
      countMetrics:
        incremental: true
        good:
          prometheus:
            promql: >-
              sum(http_request_duration_seconds_bucket{handler="/api/v1/slos",le="2.5"})
        total:
          prometheus:
            promql: sum(http_request_duration_seconds_count{handler="/api/v1/slos"})
      primary: true
  service: api-server
  timeWindows:
    - unit: Month
      count: 1
      isRolling: false
      calendar:
        startTime: '2022-12-01 00:00:00'
        timeZone: UTC
  alertPolicies:
    - fast-burn-5x-for-last-10m
  attachments:
    - url: https://docs.nobl9.com
      displayName: Nobl9 Documentation
  anomalyConfig:
    noData:
      alertMethods:
        - name: slack-notification
          project: default
      alertAfter: 1h

Click to open field reference

Field	Type	Description
`apiVersion` mandatory	string	API version. Use `n9/v1alpha`
`kind` mandatory	string	The resource type. Use `SLO`
Metadata
`metadata.name` mandatory	string	Name identifier for the SLO. Use only lowercase alphanumeric characters
`metadata.displayName`	string	User-friendly SLO name
`metadata.project` mandatory	string	The name identifier of the project where you need to host your SLO
`metadata.labels`	object (map: string[])	Grouping labels for filtering or viewing
`metadata.annotations`	object (map: string)	Flat string annotations
Spec
`spec.description`	string	SLO description
`spec.indicator.metricSource.name` mandatory	string	Data source name
`spec.indicator.metricSource.project` mandatory	string	Project containing the data source
`spec.indicator.metricSource.kind` mandatory	string	Data source connection method. Can be Agent or Direct
`spec.budgetingMethod` mandatory	enum	Error budget calculation method. Can be Occurrences or Time slices
`spec.objectives` mandatory	array	Your SLO objective definition, up to 12 objectives per SLO.
`spec.objectives[].displayName`	string	User-friendly objective name
`spec.objectives[].value` mandatory	number	Data point values that is considered "good" (e.g., `200.0`). In SLOs with two or more objectives, keep each objective's value unique. In ratio (`count`) metrics, `value` is retained for legacy purposes.
`spec.objectives[].name` mandatory	string	Name identifier for this objective
`spec.objectives[].op` mandatory	string (enum)	Operator for objective. One of: `lte` (less than or equal to) `lt` (less than) `gte` (greater than or equal to) `gt` (greater than)
`spec.objectives[].target` mandatory	float	The percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is `0.95`, the good performance is expected to be observed in at least 95% of the time window)
`spec.objectives[].rawMetric`/`.countMetric` mandatory	object	The metric type indicator. Set: `rawMetric` for a threshold metric `countMetric` for a ratio metric. A ratio metric requires the additional fields: `countMetric.incremental` (boolean) the data count method `countMetric.good`/`.bad` and `countMetric.total` a numerator and denominator queries
`spec.objectives[].countMetric.incremental` mandatory	boolean	The data count method for a ratio (`countMetric`) metric type
`spec.objectives[].primary`	boolean	The indicator of a primary SLO objective
`spec.service` mandatory	string	The name identifier of a service to host this SLO. The service must exist in the project specified in `metadata.project`
`spec.timeWindows` mandatory	array	Defines SLO time window for error budget calculation. Set: `isRolling: true` for the rolling time window type `isRolling: false` for the calendar-aligned type
`spec.timeWindows.unit` mandatory	integer	The time window units. One of: `Day` \| `Hour` \| `Minute` for the rolling time window `Year` \| `Quarter` \| `Month` \| `Week` \| `Day` for the calendar-aligned time window
`spec.timeWindows.count` mandatory	integer	The number of units in a time window
`spec.timeWindows.startTime`	string	Mandatory for calendar-aligned time windows. Date and time in the format `YYYY-MM-DDTHH:mm:ss`
`spec.timeWindows.timeZone`	string	Mandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
`spec.timeWindows.isRolling` mandatory	boolean	`true` for the rolling time window type `false` for the calendar-aligned type
`spec.alertPolicies`	array	The name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
`spec.attachments`	array	Links to any additional attributes of this SLO
`spec.anomalyConfig`	object	Settings for a manual no data anomaly detection rule
`spec.noData.alertMethods`	array	List of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the `name` and `project` fields
`spec.noData.alertAfter`	string	Waiting time before sending a no-data notification. Must be `5m` to `31d`. Default: `15m`
Source-specific fields
`query.promql` mandatory	string	Your PromQL query.

Querying the Prometheus server

The Nobl9 agent leverages the Prometheus API parameters. It pulls data at a per-minute interval from the Prometheus server.

Useful links

Check out these related guides and references:

Prometheus query language basicsPrometheus documentation

Add Prometheus as a data sourceAdding data sources

Setting up a Prometheus SLO with Nobl9Video

Creating SLOs using TerraformTerraform

Creating SLOs with Prometheus​

Nobl9 Web​

YAML​

Querying the Prometheus server​

Useful links​

Creating SLOs with Prometheus

Nobl9 Web

YAML

Querying the Prometheus server

Useful links