Prometheus
Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible query language and real-time alerting.
Prometheus parameters and supported features in Nobl9
- General support:
- Release channel: Stable, Beta
- Connection method: Agent
- Replay and SLI Analyzer: Historical data limit 30 days
- Event logs: Not supported
- Query checker: Not supported
- Query parameters retrieval: Supported
- Timestamp cache persistence: Supported
- Query parameters:
- Query interval: 1 min
- Query delay: 0 sec
- Jitter: 15 sec
- Timeout: 30 sec
- Agent details and minimum required versions for supported features:
- Plugin name: n9prometheus
- Query delay environment variable: PROM_QUERY_DELAY
- Replay and SLI Analyzer: 0.65.0
- Query parameters retrieval: 0.73.2
- Timestamp cache persistence: 0.65.0
- Custom HTTP headers: 0.88.0 / 0.83.0-beta
Authentication
Prometheus does not provide an authentication layer, the Nobl9 agent only collects the URL for the Prometheus integration definition. Authentication is up to the user. Operators are expected to run an authenticating reverse proxy in front of their services, such as NGINX using basic auth or an OAuth2 proxy.
URL
Prometheus agent makes requests to Range Queries | Prometheus documentation API endpoint in the form /api/v1/query_range
. For example:
GET /api/v1/query_range
POST /api/v1/query_range
Hence, do not include the above API path in the URL. Specify only the base URL for
the Prometheus server. For example, if your Prometheus server is available under <http://prometheus.example.com>
and you access API via <http://prometheus.example.com/api/v1>
, use only <http://prometheus.example.com>
.
Other APIs or Web UIs have similar path endings, which should also be omitted, for example, the /graph
part of the path.
The Prometheus integration does not integrate directly with data exposed from services in the Prometheus Format | Prometheus documentation, usually under /metrics
path. Do not set the URL to metrics exposed directly from such a service.
Basic authentication
Since Prometheus does not provide an authentication layer, the authentication method is up to the users. Normally, Loki's users are expected to run an authenticating reverse proxy in front of their services, such as NGINX
using basic_auth
proxy.
If that's the method you use, the Nobl9 agent version equal to or higher than 0.40.0, allows you to send an additional Authorization request header with the basic_auth
. Refer to the section below for more details.
Bearer token authentication
You can also authenticate the Nobl9 Prometheus agent using bearer token. If you choose this method, you need to specify the variables for AUTH_METHOD
and BEARER_TOKEN
when deploying your Prometheus agent in Docker or Kubernetes. Refer to the section below for more details.
Adding Prometheus as a data source
You can add the Prometheus data source using the agent connection method.
Nobl9 Web
Follow the instructions below to create your Prometheus agent connection:
- Navigate to Integrations > Sources.
- Click
.
- Click the required Source button.
- Choose Agent.
-
Select one of the following Release Channels:
- The
stable
channel is fully tested by the Nobl9 team. It represents the final product; however, this channel does not contain all the new features of abeta
release. Use it to avoid crashes and other limitations. - The
beta
channel is under active development. Here, you can check out new features and improvements without the risk of affecting any viable SLOs. Remember that features in this channel can change.
- The
-
Add the URL to connect to your data source (mandatory).
Refer to the Authentication section for more details.
- Select a Project (mandatory).
Project is a way to organize your Nobl9 resources and manage access to them.
When Project is skipped, Nobl9 uses thedefault
project. - Enter a Display Name (optional).
Spaces are allowed. - Enter a Name (mandatory).
The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example,my-project-1
). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result. - Enter a Description (optional).
Provide extra details about it, its purpose, responsible persons, etc.
Up to 1050 characters. - Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
- The default value in Prometheus integration for Query delay is
0 seconds
.
Changing the query delayChanging the query delay can affect your SLI data.
Learn more about query delay and its impact. - The default value in Prometheus integration for Query delay is
- Enter a Maximum Period for Historical Data Retrieval.
- This value defines how far back in the past your data will be retrieved when replaying your SLO based on this data source.
- The maximum period value depends on the data source.
Find the maximum value for your data source. - A greater period can extend the loading time when creating an SLO.
- The value must be a positive integer.
- Enter a Default Period for Historical Data Retrieval.
- It is used by SLOs connected to this data source.
- The value must be a positive integer or
0
. - By default, this value is set to 0. When you set it to
>0
, you will create SLOs with Replay.
- Click Add Data Source
- Deploy your agent in a Kubernetes cluster or Docker container.