Nobl9 agent

Reading time: 0 minute(s) (0 words)

The Nobl9 agent is a lightweight application for retrieving SLI metrics from data sources you configure and sending data back to the Nobl9 backend for processing and calculating the reliability of your system. You can run the agent in your Kubernetes cluster or as a Docker container.

The Nobl9 agent runs queries for your SLOs; these queries are tailored to the specific data source and sent using native APIs.
By default, the Nobl9 agent queries a data source every minute to capture the last-minute data. However, this frequency can be customized to fit your specific needs.

Deprecation of older agent versions

Starting from release 1.66, agent versions equal to or earlier than 0.55.3 are deprecated and rendered non-functional.

If any of your sources currently use deprecated agents, update them to the latest version.

Requirements and support

Systems supported by the Nobl9 agent:

• linux/amd64
• linux/arm64

For the Nobl9 agent deployment, use any Kubernetes cluster or Docker environment.

Nobl9 Agent testing

A Docker environment on your local machine with proper firewall access is enough for testing purposes. However, we do not recommend using it past the initial test, as the data flow will stop when your machine sleeps.

We strongly recommend updating your agent regularly to get the most recent functionality.

	Latest stable version	Latest beta version
Docker images	nobl9/agent:0.88.0	nobl9/agent:0.89.1-beta
Command to run	docker pull nobl9/agent:latest	docker pull nobl9/agent:latest-beta

Automatic updates

To set up automatic agent updates during restarting, replace the agent version number with the nobl9/agent:latest or nobl9/agent:latest-beta tags in your Kubernetes or Docker configuration for agent deployment.

To stay informed about the latest improvements, bug fixes, and new features, see the Nobl9 agent release notes.

Creating the agent

To create the agent, do the following:

1. Add a data source using the agent connection method on the Nobl9 Web or with sloctl.
2. Deploy the created agent in a Kubernetes cluster or Docker container.

To ensure data transmission from your data source to Nobl9, it may be necessary to list Nobl9 IP addresses as trusted.

💻ip allowlist

IP addresses to include in your allowlist for secure access:

If you're using app.nobl9.com instance:

• 18.159.114.21
• 18.158.132.186
• 3.64.154.26

If you're using us1.nobl9.com instance:

• 34.121.54.120
• 34.123.193.191
• 34.134.71.10
• 35.192.105.150
• 35.225.248.37
• 35.226.78.175
• 104.198.44.161

Creating the agent in the Nobl9 Web

Follow these steps to create your agent in the UI:

1. Go to Integrations.
2. Click .
3. Select the relevant source and choose Agent.

Find integration-specific details in the Agent Configuration in the UI section under the required data source page.

Creating the agent with sloctl

To create your agent with sloctl, apply the YAML definition of your agent using the following command:

sloctl apply -f ./agent.yaml

Points to consider

• Find the source-specific YAML definition of agents under the Sources > Adding data sources > Your required data source > Agent connection method > sloctl section of this documentation.
• You can create only one agent in one YAML file.
• If you describe infrastructure as code, consider creating agents using the Nobl9 Terraform provider.

Deploying the Nobl9 agent

Once you add a new agent using the Nobl9 Web or applied your agent YAML definition with sloctl, Nobl9 immediately generates a Kubernetes configuration and a Docker command line. To find them, do the following:

1. Go to the Integrations > Sources tab on the Nobl9 Web.
2. Open details of your required data source. Go to the Agent configuration tab.

Agent configuration

Deploying the agent in Kubernetes

To deploy your Nobl9 agent using Kubernetes, do the following:

Prerequisites: ensure you have access to the required Kubernetes cluster and the permissions to deploy resources.

1. Copy the generated YAML.
2. Apply the copied YAML to your Kubernetes cluster, replacing the placeholders with your actual values.
   For example, using Helm.
3. Check the agent connection.

You can also pass the credentials as a mounted volume. For this, instead of the explicit secrets in your agent deployment configuration, use the path to the file with secrets.

1. Remove the client_id and client_secret in kind: Secret:

   client_id and client_secret under kind: Secret

   apiVersion: v1
   kind: Secret
   metadata:
     name: nobl9-agent
     namespace: default
   type: Opaque
   stringData:
     client_id: "YOUR_CLIENT_ID"
     client_secret: "YOUR_CLIENT_SECRET"

2. Remove N9_CLIENT_ID and N9_CLIENT_SECRET in kind: Deployment:

   N9_CLIENT_ID and N9_CLIENT_SECRET under kind: Deployment

      spec:
           containers:
             # ...
             env:
              - name: N9_CLIENT_ID
                valueFrom:
                  secretKeyRef:
                    key: client_id
                    name: nobl9-agent
              - name: N9_CLIENT_SECRET
                valueFrom:
                  secretKeyRef:
                    key: client_secret
                    name: nobl9-agent

3. Add volumes in kind: Deployment:

   volumes under kind: Deployment

   spec:
         containers:
             env:
               # ...
             volumeMounts:
               - name: volume-name
                 mountPath: /etc/n9/
         volumes:
           - name: volume-name

By default, the .env file mounted path is /etc/n9/secrets.env. To set a different path, do the following:

1. Specify the secrets in the secrets.env file.
2. Set the path to your secrets.env in your agent configuration under the N9_SECRETS_PATH environment variable:

Setting the path to secrets.env

    apiVersion: v1
    kind: Deployment
    # ...
    spec:
      containers:
      # ...
        env:
          - name: N9_SECRETS_PATH
            value: /etc/n9/secrets.env

Deploying the agent in a Docker container

As a base image, the Nobl9 agent uses a distroless image: gcr.io/distroless/static-debian11.
Distroless images are minimal container images designed for security and efficiency, containing only the necessary components to run the application.

To deploy your Nobl9 agent using Docker, ensure Docker is running on your system and do the following:

1. Copy the generated Docker command.
2. Run the command in your terminal, replacing the placeholders with your actual values.
3. Check the agent connection.

Sending custom HTTP headers to a data source

When deploying some Nobl9 agents¹, you can set it up to send custom HTTP headers to your data source.
¹Supported data sources

The following schema applies:

Custom HTTP header schema

[
    {
        "key": "<header key>",
        "values": ["<value1>", "<value2>"]
    }
]

To inject custom headers when deploying your agent, do the following:

1. Provide the required headers encoded as a Base64 string in a JSON file:

Encoded custom headers template

 data:
 # base64 encoded [{"key":"Authorization","values":["Bearer my-token"]}]
 custom-headers.json: base64_encoded_string

Replace the base64_encoded_string placeholder with your encoded string.

2. Reference your Secret object in the N9_CUSTOM_HTTP_HEADER environment variable:

Referencing a secret object in agent deployment in a Kubernetes cluster

 - name: N9_CUSTOM_HTTP_HEADERS
   valueFrom:
     secretKeyRef:
       name: your-nobl9-agent-name
       key: custom-headers.json

The encoded string is stored as data within a Kubernetes Secret object.

Custom HTTP headers support in data sources

Data source	Custom HTTP headers
Amazon Prometheus	0.83.0-beta
Azure Monitor managed service for Prometheus	0.83.0-beta
Grafana Loki	0.88.0-beta
Prometheus	0.83.0-beta
Splunk	0.89.1-beta

Checking the agent connection in the Nobl9 Web

To verify if the agent is connected to Nobl9, check the timestamp under Connection Status in your data source details:

Agent connection status in the Nobl9 Web

Connection side

The connection status indicates your agent's connection to the Nobl9 backend only. It doesn't provide information about agent's connection to your data source.

For Prometheus-compatible data sources: to check the connection status, you can also scrape in from agent's /health endpoint. Learn more about Agent metrics.

Updating the Nobl9 agent

To update your Nobl9 agent, apply the new configuration file with the latest image of the agent in your Kubernetes cluster or Docker container.

Go to Integrations and click Update in the required agent row, next to the version:

Nobl9 agent update

The new Docker and Kubernetes configuration will be applied for you immediately, copy-paste the generated file and apply it to your Kubernetes cluster or run it in your Docker container:

Nobl9 agent update configuration

To pull the latest agent version from Docker Hub, run one of the following commands:

• docker pull nobl9/agent:latest to update the agent stable version to the latest
• docker pull nobl9/agent:latest-beta to update the agent beta version

To set up automatic agent updates on restarting, use the latest agent version tags instead of the agent version in your Kubernetes or Docker configuration for agent deployment:

• nobl9/agent:latest for the agent stable version
• nobl9/agent:latest-beta for the agent beta version

With them, agent is updated automatically when restarting.

For example, a Kubernetes configuration:

Using the latest agent version tags in the agent configuration

apiVersion: apps/v1
kind: Deployment
spec:
  name: nobl9-agent
  containers:
    - name: agent-container
      image: nobl9/agent:latest
      # The rest of your agent deployment YAML

Exposing the agent to metrics

To expose the agent's internal metrics, modify its Kubernetes configuration or Docker command line as follows:

• Deploy the agent: add the N9_METRICS_PORT environment variable to specify the port required for metric exposure (for example, N9_METRICS_PORT=9876). This makes metrics available for scraping at /metrics.

• Deploy the agent behind a firewall: make any changes required to allow your metrics system to scrape that port.

• The agent must be able to open connections to https://app.nobl9.com/api/input to send data to Nobl9.

Customizing query delay

The QUERY_DELAY environment variable allows customizing the delay applied to sending queries to your data source.

You can set the query delay via Nobl9 the Web or sloctl (requires agent version 0.65.0-beta09 or later).

By default, Nobl9 retrieves data from the previous minute. To increase this range—for example, if your data source cannot send data from the previous minute—you can set a greater query delay, so your agent will query data from more than one minute in the past. For this, set the required value using the QUERY_DELAY variable, corresponding to your agent.

The following table specifies the list of the QUERY_DELAY environment variable names specific to each data source:

Data source	Variable name
Amazon CloudWatch	CW_QUERY_DELAY
Amazon Redshift	REDSHIFT_QUERY_DELAY
AppDynamics	APPD_QUERY_DELAY
Azure Monitor beta	AZURE_MONITOR_QUERY_DELAY
Azure Monitor managed service for Prometheus beta	PROM_QUERY_DELAY
BigQuery	BIGQUERY_QUERY_DELAY
DataDog	DATADOG_QUERY_DELAY
Dynatrace	DYNATRACE_QUERY_DELAY
Elasticsearch	ES_QUERY_DELAY
Google Cloud Monitoring	GCM_QUERY_DELAY
Grafana loki	GRAFANA_LOKI_QUERY_DELAY
Graphite	GRAPHITE_QUERY_DELAY
InfluxDB	INFLUXDB_QUERY_DELAY
Instana	INSTANA_QUERY_DELAY
LogicMonitor beta	LOGIC_MONITOR_QUERY_DELAY
NewRelic	NR_QUERY_DELAY
OpenTSDB	OPENTSDB_QUERY_DELAY
Pingdom	PINGDOM_QUERY_DELAY
Prometheus	PROM_QUERY_DELAY
ServiceNow Cloud Observability	LS_QUERY_DELAY
Splunk	SPLUNK_QUERY_DELAY
Sumo Logic	SUMOLOGIC_QUERY_DELAY
Thousandeyes	THOUSANDEYES_QUERY_DELAY

Value format requirement

Set the query delay value in the Golang Duration format.

The following is an example of YAML configuration with the DYNATRACE_QUERY_DELAY variable:

Customizing the query delay on the Dynatrace Nobl9 agent deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-dynatrace
spec:
  template:
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:0.88.0
          env:
            - name: DYNATRACE_QUERY_DELAY
              value: 60s

Useful links

For a more in-depth look, consult additional resources:

Nobl9 agent release notesRelease notesHelm chartsNobl9 agentAgent metricsNobl9 agentAgent timestamps cache persistenceNobl9 agentQuery customizationNobl9 featuresQuery delayNobl9 featuresNobl9 agent beta version functionalityGetting startednobl9/agentdockerhubNobl9 ProviderTerraform registryData sources available in Nobl9SourcesNobl9 agent troubleshootingTroubleshooting