Nobl9 agent

Reading time: 0 minute(s) (0 words)

The Nobl9 agent is a lightweight application for retrieving SLI metrics from data sources you configure and sending data back to the Nobl9 backend for processing and calculating the reliability of your system. You can run the agent in your Kubernetes cluster or as a Docker container.

The Nobl9 agent runs queries for your SLOs; these queries are tailored to the specific data source and sent using native APIs.
By default, the Nobl9 agent queries a data source every minute to capture the last-minute data. However, this frequency can be customized to fit your specific needs.

Deprecation of Older Agent Versions

Starting from release 1.66, any agent versions equal to or earlier than 0.55.3 will be deprecated and rendered non-functional.

If any of your sources currently use deprecated agents, make sure they are updated to the latest agent version.

Supported platforms and architectures

• linux/amd64
• linux/arm64

Nobl9 agent requirements

• Nobl9 agent deployment: use any Kubernetes cluster or Docker environment.

  Agent testing

  A Docker environment on your local machine with proper firewall access is enough for testing purposes. However, we do not recommend using it past the initial test, as the data flow will stop when your machine sleeps.

• For access to the most recent functionality, regularly update your agent.

	Latest stable version	Latest beta version
Docker images	nobl9/agent:0.82.2	nobl9/agent:0.84.0-beta
Command to run	docker pull nobl9/agent:latest	docker pull nobl9/agent:latest-beta

Important

Nobl9 strongly recommends regularly updating agents to the newest versions.

To set up automatic agent updates during restarting, replace the agent version number with the nobl9/agent:latest or nobl9/agent:latest-beta tags in your Kubernetes or Docker configuration for agent deployment.

Learn more about updating the agent.

Read more about agent releases.

Creating the agent

To ensure data transmission from your data source to Nobl9, it may be necessary to list Nobl9 IP addresses as trusted.

IP addresses to add to your allowlist:

• 18.159.114.21
• 18.158.132.186
• 3.64.154.26

⚠ Applies to app.nobl9.com only. In all other cases, contact Nobl9 support.

Creating the agent in the Nobl9 Web

Follow these steps to create your agent in the UI:

1. Go to Integrations.
2. Click .
3. Select the relevant source and choose Agent.

Find integration-specific details in the Agent Configuration in the UI section under the required data source page.

Creating the agent through sloctl

To create your agent through sloctl, you can apply the YAML file to deploy the agent by using the sloctl apply command, for example:

sloctl apply -f ./agent.yaml

Find the source-specific YAML definition of agents under the Sources > Agent connection method > sloctl sections for your required data source.

caution

You can create only one agent in one YAML file.

note

If you describe infrastructure as code, you might also consider creating agents via Terraform. You can find more details in our Terraform documentation.

Deploying the Nobl9 agent

When you add a new agent using the Nobl9 Web or sloctl, Nobl9 automatically generates a Kubernetes configuration in YAML and a Docker command line for you:

• On the Nobl9 Web, the configurations are available immediately.

• For sloctl:

  1. Go to the Integrations > Sources tab.
  2. Open details of your required data source—the configuration YAMLs are available under the Agent configuration tab.
  3. Swap in your own credentials (e.g., <API_TOKEN>, <CREDENTIALS_FILE>).
     Follow the instructions on the Web to find out what credentials you need to pass, as this depends on the data source.

Image 1: Agent configuration

Deploying the agent in Kubernetes

To deploy the Nobl9 agent in your Kubernetes cluster, copy and paste the generated YAML into your Kubernetes configuration.

You can also pass the credentials as a mounted volume. For this, replace the explicit secrets in your agent deployment configuration with the path to the file with secrets.

1. Remove the client_id and client_secret in kind: Secret:

   apiVersion: v1
   kind: Secret
   metadata:
     name: nobl9-agent
     namespace: default
   type: Opaque
   stringData:
     client_id: "YOUR_CLIENT_ID"
     client_secret: "YOUR_CLIENT_SECRET"

2. Remove N9_CLIENT_ID and N9_CLIENT_SECRET in kind: Deployment:

   - name: N9_CLIENT_ID
   valueFrom:
       secretKeyRef:
           key: client_id
           name: nobl9-agent
   - name: N9_CLIENT_SECRET
   valueFrom:
       secretKeyRef:
       key: client_secret
       name: nobl9-agent

3. Add volumes in kind: Deployment:

spec:
      containers:
        - name: agent-container
          image: nobl9/agent:0.82.2
          resources:
            requests:
              memory: "900Mi"
              cpu: "0.2"
          env:
            ....
            - name: N9_INTAKE_URL
              value: "YOUR_VALUE"
            - name: N9_QUERYENGINE_URL
              value: "YOUR_VALUE"
            - name: N9_NATS_URL
              value: "YOUR_VALUE"
          volumeMounts:
            - name: volume-name
              mountPath: /etc/n9/
      volumes:
        - name: volume-name
          ...

By default, the .env file mounted path is /etc/n9/secrets.env. You can specify another path using the N9_SECRETS_PATH environment variable.

Example

Specify the secrets in the secrets.env file:

 N9_CLIENT_ID: <YOUR_N9_CLIENT_ID>
 N9_CLIENT_SECRET: <YOUR_N9_CLIENT_SECRET>

Set the path to your secrets.env in your agent configuration:

    apiVersion: v1
    kind: Deployment
    ...
    spec:
      containers:
      ...
        env:
          - name: N9_SECRETS_PATH
            value: /etc/n9/secrets.env

Deploying the agent in a Docker container

As a base image, the Nobl9 agent uses a distroless image: gcr.io/distroless/static-debian11.

When you add a new data source through the Nobl9 UI, a Docker command line is generated automatically for you to use to deploy the agent.

Swap in your own credentials (e.g., <API_TOKEN>, <CREDENTIALS_FILE>); instructions in the UI specify the credentials to pass.

Image 2: Docker invocation for the Nobl9 agent

Checking the agent connection in the Nobl9 Web

To verify that the agent has successfully connected to Nobl9, check for a valid timestamp in the Last Connection field in the UI.

Image 3: Agent connection status in the Nobl9 Web

caution

Note that the Connected status does not indicate that the agent is connected to the specified data source, only that it has successfully established a connection to the Nobl9 backend.

If you're using Prometheus, you can also check the connection status by scraping in from agent's /health endpoint. Check Agent metrics for more details.

Updating the Nobl9 agent

To update your Nobl9 agent, apply the new configuration file with the latest image of the agent in your Kubernetes cluster or Docker container.

Go to Integrations and click Update in the required agent row, next to the version:

Image 4: Nobl9 agent update

The new Docker and Kubernetes configuration will be applied for you immediately, copy-paste the generated file and apply it to your Kubernetes cluster or run it in your Docker container:

Image 5: Nobl9 agent update configuration

To pull the latest agent version from Docker Hub, run one of the following commands:

• docker pull nobl9/agent:latest to update the agent stable version to the latest
• docker pull nobl9/agent:latest-beta to update the agent beta version

To set up automatic agent updates on restarting, use the latest agent version tags instead of the agent version in your Kubernetes or Docker configuration for agent deployment:

• nobl9/agent:latest for the agent stable version
• nobl9/agent:latest-beta for the agent beta version

With them, agent is updated automatically when restarting.

For example, a Kubernetes configuration:

apiVersion: apps/v1
kind: Deployment
spec:
  name: nobl9-agent
  containers:
    - name: agent-container
      image: nobl9/agent:latest
      # The rest of your agent deployment YAML

Exposing the agent to metrics

To expose the agent's internal metrics, you must make the following changes in your Kubernetes cluster or Docker invocation:

• To deploy the agent, add the environment variable N9_METRICS_PORT, specifying the port on which you would like to expose the metrics (e.g., N9_METRICS_PORT=9876). These metrics are then available for scraping at /metrics.

• If you deploy the agent behind a firewall, make any changes required to allow your metrics system to scrape that port.

• The agent must be able to open connections to https://app.nobl9.com/api/input to send data to Nobl9.

Customizing query delay

Nobl9 agent supports a QUERY_DELAY environment variable that allows to set a customized delay for queries when pulling the data from the data source.

note

You can also set the Query delay via Nobl9 the Web or sloctl (the minimum agent's version is 0.65.0-beta09).

Check the Query delay documentation for more details.

Nobl9, by default, tries to pull data from the previous minute. Sometimes the data in the observability platform may not be available, and, as such, specifying the query delay variable allows you to pull data from a time further than one minute in the past.

The following table specifies the list of the QUERY_DELAY environment variable names specific to each data source:

Data source	Variable name
Amazon CloudWatch	CW_QUERY_DELAY
Amazon Redshift	REDSHIFT_QUERY_DELAY
AppDynamics	APPD_QUERY_DELAY
Azure Monitor beta	AZURE_MONITOR_QUERY_DELAY
Azure Monitor managed service for Prometheus beta	PROM_QUERY_DELAY
BigQuery	BIGQUERY_QUERY_DELAY
DataDog	DATADOG_QUERY_DELAY
Dynatrace	DYNATRACE_QUERY_DELAY
Elasticsearch	ES_QUERY_DELAY
Google Cloud Monitoring	GCM_QUERY_DELAY
Grafana loki	GRAFANA_LOKI_QUERY_DELAY
Graphite	GRAPHITE_QUERY_DELAY
InfluxDB	INFLUXDB_QUERY_DELAY
Instana	INSTANA_QUERY_DELAY
LogicMonitor beta	LOGIC_MONITOR_QUERY_DELAY
NewRelic	NR_QUERY_DELAY
OpenTSDB	OPENTSDB_QUERY_DELAY
Pingdom	PINGDOM_QUERY_DELAY
Prometheus	PROM_QUERY_DELAY
ServiceNow Cloud Observability	LS_QUERY_DELAY
Splunk	SPLUNK_QUERY_DELAY
Sumo Logic	SUMOLOGIC_QUERY_DELAY
Thousandeyes	THOUSANDEYES_QUERY_DELAY

note

The variable value must be specified in the Golang Duration format. For more information, refer to the go documentation.

The following is an example of YAML configuration with the DYNATRACE_QUERY_DELAY variable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nobl9-agent-nobl9--default-dynatrace-agent
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      nobl9-agent-name: dynatrace-agent
      nobl9-agent-project: default
      nobl9-agent-organization: nobl9-org
  template:
    metadata:
      labels:
        nobl9-agent-name: dynatrace-agent
        nobl9-agent-project: default
        nobl9-agent-organization: nobl9-org
    spec:
      containers:
        - name: agent-container
          image: nobl9/agent:0.82.2
          resources:
            requests:
              memory: "350Mi"
              cpu: "0.1"
          env:
            - name: N9_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  key: client_id
                  name: nobl9-agent-nobl9-default-dynatrace-agent
            - name: N9_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  key: client_secret
                  name: nobl9-agent-nobl9-default-dynatrace-agent
            - name: DYNATRACE_TOKEN
              valueFrom:
                secretKeyRef:
                  key: dynatrace_token
                  name: nobl9-agent-nobl9-default-dynatrace-agent
            - name: DYNATRACE_QUERY_DELAY
              value: 60s

Useful links

For a more in-depth look, consult additional resources:

Agent timestamps cache persistenceAgentAgent metricsAgentHelm chartsAgent