Skip to main content

Agent timestamp cache persistence

Reading time: 0 minute(s) (0 words)

The Nobl9 agent keeps information about the last successfully fetched data in the timestamps cache. When it gets data from a data source, it always attempts to do so beginning from the cached timestamp. The cache is by default kept in memory and therefore is lost when the agent is restarted.

Nobl9 has introduced the timestamps cache persistence (or simply, persistence) feature for the Nobl9 agent.

Scope of support​

The Nobl9 agent timestamps cache persistence is not yet available for all data sources Nobl9 supports.

The table below presents the list of data sources along with their corresponding plugin names. You can also determine the minimum agent version required to use this feature:

Data sourcePlugin nameAgent version
Amazon CloudWatchn9cloudwatchβ‰₯ 0.56.1
Amazon Prometheusn9prometheusβ‰₯ 0.65.0
Amazon Redshiftn9redshiftβ‰₯ 0.56.1
AppDynamicsn9appdβ‰₯ 0.56.1
Azure Monitorn9azure_monitorβ‰₯ 0.69.0-beta01
BigQueryn9bigqueryβ‰₯ 0.56.1
Datadogn9datadogβ‰₯ 0.64.0
Elasticsearchn9elasticsearchβ‰₯ 0.56.1
Dynatracen9dynatraceβ‰₯ 0.56.1
Google Cloud Monitoringn9gcmβ‰₯ 0.56.1
Grafana Lokin9grafana_lokicoming soon
Graphiten9graphitecoming soon
InfluxDBn9influxdbβ‰₯ 0.56.1
Instanan9instanaβ‰₯ 0.56.1
ServiceNow Cloud Observabilityn9lightstepβ‰₯ 0.56.1
New Relicn9newrelicβ‰₯ 0.56.1
OpenTSDBn9opentsdbβ‰₯ 0.56.1
Pingdomn9pingdomβ‰₯ 0.56.1
Prometheusn9prometheusβ‰₯ 0.65.0
Splunk Observabilityn9splunk_observabilityβ‰₯ 0.56.1
Splunkn9splunkβ‰₯ 0.56.1
Sumo Logicn9sumologicβ‰₯ 0.56.1
ThousandEyesn9thousandeyesβ‰₯ 0.56.1

Rules at a glance​

  • Persistence is available for agent versions >= 0.56.1.

  • Persistence is deactivated by default and can be activated with the following optional parameters that can be set by environment variables:

    N9_PERSISTENCE_ENABLED=true
    N9_PERSISTENCE_ENABLED_PLUGINS= # specify one of the supported data sources by providing its plugin name. For more information, refer to the Scope of Support section of the documentation.
    N9_PERSISTENCE_PATH=/var/timestamps-cache
    note

    You can activate or deactivate persistence at any time. To disable this feature, set the value of the N9_PERSISTENCE_ENABLED variable to false.

  • Persistence requires an additional PersistentVolume when the agent is run in a Kubernetes cluster or dedicated disk space when it is run in Docker. Refer to the code snippets below for details.

  • The cache is updated only once the agent has successfully pushed data to the Nobl9 data intake.

  • When restarted, the Nobl9 agent will fetch data starting from the last successfully pushed data point. As a consequence, if the agent has been disabled for a long time, the data source might reject requests for the resulting large time window or not return all the data requested. To prevent these situations, always consult the external data source’s documentation to check its limits.

Examples​

Kubernetes​

Here’s an example Kubernetes deployment YAML spec with persistence support and an additional specification for the PersistentVolumeClaim (highlighted in the snippet below):

apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent
namespace: default
type: Opaque
stringData:
client_id: "unique_client_id"
client_secret: "unique_client_secret"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "nobl9-agent"
nobl9-agent-project: "nobl9-agent"
nobl9-agent-organization: "nobl9"
template:
metadata:
labels:
nobl9-agent-name: "nobl9-agent"
nobl9-agent-project: "nobl9-agent"
nobl9-agent-organization: "nobl9"
spec:
# Use this section if you want to authenticate with IAM role provided through the k8s ServiceAccount.
# serviceAccount: <SERVICE_ACCOUNT>
# serviceAccountName: <SERVICE_ACCOUNT_NAME>
containers:
- name: agent-container
image: nobl9/agent:0.69.4
resources:
requests:
memory: "350Mi"
cpu: "0.1"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent
- name: N9_AUTH_SERVER
value: "auseg9kiegWKEtJZC416"
- name: N9_PERSISTENCE_ENABLED
value: "true"
- name: N9_PERSISTENCE_ENABLED_PLUGINS
value: "n9appd" # specify one of the supported data sources by providing its plugin name
- name: N9_PERSISTENCE_PATH
value: "/var/timestamps-cache"
volumeMounts:
- name: timestamps-cache
mountPath: /var/timestamps-cache
volumes:
- name: timestamps-cache
persistentVolumeClaim:
claimName: nobl9-agent-cache
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nobl9-agent-cache
labels:
app: nobl9
component: agent-cache
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

Docker​

Here’s an example Docker deployment command with persistence support:

docker run -d --restart on-failure \
--name nobl9-agent \
-e N9_CLIENT_SECRET="unique_client_secret" \
-e N9_CLIENT_ID="unique_client_id" \
-e AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>" \
-e AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>" \
-e N9_PERSISTENCE_ENABLED="true" \
-e N9_PERSISTENCE_ENABLED_PLUGINS="plugin_name" \ # specify one of the supported data sources by providing its plugin name
-e N9_PERSISTENCE_PATH="/var/timestamps-cache" \
-v /var/timestamps-cache:/var/timestamps-cache \
nobl9/agent:0.73.2-beta