Query delay
The query delay (or query customization) feature functions as an offset to your queries. Instead of requesting data for the last minute, Nobl9 can ask your data source for a specific time interval based on the Query delay to ensure SLI data consistency
Take the following example:
- Suppose it is
15:00 UTC
: without a custom query delay; the Nobl9 agent would ask the data source for data from14:59-15:00
. - If you set the query delay to 10 minutes, the Nobl9 agent will instead ask the data source for data from
14:49-14:50
.
Setting a longer query delay can be helpful when your data source may not have a complete data set during the standard Nobl9 agent workflow. Setting a query delay can help with all kinds of SLI problems, for example:
-
Reliability Burn Down chart indicating over 100%.
This might happen in cases when the Nobl9 agent hasn't collected some of the data points due to a delay greater than one minute, and your ratio metric is collecting more good data points than the total:Image 1: Reliability Burndown Chart showing reliability >100% -
Differences in raw data between Nobl9 and a data source.
Sometimes the data in the observability platform may not be available. Specifying a longer Query delay allows you to pull data from a time further than one minute in the past.
You can also configure query delay via Agent's environment variables.
Configuring query delay
Configuring query delay in the UI
You can configure query delay in the Source Configuration Wizard in Integrations > Sources:

Enter the required value for the query delay and click the Save Data Source button to finish your configuration.
Configuring query delay via sloctl
You can configure query delay via sloctl by defining the spec[n].queryDelay.unit
and spec[n].queryDelay.value
fields in your YAML.
Refer to the required source documentation for more details. The following are YAML samples containing the configuration of query delay for Datadog's agent and direct connections:
- Agent YAML
- Direct YAML
Here's a YAML snippet with a defined queryDelay
for Datadog agent:
- apiVersion: n9/v1alpha
kind: Agent
metadata:
name: datadog-agent
project: default
spec:
datadog:
site: com
sourceOf:
- Metrics
releaseChannel: beta # string, one of: beta || stable
queryDelay:
unit: Minute # string, one of: Second || Minute
value: 720 # numeric, must be a number less than 1440 minutes (24 hours)
status:
agentType: Datadog
Here's a YAML snippet with a defined queryDelay
for Datadog direct:
apiVersion: n9/v1alpha
kind: Direct
metadata:
name: datadog-direct
displayName: Datadog direct
project: datadog-direct
spec:
description: direct integration with Datadog
sourceOf: # One or many values from this list are allowed: Metrics, Services
- Metrics
- Services
releaseChannel: beta # string, one of: beta || stable
queryDelay:
unit: Minute # string, one of: Second || Minute
value: 720 # numeric, must be a number less than 1440 minutes (24 hours)
datadog:
site: com
apiKey: "" # secret
applicationKey: "" # secret
If you select a value greater than 24 hours
(only Minutes and Seconds are allowed) sloctl
will return the following error:
Error: agent.yaml: Key: 'Agent.spec.QueryDelayDuration' Error:Field validation for 'QueryDelayDuration' failed on the 'queryDelayDurationBiggerThanMaximumAllowed' tag
We recommend selecting more sensible query delay values. For most data sources, query delay should be set to at most 15 minutes
, but it can vary per specific use case.