Skip to main content

Honeycomb

Reading time: 0 minute(s) (0 words)

Honeycomb is a tool for code performance analysis and troubleshooting. With Honeycomb as the Nobl9 data source, you can create SLOs based on Honeycomb metrics.

Scope of support​

Leveraging Honeycomb metrics, you can create SLOs with bad over total ratio metrics.
Nobl9 integration with Honeycomb also supports direct logs for the direct connection method, SLI Analyzer, and historical data retrieval through Replay.

Prerequisites​

The minimum required agent version is 0.70.0-beta05.

Authentication​

  1. A Honeycomb account.
  2. A Honeycomb API Key.
    It's a 22-character string containing letters Aa-Zz and numbers 0-9.

Adding Honeycomb as a data source​

Currently, the Honeycomb data source is available in the Beta channel only.

To ensure data transmission between Nobl9 and your data source, it may be necessary to list Nobl9 IP addresses as trusted.

IP addresses to add to your allowlist:
  • 18.159.114.21
  • 18.158.132.186
  • 3.64.154.26
⚠ Applies to app.nobl9.com only. In all other cases, contact Nobl9 support.

You can add the Honeycomb data source using the direct or agent connection methods. For both methods, start with these steps:

  1. Navigate to Integrations > Sources.
  2. Click .
    The Data Source wizard opens.
  3. Select Honeycomb.

Honeycomb direct​

Direct configuration in the UI​

  1. Click Direct.

  2. Enter your Honeycomb API Key.

  1. Select a Project.
    Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
  2. Enter a Display Name.
    You can enter a user-friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
    • The default value in Honeycomb integration for Query delay is 5 minutes.
    info
    Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
  6. Enter a Maximum Period for Historical Data Retrieval.
    • This value defines how far back in the past your data will be retrieved.
    • The value for the maximum period of data retrieval depends on the data source. Check the Replay documentation for details.
    • A greater period can extend the loading time when creating an SLO.
      • The value must be a positive integer.
  7. Enter a Default Period for Historical Data Retrieval.
    • It is used by SLOs connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
  8. Click Add Data Source.

Direct using CLI - YAML configuration​

- apiVersion: n9/v1alpha
kind: Direct
metadata:
name: honeycomb-direct
displayName: Honeycomb
project: honeycomb
spec:
sourceOf:
- Metrics
- Services
honeycomb:
apiKey: "" # secret
releaseChannel: beta # string, one of: beta || stable. When adding a new Direct without specifying the releaseChannel, the default release channel will be set to beta
queryDelay:
unit: Minute # string, one of: Second || Minute
value: 2 # numeric, must be a number lesser than 1440 minutes (24 hours)
historicalDataRetrieval:
maxDuration:
value: 7
unit: Day # allowed values: Minute, Hour, Day
defaultDuration:
value: 0
unit: Day # allowed values: Minute, Hour, Day

Honeycomb agent​

Agent configuration in the UI​

  1. Click Agent.
  1. Select a Project.
    Specifying a project is helpful when multiple users are spread across multiple teams or projects. When the Project field is left blank, Nobl9 uses the default project.
  2. Enter a Display Name.
    You can enter a user-friendly name with spaces in this field.
  3. Enter a Name.
    The name is mandatory and can only contain lowercase, alphanumeric characters, and dashes (for example, my-project-1). Nobl9 duplicates the display name here, transforming it into the supported format, but you can edit the result.
  4. Enter a Description.
    Here you can add details such as who is responsible for the integration (team/owner) and the purpose of creating it.
  5. Specify the Query delay to set a customized delay for queries when pulling the data from the data source.
    • The default value in Honeycomb integration for Query delay is 5 minutes.
    info
    Changing the Query delay may affect your SLI data. For more details, check the Query delay documentation.
  6. Enter a Maximum Period for Historical Data Retrieval.
    • This value defines how far back in the past your data will be retrieved.
    • The value for the maximum period of data retrieval depends on the data source. Check the Replay documentation for details.
    • A greater period can extend the loading time when creating an SLO.
      • The value must be a positive integer.
  7. Enter a Default Period for Historical Data Retrieval.
    • It is used by SLOs connected to this data source.
    • The value must be a positive integer or 0.
    • By default, this value is set to 0. When you set it to >0, you will create SLOs with Replay.
  8. Click Add Data Source.

Agent using CLI - YAML configuration​

Nobl9 integration with Honeycomb supports the HONEYCOMB_QUERY_DELAY environment variable.

- apiVersion: n9/v1alpha
kind: Agent
metadata:
name: honeycomb-agent
displayName: Honeycomb
project: honeycomb
spec:
sourceOf:
- Metrics
- Services
honeycomb: { }
releaseChannel: beta # string, one of: beta || stable
queryDelay:
unit: Minute # string, one of: Second || Minute
value: 2 # numeric, must be a number lesser than 1440 minutes (24 hours)
historicalDataRetrieval:
maxDuration:
value: 7
unit: Day # allowed values: Minute, Hour, Day
defaultDuration:
value: 0
unit: Day # allowed values: Minute, Hour, Day

Deploying the Honeycomb agent​

When you add the data source, Nobl9 automatically generates a Kubernetes configuration and a Docker command line for you to use to deploy the agent. Both of these are available in the web UI, under the Agent Configuration section. Replace <N9_HONEYCOMB_API_KEY> with your Honeycomb API Key actual value, and <CLIENT_ID> and <CLIENT_SECRET> with your application's credentials.

If you use Kubernetes, you can apply the supplied YAML config file to a Kubernetes cluster to deploy the agent. The agent facilitates Nobl9 to import your service metrics.

apiVersion: v1
kind: Secret
metadata:
name: nobl9-agent-nobl9-my-honeycomb-project-my-agent
namespace: default
type: Opaque
stringData:
honeycomb_api_key: "<N9_HONEYCOMB_API_KEY>"
client_id: "<CLIENT_ID>"
client_secret: "<CLIENT_SECRET>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nobl9-agent-nobl9-my-honeycomb-project-my-agent
namespace: default
spec:
replicas: 1
selector:
matchLabels:
nobl9-agent-name: "my-agent"
nobl9-agent-project: "my-honeycomb-project"
nobl9-agent-organization: "nobl9"
template:
metadata:
labels:
nobl9-agent-name: "my-agent"
nobl9-agent-project: "my-honeycomb-project"
nobl9-agent-organization: "nobl9"
spec:
containers:
- name: agent-container
image: nobl9/agent:0.71.2-beta
resources:
requests:
memory: "700Mi"
cpu: "0.2"
env:
- name: N9_CLIENT_ID
valueFrom:
secretKeyRef:
key: client_id
name: nobl9-agent-nobl9-my-honeycomb-project-my-agent
- name: N9_HONEYCOMB_API_KEY
valueFrom:
secretKeyRef:
key: honeycomb_api_key
name: nobl9-agent-nobl9-my-honeycomb-project-my-agent
- name: N9_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client_secret
name: nobl9-agent-nobl9-my-honeycomb-project-my-agent
- name: N9_AUTH_SERVER
value: "<AUTH_SERVER_ID>"
- name: N9_METRICS_PORT
value: "9090"

Optionally, you can add the following query customization variables for your Honeycomb integration:

- name: N9_HONEYCOMB_COLLECTION_JITTER
value: "15s" # Default: 15s
- name: N9_HONEYCOMB_QUERY_INTERVAL
value: "1m" # Default: 1m
- name: N9_HONEYCOMB_HTTP_CLIENT_TIMEOUT_DURATION
value: "15s" # Default: 15s

Creating SLOs with Honeycomb​

Creating SLOs in the UI​

Follow the instructions below to create your SLOs with Honeycomb in the UI:

  1. Navigate to Service Level Objectives.

  2. Click .
    The SLO wizard opens.

  3. In step 1, select the Service the SLO will be associated with.

  4. In step 2, select your Honeycomb integration as the Data Source for your SLO.

  5. Modify the Period for Historical Data Retrieval, if required.
    You can extend this period up to seven daysβ€”the maximum value for Honeycomb.

  6. Configure Metric.

    1. Select the metric type:
    • For the Threshold metric, a single time series is evaluated against a threshold.
    • A Ratio metric allows you to enter two-time series for comparison. For the ratio metric, set the additional parameters.
      1. The numerator type:
      • Good metric, a ratio of good requests and total requests.
      • Bad metric, a ratio of bad requests and total requests.
      1. The Data Count Method:
      • Non-incremental, counts incoming metric values one-by-one. So the resulting SLO graph is pike-shaped.
      • Incremental, counts the incoming metric values incrementally, adding every next value to previous values. It results in a constantly increasing SLO graph.
    1. Select Calculation to process the incoming data.
      It's the Honeycomb VISUALIZE clause to calculate data.
    2. Enter metric Attribute (up to 255 characters).
    • It's the Honeycomb WHERE clause that restricts the incoming data.
    • You can also use the Derived Column names from the Dataset Settings > Schema tab in Honeycomb.
  7. In step 3, define a Time Window for the SLO.

  8. In step 4, specify the Error Budget Calculation Method and your Objective(s).

  9. In step 5, add a Name, Description, and other details about your SLO. You can also select Alert Policies and Labels on this screen.

  10. Click Create SLO.

SLOs using Honeycomb - YAML samples​

Field limits for YAML configuration

Use uppercase for calculation methods (VISUALIZE: COUNT, SUM, etc.)

Here’s a YAML sample of the Honeycomb-based SLO using a rawMetric (the threshold metric):

- apiVersion: n9/v1alpha
kind: SLO
metadata:
displayName: Honeycomb sample SLO raw metric
name: honeycomb-sample-slo-raw-metric
project: honeycomb
spec:
alertPolicies: []
budgetingMethod: Occurrences
createdAt: "2023-12-01T15:09:47Z"
description: ""
indicator:
metricSource:
kind: Agent
name: honeycomb
project: my-honeycomb-project
objectives:
- displayName: "Objective 1"
name: objective-1
op: lte
rawMetric:
query:
honeycomb:
dataset: devops-api
attribute: latency # Supported the Derived Column names from Dataset Settings > Schema in Honeycomb. For calculation: COUNT, must return boolean, for non-COUNT, must return int/float.
calculation: "SUM" # one of: "COUNT" "CONCURRENCY" "SUM" "AVG" "COUNT_DISTINCT" "MAX" "MIN" "P001" "P01" "P05" "P10" "P25" "P50" "P75" "P90" "P95" "P99" "P999" "RATE_AVG" "RATE_SUM" "RATE_MAX"
target: 0.9
value: 90
service: my-honeycomb-service
timeWindows:
- calendar:
startTime: "2023-12-01 00:00:00"
timeZone: UTC
count: 1
isRolling: false
period:
begin: "2023-12-07T00:00:00Z"
end: "2023-12-08T00:00:00Z"
unit: Day
status:
timeTravel:
startTime: "2023-11-29T15:12:41Z"
status: draining
unit: Minute
value: 2879

Derived Column names as attribute​

You can use the names of Derived Column from your Honeycomb's Dataset Settings > Schema tab as attribute.

For calculation: COUNT, this field must return a boolean value.
For non-COUNT calculation methods, for example, AVG, P99, etc., this field must return an integer or floating value.

The examples below show the attribute definition for the Derived Column names.

Here's how we can define latency_good:

Good metric, calculation: COUNT
IF (IN($service.name, "my-service"),
IF(LT($duration_ms, 0.45),true))

It gets the events for "my-service" where duration_ms is lower than 0.45 and mark them as true.

Similarly, to define latency_total:

Total metric, calculation: COUNT
IF (IN($service.name, "my-service"),
IF(EXISTS($duration_ms),true))

It gets ALL the duration_ms for "my-service".

Honeycomb metric configuration​

By default, Nobl9 queries Honeycomb Data API once per minute. Metrics are configured with the Honeycomb's Query Builder using the calculations (VISUALIZE) and attribute (WHERE) clauses.

Limitations​

Data source instances

  • We recommend configuring no more than one data source per Honeycomb environment to avoid hitting rate limits.

Replay
To prevent potential data loss risks caused by rate limit collisions when approaching the 7-day limit and ensure data integrity:

  • Replay durations are capped at 6d 23h 45m (10–15 minutes less than the full seven days).
    This limit relates to the first day of the Replay period.

  • Run no more than one Replay at a time for the Honeycomb data source.

SLI per organization
You can add up to 500 SLIs per Honeycomb environment, combining any metric type.

Find API Keys | Honeycomb documentation

Dataset Best Practices | Honeycomb documentation

Query Reference | Honeycomb documentation