Honeycomb

Reading time: 0 minute(s) (0 words)

Honeycomb is a tool for code performance analysis and troubleshooting. With Honeycomb as the Nobl9 data source, you can create single-query SLOs based on Honeycomb metrics.

Scope of support

• Honeycomb metrics: Single-query ratio SLOs
  • Nobl9 agent minimum version: 0.91.0-beta
• Release channel: Beta, Stable
• Connection method: Agent, Direct
• Replay and SLI Analyzer: Historical data limit is 6 days 23 hours 45 minutes
• Event logs: Full coverage
• Query parameters retrieval: Supported
  • Query interval: 1 min
  • Query delay: 5 min
  • Jitter: 15 sec
  • Timeout: 10 sec
  • Nobl9 agent minimum version: 0.91.0-beta
• Timestamp cache persistence: Supported

Creating SLOs with Honeycomb

Nobl9 Web

Follow the instructions below to create your SLOs with Honeycomb in the UI:

1. Navigate to Service Level Objectives.
2. Click .
3. Select a Service.
   It will be the location for your SLO in Nobl9.
4. Select your Honeycomb data source.
   
5. Modify Period for Historical Data Retrieval, if necessary.

   • This value defines how far back in the past your data will be retrieved when replaying your SLO based on Honeycomb.
   • A longer period can extend the data loading time for your SLO.
   • Must be a positive whole number up to the maximum period value you've set when adding the Honeycomb data source.
   • For your Honeycomb data source, you can extend this period up to 7 days.

   
   

   Non-editable Replay period

   Non-editable Replay period indicates that the maximum period for historical data retrieval set for your Honeycomb data source is set to zero.
   Adjust the data source settings to create the SLO with Replay.
6. Metric refers to the way of calculating and interpreting data from your data source.

   • Ratio metric expresses the performance as a fraction or proportion, typically by dividing the number of successful events by the total number of potential events (successes + failures). For your Honeycomb SLO, you can configure a single-query ratio to compare good and total events.
     Select the Data count method for your ratio metric.
     
     
     SLI values for good and total

     When choosing the query for the ratio SLI (countMetrics), keep in mind that the values ​​resulting from that query for both good and total:

     • Must be positive.
     • While we recommend using integers, fractions are also acceptable.
     • If using fractions, we recommend them to be larger than 1e-4 = 0.0001.
     • Shouldn't be larger than 1e+20.

7. Specify the attribute for your Single query ratio metric.

   • Attribute refers to a Honeycomb Derived Column name and must be up to 255 characters long. You can retrieve them under the Dataset Settings > Schema tab in Honeycomb.
   Agent version requirements

   Single-query ratio SLOs are supported in the agent version 0.91.0-beta and higher.

7. Define the Time window for your SLO:

   • Rolling time windows constantly move forward as time passes. This type can help track the most recent events.
   • Calendar-aligned time windows are usable for SLOs intended to map to business metrics measured on a calendar-aligned basis.
8. Configure the Error budget calculation method and Objectives:

   • Occurrences method counts good attempts against the count of total attempts.
   • Time Slices method measures how many good minutes were achieved (when a system operates within defined boundaries) during a time window.
   • You can define up to 12 objectives for an SLO.
   
   
   Similar threshold values for objectives
   To use similar threshold values for different objectives in your SLO, we recommend differentiating them by setting varying decimal points for each objective.
   For example, if you want to use threshold value 1 for two objectives, set it to 1.0000001 for the first objective and to 1.0000002 for the second one.
9. Add the Display name, Name, and other settings for your SLO:

   • Name identifies your SLO in Nobl9. After you save the SLO, its name becomes read-only.
     Use only lowercase letters, numbers, and dashes.
   • Select No data anomaly alert to receive notifications when your SLO stops reporting data for a specified period:
     • Choose up to five supported Alert methods.
     • Specify the delay period before Nobl9 sends an alert about the missing data.
       From 5 minutes to 31 days. Default: 15 minutes
   • Add alert policies, labels, and links, if required.
     Limits per SLO: 20 alert policies or links, 30 labels.
10. Click CREATE SLO.


SLO configuration use case
Check the SLO configuration use case for a real-life SLO example.

YAML

Sample Honeycomb SLO

- apiVersion: n9/v1alpha
  kind: SLO
  metadata:
    name: api-server-slo
    displayName: API Server SLO
    project: default
    labels:
      area:
        - latency
        - slow-check
      env:
        - prod
        - dev
      region:
        - us
        - eu
      team:
        - green
        - sales
    annotations:
      area: latency
      env: prod
      region: us
      team: sales
  spec:
    description: Example Honeycomb SLO
    indicator:
      metricSource:
        name: honeycomb
        project: default
        kind: Agent
    budgetingMethod: Occurrences
    objectives:
      - displayName: Good response (200)
        value: 1.0
        name: ok
        target: 0.95
        countMetrics:
          incremental: true
          goodTotal:
            honeycomb:
              attribute: dc.sli.some-service-availability
        primary: true
    service: api-server
    timeWindows:
      - unit: Month
        count: 1
        isRolling: false
        calendar:
          startTime: "2022-12-01 00:00:00"
          timeZone: UTC
    alertPolicies:
      - fast-burn-5x-for-last-10m
    attachments:
      - url: https://docs.nobl9.com
        displayName: Nobl9 Documentation
    anomalyConfig:
      noData:
        alertMethods:
          - name: slack-notification
            project: default
        alertAfter: 1h

Click to open field reference

Field	Type	Description
apiVersion Mandatory	string	API version. Use n9/v1alpha
kind Mandatory	string	The resource type. Use SLO
Metadata
metadata.name Mandatory	string	Name identifier for the SLO. Use only lowercase alphanumeric characters
metadata.displayName	string	User-friendly SLO name
metadata.project Mandatory	string	The name identifier of the project where you need to host your SLO
metadata.labels	object (map: string[])	Grouping labels for filtering or viewing
metadata.annotations	object (map: string)	Flat string annotations
Spec
spec.description	string	SLO description
spec.indicator.metricSource.name Mandatory	string	Data source name
spec.indicator.metricSource.project Mandatory	string	Project containing the data source
spec.indicator.metricSource.kind Mandatory	string	Data source connection method. Can be Agent or Direct
spec.budgetingMethod Mandatory	enum	Error budget calculation method. Can be Occurrences or Time slices
spec.objectives Mandatory	array	Your SLO objective definition, up to 12 objectives per SLO.
spec.objectives[].displayName	string	User-friendly objective name
spec.objectives[].value Mandatory	number	Data point values that is considered "good" (e.g., 200.0). In SLOs with two or more objectives, keep each objective's value unique. In ratio (count) metrics, value is retained for legacy purposes.
spec.objectives[].name Mandatory	string	Name identifier for this objective
spec.objectives[].op Mandatory	string (enum)	Operator for objective. One of: lte (less than or equal to) lt (less than) gte (greater than or equal to) gt (greater than)
spec.objectives[].target Mandatory	float	The percentage of the good minutes or occurrences that must meet the desired performance (e.g., is the target is 0.95, the good performance is expected to be observed in at least 95% of the time window)
spec.objectives[].rawMetric/.countMetric Mandatory	object	The metric type indicator. Set: rawMetric for a threshold metric countMetric for a ratio metric. A ratio metric requires the additional fields: countMetric.incremental (boolean) the data count method countMetric.good/.bad and countMetric.total a numerator and denominator queries
spec.objectives[].countMetric.incremental Mandatory	boolean	The data count method for a ratio (countMetric) metric type
spec.objectives[].primary	boolean	The indicator of a primary SLO objective
spec.service Mandatory	string	The name identifier of a service to host this SLO. The service must exist in the project specified in metadata.project
spec.timeWindows Mandatory	array	Defines SLO time window for error budget calculation. Set: isRolling: true for the rolling time window type isRolling: false for the calendar-aligned type
spec.timeWindows.unit Mandatory	integer	The time window units. One of: Day | Hour | Minute for the rolling time window Year | Quarter | Month | Week | Day for the calendar-aligned time window
spec.timeWindows.count Mandatory	integer	The number of units in a time window
spec.timeWindows.startTime	string	Mandatory for calendar-aligned time windows. Date and time in the format YYYY-MM-DDTHH:mm:ss
spec.timeWindows.timeZone	string	Mandatory for calendar-aligned time-windows. A valid IANA Time Zone Database name
spec.timeWindows.isRolling Mandatory	boolean	true for the rolling time window type false for the calendar-aligned type
spec.alertPolicies	array	The name identifiers of alert policies to be linked to this SLO (must be from the same project as the SLO). Up to 20 alert policies per SLO.
spec.attachments	array	Links to any additional attributes of this SLO
spec.anomalyConfig	object	Settings for a manual no data anomaly detection rule
spec.noData.alertMethods	array	List of alert methods for no-data anomaly. Up to five alert methods per SLO. Every alert method must have the name and project fields
spec.noData.alertAfter	string	Waiting time before sending a no-data notification. Must be 5m to 31d. Default: 15m
spec.noData.treatZeroAsNoData	boolean	Opt-in setting for manual no-data anomaly detection. When set to true, zero-valued points are treated as no data for this alerting flow. Default: false
Source-specific fields
honeycomb.attribute Mandatory	string	A Derived Column name from your Honeycomb dataset schema

Agent version and metric configuration requirements

• Single-query ratio SLOs are supported in the agent version 0.91.0-beta and higher.

• To configure the SLO, ensure you're specifying the metric as follows:

  countMetrics:
    incremental: true | false
    goodTotal:
      honeycomb:
        attribute: your-attribute

  The goodTotal field substitutes individual fields for good and total.

Derived Column names as attribute

Use the Derived Column names from your Honeycomb dataset's schema (Dataset Settings > Schema) as the attribute value.

For example, if your Derived Column is named latency_good, you would use that name for the attribute. Here's the example definition of latency_good:

IF(
    AND(
        IN($service.name, "my-service"),
        EXISTS($duration_ms)
    ),
    IF(LT($duration_ms, 0.45), true, false)
)

• This query marks events as true (good) if the following conditions are met:
  • service.name is my-service
  • The duration_ms field exists
  • The duration_ms value is less than 0.45
• Other events are implicitly marked as false (bad)
• The total count is the sum of good and bad events

Honeycomb metric configuration

By default, Nobl9 queries Honeycomb Data API once per minute. Metrics are configured with the Honeycomb's Query Builder using the attribute (WHERE) clauses.

Limitations

Data source instances

• We recommend configuring no more than one data source per Honeycomb environment to avoid hitting rate limits.

Replay
To prevent potential data loss risks caused by rate limit collisions when approaching the 7-day limit and ensure data integrity:

• Replay durations are capped at 6d 23h 45m (10–15 minutes less than the full seven days).
  This limit relates to the first day of the Replay period.

• Run no more than one Replay at a time for the Honeycomb data source.

SLI per organization
You can add up to 1000 SLIs per Honeycomb environment, using the single-query ratio metric.

SLI per organization limit

The 1000 SLI per environment limit is based on the following:

• 1-minute query interval
• 10 requests per minute rate limit for the Honeycomb query data API

This limit assumes that Nobl9 is the only user of the Honeycomb query data API for SLI calculations. API calls from any other source consume Honeycomb quota and affect the SLI limit.

Useful links

Check out these related guides and references:

Add Honeycomb as a data sourceAdding data sourcesDataset Best PracticesHoneycomb documentationQuery ReferenceHoneycomb documentation