Observability Honeycomb

Honeycomb Analysis

Authentication

IMPORTANT: Credentials are injected automatically by a proxy layer. Do NOT check for HONEYCOMB_API_KEY in environment variables - it won't be visible to you. Just run the scripts directly; authentication is handled transparently.

Configuration environment variables you CAN check (non-secret):

• HONEYCOMB_API_ENDPOINT - Honeycomb API endpoint (default: https://api.honeycomb.io)

---

MANDATORY: Statistics-First Investigation

NEVER dump raw events. Always follow this pattern:

STATISTICS → SAMPLE → PATTERNS → CORRELATE

1. Statistics First - Know volume, error rate, and top patterns before sampling
2. Strategic Sampling - Choose the right strategy based on statistics
3. Pattern Extraction - Cluster similar errors to find root causes
4. Context Correlation - Investigate around anomaly timestamps

Available Scripts

All scripts are in .claude/skills/observability-honeycomb/scripts/

PRIMARY INVESTIGATION SCRIPTS

get_statistics.py - ALWAYS START HERE

Comprehensive statistics with pattern extraction.

python .claude/skills/observability-honeycomb/scripts/get_statistics.py DATASET [--time-range SECONDS] [--filter FILTER]

# Examples:
python .claude/skills/observability-honeycomb/scripts/get_statistics.py production --time-range 3600
python .claude/skills/observability-honeycomb/scripts/get_statistics.py api-requests --filter "http.status_code >= 500"

Output includes:

• Total event count
• Error distribution by status code
• Top services/endpoints
• Top error patterns (crucial for quick triage)
• Actionable recommendation

run_query.py - Custom Queries

Run custom analytics queries with aggregations.

python .claude/skills/observability-honeycomb/scripts/run_query.py DATASET --calc CALCULATION [--breakdown FIELD] [--filter FILTER]

# Calculations: COUNT, SUM, AVG, MAX, MIN, P50, P75, P90, P95, P99, HEATMAP, COUNT_DISTINCT
# Examples:
python .claude/skills/observability-honeycomb/scripts/run_query.py production --calc COUNT
python .claude/skills/observability-honeycomb/scripts/run_query.py production --calc P99 --column duration_ms --breakdown service.name
python .claude/skills/observability-honeycomb/scripts/run_query.py production --calc COUNT --filter "http.status_code >= 500" --breakdown error.message

list_datasets.py - Dataset Discovery

List available datasets in the environment.

python .claude/skills/observability-honeycomb/scripts/list_datasets.py

# Output: List of datasets with names and last write times

---

Honeycomb Query Concepts

Calculations (Aggregations)

| Calculation | Description | Example | |-------------|-------------|---------| | COUNT | Count events | Total requests | | SUM | Sum a column | Total bytes transferred | | AVG | Average value | Average duration | | MAX / MIN | Extremes | Peak latency | | P50, P75, P90, P95, P99 | Percentiles | P99 latency | | HEATMAP | Distribution | Latency heatmap | | COUNT_DISTINCT | Unique values | Unique users | | RATE_AVG, RATE_SUM, RATE_MAX | Rate per second | Requests/second |

Filters

Filters use operators to narrow results:

column = value          # Exact match
column != value         # Not equal
column > value          # Greater than
column >= value         # Greater or equal
column                         │
│    → Know volume, error rate, top patterns                   │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
                     High Error Rate?
               ┌─────────────┴─────────────┐
               │                           │
       YES (>5%)                           NO
               │                           │
               ▼                           ▼
┌─────────────────────────────┐  ┌───────────────────────────────────────────┐
│ 2. FAST PATH                │  │ 2. TARGETED INVESTIGATION                 │
│    Query errors directly    │  │    Filter by specific criteria            │
│    python run_query.py      │  │    python run_query.py dataset            │
│    --filter "error=true"    │  │    --filter "duration_ms > 1000"          │
│    --breakdown error.message│  │    → Look for anomalies                   │
└─────────────────────────────┘  └───────────────────────────────────────────┘

Quick Commands Reference

| Goal | Command | |------|---------| | Start investigation | get_statistics.py | | Count errors | run_query.py --calc COUNT --filter "error=true" | | P99 latency by service | run_query.py --calc P99 --column duration_ms --breakdown service.name | | Error distribution | run_query.py --calc COUNT --filter "error=true" --breakdown error.message | | List datasets | list_datasets.py |

---

SLOs and Triggers

Checking SLOs

python .claude/skills/observability-honeycomb/scripts/run_query.py  --list-slos

Checking Triggers (Alerts)

python .claude/skills/observability-honeycomb/scripts/run_query.py  --list-triggers

---

Anti-Patterns to Avoid

1. NEVER skip statistics - get_statistics.py is MANDATORY first step
2. Unbounded queries - Always specify time ranges (default: 1 hour)
3. Fetching all events - Use aggregations, not raw event dumps
4. Ignoring error rate - High error rate means immediate investigation
5. Missing service filter - For multi-service datasets, always filter by service

Key Differences from Other Platforms

• High cardinality native - Honeycomb excels at high-cardinality fields (user IDs, request IDs)
• No pre-aggregation - Queries run on raw events, enabling ad-hoc exploration
• Trace-first - Designed for distributed tracing, not just logs
• BubbleUp - Use breakdowns to identify anomalous dimensions automatically

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: swapnildahiphale
• Source: swapnildahiphale/OpenSRE
• License: Apache-2.0
• Homepage: https://opensre.in

Install and usage instructions live in the source repository linked above.