Streaming Debugging

Redpanda Streaming: Debugging & Diagnostics

A practical debugging playbook for Redpanda operators. Covers the first-response workflow when something is wrong, how to collect a debug bundle for support, how to read the Prometheus metrics endpoints, and step-by-step triage for the most common failure modes.

For raw HTTP Admin API endpoint detail see the streaming-admin-api skill. For CLI flag depth on rpk debug bundle see the rpk-debug skill.

Quickstart

The first three moves when something is wrong.

1. Check cluster health

# Overall cluster health — are all brokers up? Any leaderless/under-replicated partitions?
rpk cluster health

# List brokers with their current status
rpk cluster info

# Get broker list via Admin API (port 9644)
curl -s http://localhost:9644/v1/brokers | jq

2. Scrape key /public_metrics

# The three most important health signals — pipe through grep to see them fast
curl -s http://localhost:9644/public_metrics \
  | grep -E '(under_replicated|unavailable_partitions|storage_disk_free_space_alert|memory_available_memory\{)'

Expected healthy output:

redpanda_kafka_under_replicated_replicas{...} 0
redpanda_cluster_unavailable_partitions 0
redpanda_storage_disk_free_space_alert 0
redpanda_memory_available_memory{...} 

Non-zero under_replicated_replicas or unavailable_partitions or a disk_free_space_alert value of 1 (low) or 2 (degraded) all require immediate investigation.

3. Collect a debug bundle

# On a Linux broker node — defaults: logs since yesterday, 30s CPU profile, 2 metrics samples
rpk debug bundle

# Narrow the log window for faster collection
rpk debug bundle --logs-since 2024-01-15 --logs-until 2024-01-16

# From a remote machine — collects from ALL brokers configured in your profile
rpk debug remote-bundle start
rpk debug remote-bundle status   # poll until "success"
rpk debug remote-bundle download
# Add --no-confirm for non-interactive/scripted use
rpk debug remote-bundle download --no-confirm

The ZIP lands at ./-bundle.zip (local). Remote download prompts for confirmation by default; pass --no-confirm to skip the prompt in scripts. The default download filename is -remote-bundle.zip.

Unzip and inspect with jq:

unzip 1675440652-bundle.zip -d bundle/
cd bundle/

# Broker versions and maintenance status
cat admin/brokers.json | jq '.[] | {node_id, version, maintenance_status}'

# Cluster health overview
cat admin/health_overview.json | jq

# Check for under-replicated/leaderless partitions
cat admin/health_overview.json | jq '.leaderless_count, .under_replicated_count'

# Disk usage per data directory
cat utils/du.txt

# NTP clock drift
cat utils/ntp.txt | jq .offset

Note: cluster-scoped Admin API files appear once in the bundle (admin/brokers.json, admin/health_overview.json, admin/cluster_config.json, admin/license.json, admin/partition_balancer_status.json). Node-scoped files are suffixed per broker address (e.g., admin/controller_status_.json, admin/partition_leader_table_.json, admin/disk_stat_data_.json, admin/cpu_profile_.json). Metrics snapshots live under metrics//t0_public_metrics.txt, metrics//t0_metrics.txt, etc. (one subdir per broker, N samples).

---

Debug Bundle

rpk debug bundle (local)

Run directly on the broker node. Requires access to the broker host.

# Minimal — uses defaults
rpk debug bundle

# Custom time window and CPU profiler duration
rpk debug bundle \
  --logs-since 2024-01-15 \
  --logs-until 2024-01-16 \
  --cpu-profiler-wait 60s \
  --output /tmp/bundle.zip

# Save additional Admin API data for specific partitions
rpk debug bundle --partition my-topic/0,1,2

# Kubernetes — run inside the Pod, pass the namespace for K8s resource collection
kubectl exec -it -n redpanda redpanda-0 -c redpanda -- \
  rpk debug bundle --namespace redpanda

Key flags (from debugbundle.DebugBundleSharedOptions):

| Flag | Default | Description | |---|---|---| | --logs-since | yesterday | journalctl-format start date (YYYY-MM-DD, 'yesterday', 'today') | | --logs-until | (none) | journalctl-format end date | | --logs-size-limit | 100MiB | Stop reading logs once this size is reached | | --controller-logs-size-limit | 132MB | Max controller log size in bundle | | --cpu-profiler-wait | 30s | Must be > 15s; Seastar samples every ~13s | | --metrics-samples | 2 | Number of metrics snapshots (must be >= 2) | | --metrics-interval | 10s | Interval between metrics snapshots | | --partition / -p | (none) | Extra Admin API requests for {ns}/topic/partition,... | | --namespace / -n | (none) | Kubernetes namespace (K8s only) | | --label-selector / -l | app.kubernetes.io/name=redpanda | K8s label selector | | --kafka-connections-limit | 256 | Max Kafka connections stored | | --output / -o | ./-bundle.zip | Output file path | | --upload-url | (none) | S3-signed URL from Redpanda support for direct upload | | --timeout | 60s | Child command timeout |

rpk debug remote-bundle (cluster-wide)

Drives bundle collection through the Admin API — no node SSH required.

# Create an rpk profile pointing at all admin endpoints first
rpk profile create prod --set admin.hosts=broker1:9644,broker2:9644,broker3:9644

# Start collection on all brokers
rpk debug remote-bundle start

# Skip confirmation prompt (useful in scripts)
rpk debug remote-bundle start --no-confirm

# Wait for completion inline (up to 5 minutes by default)
rpk debug remote-bundle start --wait

# Poll status manually
rpk debug remote-bundle status

# Download when status is "success"
rpk debug remote-bundle download
rpk debug remote-bundle download --output ~/bundles/cluster1

# Cancel a running collection
rpk debug remote-bundle cancel

What the bundle contains

Common (all environments):

• admin/brokers.json — broker list, versions, maintenance status
• admin/cluster_config.json — full cluster config (SASL credentials stripped)
• admin/health_overview.json — cluster health summary
• admin/partition_balancer_status.json — partition balancer status
• admin/cloud_storage_lifecycle.json — Tiered Storage status
• admin/cpu_profile_.json — CPU profiler samples (one per broker)
• admin/license.json — enterprise license info
• kafka.json — metadata, topic/broker configs, offsets, groups
• data-dir.txt — data directory structure (permissions, sizes, mtimes)
• metrics//t0_public_metrics.txt, t0_metrics.txt, ... — Prometheus metric snapshots (N samples per broker)
• crash_reports/ — crash reports from the data directory
• startup_log — startup/crash counter log (top-level bundle file)
• utils/ntp.txt — NTP clock delta and RTT
• utils/du.txt — disk usage per directory

Bare-metal additional:

• redpanda.log — journald logs for the time window
• utils/syslog.txt — kernel ring buffer
• proc/ — /proc files (CPU, memory, filesystems, interrupts)
• utils/ss.txt — active socket info
• utils/top.txt — running process info
• utils/vmstat.txt — virtual memory stats
• utils/ip.txt — network config
• utils/lspci.txt — PCI devices

Kubernetes additional:

• logs/redpanda-N.txt — per-Pod logs
• k8s/ — Kubernetes manifests (configmaps, events, pods, PVCs, services, etc.)

---

Metrics Endpoints

Redpanda exposes two Prometheus-format endpoints on the Admin API (port 9644):

| Endpoint | Purpose | |---|---| | /public_metrics | Stable, supported metrics for dashboards and alerting | | /metrics | Internal Seastar/Redpanda metrics — more verbose, may change between versions |

# Public metrics — recommended for monitoring
curl -s http://localhost:9644/public_metrics

# Internal metrics
curl -s http://localhost:9644/metrics

# See all available public metrics
curl -s http://localhost:9644/public_metrics | grep '^# HELP'

Note: metrics are only exported for features in use. If no consumer groups exist, consumer-group metrics will not appear.

See [Metrics Reference](references/metrics.md) for the full list of key metrics and example Prometheus queries.

---

Triage Playbooks

Quick-reference for the most common failure modes. Each links to the detailed playbook in references.

Under-replicated or leaderless partitions

# Check count
curl -s http://localhost:9644/public_metrics \
  | grep -E '(under_replicated_replicas|unavailable_partitions)'

# Which specific partitions are under-replicated — use /v1/partitions (no namespace)
# The response is partition_summary objects with fields: ns, topic, partition_id, core, materialized, leader
rpk cluster health --watch   # shows counts

# Inspect a specific partition's full state (status, leader_id, replicas)
curl -s http://localhost:9644/v1/partitions/kafka/my-topic/0 | jq

# Or use the debug endpoint for deeper per-replica state
curl -s http://localhost:9644/v1/debug/partition/kafka/my-topic/0 | jq

Disk pressure

# disk_free_space_alert: 0=OK 1=Low 2=Degraded
curl -s http://localhost:9644/public_metrics | grep disk_free_space_alert

# Free bytes
curl -s http://localhost:9644/public_metrics | grep storage_disk_free_bytes

# Disk stat via Admin API
curl -s http://localhost:9644/v1/debug/storage/disk_stat/data | jq

Memory pressure

# Available memory (deducts reclaimable batch-cache memory)
curl -s http://localhost:9644/public_metrics | grep memory_available_memory

# Sampled live memory profile per shard
curl -s "http://localhost:9644/v1/debug/sampled_memory_profile" | jq

Leadership imbalance

# Leadership changes counter — sustained high rate = instability
curl -s http://localhost:9644/public_metrics | grep raft_leadership_changes

# Partition leader table on this node
curl -s http://localhost:9644/v1/debug/partition_leaders_table | jq '.[0:5]'

Raft recovery stuck

# Partitions pending recovery on this broker
curl -s http://localhost:9644/public_metrics | grep -E 'raft_recovery_partitions'

# Controller status (last applied / committed offset)
curl -s http://localhost:9644/v1/debug/controller_status | jq

Slow produce/consume (high latency)

# p99 produce latency (histogram)
curl -s http://localhost:9644/public_metrics \
  | grep 'kafka_request_latency_seconds_bucket{.*produce.*}'

# Consumer group lag
rpk group describe 
curl -s http://localhost:9644/public_metrics | grep consumer_group_lag_max

See [Triage Playbooks](references/triage-playbooks.md) for step-by-step workflows covering each failure mode.

---

CPU Profiling and Self-Test

CPU profiling via Admin API

# Collect 30 seconds of CPU profiler samples (wait_ms in milliseconds)
curl -s "http://localhost:9644/v1/debug/cpu_profile?wait_ms=30000" | jq > cpu_profile.json

# Single shard only
curl -s "http://localhost:9644/v1/debug/cpu_profile?shard=0&wait_ms=30000" | jq

The endpoint path is /v1/debug/cpu_profile (GET). The wait_ms parameter controls how long to collect samples. Seastar samples approximately every 13 seconds so setting wait_ms an enterprise feature is enabled without a valid license rpk cluster license info rpk cluster license info --format json

In a debug bundle: admin/license.json

On license expiration the cluster keeps running without data loss, but you can
no longer enable/modify enterprise features (each degrades differently).

Quick pointers for the features most relevant to broker/cluster debugging:

```bash
# Continuous Data Balancing — status: off|ready|starting|in-progress|stalled
rpk cluster partitions balancer-status
# bundle: admin/partition_balancer_status.json
# Enable/disable via cluster property partition_autobalancing_mode
#   continuous (enterprise default) | node_add (community default) | off

# Tiered Storage health (object storage offload) — watch on /public_metrics:
curl -s http://localhost:9644/public_metrics \
  | grep -E 'cloud_storage_(errors_total|anomalies|segment_readers_delayed|cache_op_(hit|miss))'
# bundle: admin/cloud_storage_lifecycle.json ; master switch cloud_storage_enabled

# Iceberg Topics — DLQ / translation failures:
curl -s http://localhost:9644/public_metrics \
  | grep -E 'iceberg_translation_(dlq_files_created|invalid_records)'
# topic property redpanda.iceberg.mode (key_value|value_schema_id_prefix|value_schema_latest|disabled)

# Shadow Linking (cross-cluster DR) — lag + task health:
rpk shadow list
rpk shadow status 
curl -s http://localhost:9644/public_metrics | grep shadow_link_shadow_lag

See [Enterprise Features](references/enterprise-features.md) for the full set of config keys (including nested partition_autobalancing_*, redpanda.iceberg.*, cloud_storage_*, redpanda.remote.*, audit_*, and *_leaders_preference properties), the health metrics for each feature, what breaks on license expiry, and the disable-for-compliance action per feature.

---

Reference Directory

• [Debug Bundle](references/debug-bundle.md): rpk debug bundle and rpk debug remote-bundle — all flags, what gets collected, generating on Linux vs Kubernetes, and how to inspect the archive with jq.
• [Metrics Reference](references/metrics.md): /public_metrics vs /metrics, the most important health/SLO metrics with Prometheus query examples, and how to enable consumer-group metrics.
• [Triage Playbooks](references/triage-playbooks.md): Step-by-step playbooks for disk pressure, leadership imbalance, under-replicated/leaderless partitions, raft recovery stuck, slow produce/consume, and broker won't start.
• [Profiling and Self-Test](references/profiling-and-selftest.md): CPU profiling via the Admin API (/v1/debug/cpu_profile), disk stat, rpk cluster self-test (disk/network benchmarks), and reading controller/raft recovery status.
• [Enterprise Features](references/enterprise-features.md): Debugging Redpanda's licensed differentiators — license status/violation checks (rpk cluster license info, admin/license.json), Continuous Data Balancing (partition_autobalancing_*, balancer-status) and intra-broker core_balancing_continuous, Tiered Storage (cloud_storage_*, redpanda.remote.*, cache/upload/error metrics), Remote Read Replicas, Cloud Topics (cloud_topics_enabled, redpanda.cloud_topic.enabled), Iceberg Topics (iceberg_*, redpanda.iceberg.*, DLQ + translation metrics), Leader Pinning (default_leaders_preference), Shadow Linking DR (rpk shadow, redpanda_shadow_link_* metrics, task/topic states), and Audit Logging (audit_*). Notes license requirements and disable-for-compliance actions.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: redpanda-data
• Source: redpanda-data/skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.