Deep Debug

Deep Debug Skill

Adversarial hypothesis-driven debugging. Given a bug, generate competing hypotheses across orthogonal dimensions, judge each independently (two-pass blind), run discriminating probes that falsify the weaker, fix the survivor with test-first discipline, and escalate to architectural review after 3 failed attempts. Output is a verified fix with a failing-test-now-passes proof, OR an honest termination label naming why the bug resisted a code-level fix.

Execution Model

All operations use Claude Code primitives. These contracts are non-negotiable:

• All data passed to agents via files, never inline. Evidence, hypothesis lists, probe specs, known-hypothesis IDs — all written to disk before the agent prompt. Inline data is silently truncated.
• State written before agent spawn, not after. spawn_time_iso is written to state.json before the Agent tool call. Spawn failure records spawn_failed status. Resume uses persisted state, never in-memory reconstruction.
• Structured output is the contract; free-text is ignored. Every judge, probe-verdict, and validator produces machine-parseable structured lines. Coordinator reads only structured fields. Unparseable output triggers fail-safe classification (disputed, never rejected). Hypothesis output files MUST contain STRUCTURED_OUTPUT_START/STRUCTURED_OUTPUT_END markers; files without these markers are treated as failed.
• No coordinator self-review of anything load-bearing. Hypothesis plausibility, evidence tier, rebuttal verdicts, probe winner classification — all delegated to independent agents. The coordinator orchestrates; it does not evaluate.
• Termination labels are honest. One of 7 defined labels. Never "probably fixed," never "looks right," never "no critical bugs remain." A fix is Fixed — reproducing test now passes only when the failing test exists, passes with the change, and the full suite is clean.
• Hard ceilings are absolute. max_cycles = 3, max_probes_per_cycle = 3, fix_attempt_count ≤ 3. Three fix attempts is evidence the hypothesis space is wrong — Phase 7 architectural escalation is mandatory, never optional.

Shared contracts: this skill inherits the four execution-model contracts (files-not-inline, state-before-agent-spawn, structured-output, independence-invariant) from [_shared/execution-model-contracts.md](../_shared/execution-model-contracts.md). The items listed above are the skill-specific elaborations; the shared file is authoritative for the base contracts.

Pressure awareness: this skill applies the pressure circuit breakers from [_shared/pressure-awareness.md](../_shared/pressure-awareness.md). After 3 probes with no hypothesis falsification, regenerate the hypothesis space rather than probing a 4th time. The existing max_cycles = 3 and fix_attempt_count ≤ 3 ceilings serve as the same-approach ceiling; this contract adds the diminishing-returns check between probe cycles.

Cross-finding coherence: this skill applies the coherence-integrator pattern from [_shared/cross-finding-coherence.md](../_shared/cross-finding-coherence.md) at Phase 3, after hypothesis agents complete and BEFORE the judge classifies plausibility. The integrator reads all hypothesis files simultaneously and annotates each hypothesis with cross-hypothesis relationships (contradictions indicating the same root cause viewed from different angles, emergent patterns suggesting a higher-level architectural issue, coverage gaps in the 8-dimension space). These annotations are included in judge input files.

Subagent watchdog: every run_in_background=true spawn (hypothesis agents, judges, evidence-gatherer, architect) MUST be armed with a staleness monitor per [_shared/subagent-watchdog.md](../_shared/subagent-watchdog.md). Use Flavor A with thresholds STALE=5 min, HUNG=20 min for Sonnet hypothesis agents; STALE=3 min, HUNG=10 min for Haiku judges and evidence-gatherer. Debugging agents that hang silently are the exact failure mode this skill is meant to prevent — applying it to the skill itself is load-bearing. Contract inheritance: timed_out_heartbeat joins this skill's 7-label termination vocabulary at the per-lane level (hypothesis agent / judge / probe); stalled_watchdog and hung_killed join hypotheses.{id}.status. A watchdog-killed hypothesis lane never returns leading or rejected — its verdict is absent, not assessed.

Philosophy

Debugging fails most often in three ways:

1. Fixation on the symptom site — the stack trace points at file:line, the fix goes there, the real cause is 5 calls upstream. Solo debuggers rarely trace data-flow all the way back.
2. Hypothesis anchoring — you form the first plausible hypothesis, evidence confirms it (because you're looking for confirmation), you fix it, the bug returns.
3. Fix-and-retry spiral — 3 failed attempts is the signal the abstraction is wrong, but under time pressure debuggers try fix #4, #5, #6, each one revealing a new problem in a different location.

Deep-debug addresses all three structurally:

• Orthogonal dimensions force at least one hypothesis in concurrency, environment, and architecture categories — the three most commonly under-investigated
• Independent judge with disconfirmation mandate prevents anchoring on the first fit
• Hard architectural escalation at 3 failed fixes converts fix-spiral into a conscious pattern-level decision

If you only take one idea from this skill: evidence-for and evidence-against are the hypothesis contract. A hypothesis without disconfirming searches is never leading; it's plausible at best. See EVIDENCE.md.

The Iron Law

NO FIXES WITHOUT A HYPOTHESIS THAT SURVIVES INDEPENDENT JUDGE + DISCRIMINATING PROBE

If Phase 3 judge has not classified the hypothesis as leading AND Phase 4 probe has not confirmed it against the next-best alternative, you cannot advance to Phase 5. Violating the letter of this process is violating the spirit of debugging.

When to Use

Use deep-debug for:

• Bugs that survived a quick-fix attempt (systematic-debugging already ran and the symptom remains or moved)
• Production incidents requiring post-mortem rigor
• Test failures that cannot be reproduced locally
• Performance regressions where the cause is ambiguous
• Build / deploy / signing / infrastructure failures with multi-layer pipelines
• Behavior divergence between environments (local ✓, CI ✗, or one deployment region only)
• Any bug where ≥ 2 plausible root causes exist and the coordinator is tempted to guess

Don't use for:

• Obvious typos, compiler errors with clear messages, or known-issue-with-known-fix cases → use superpowers:systematic-debugging
• Test flakiness where the target is "is this test flaky, and why specifically?" → use flaky-test-diagnoser (it's purpose-built)
• Bugs the user already has a hypothesis for and just wants implemented → implement directly; save deep-debug for when the cause is genuinely unknown
• Pure code review or static audit → use deep-qa with --type code

Workflow

Phase -1: Context-Aware Composition Routing

Before the debug cycle, detect if a specialized sub-skill applies. Invoke it via Skill() — deep-debug is the orchestrator, not a bypass.

| Context detected | Sub-skill to invoke | How to detect | |---|---|---| | GitHub PR URL or "CI failed on PR" | debug-pr | URL matches github.*/pull/\d+ or user mentions PR + CI failure | | Metaflow pathspec or "flow failed" | debug-run | Input contains FlowName/RunID pattern or "metaflow" + failure language | | Jenkins build URL or "build failed" | debug-jenkins | URL matches a Jenkins host or "jenkins" + failure language |

If a sub-skill matches: invoke it first to gather structured context (CI logs, run status, build output). Feed its output into Phase 0 as the symptom — then continue with deep-debug's hypothesis-driven cycle on the root cause. The sub-skill handles evidence collection; deep-debug handles root-cause analysis.

If no sub-skill matches: proceed directly to Phase 0.

Phase 0: Input Validation Gate

Before any cycle begins, validate the symptom. Batch clarifying questions — if multiple surface here, present them in one message as a numbered list. Never serially.

Step 0a — Safety check:

• Symptom describes a bug in the user's code, not a request to attack a third-party system
• If harmful: refuse

Step 0b — Scope check:

• Is the symptom specific enough to investigate? ("the app is slow" is too vague; "POST /api/orders p99 latency jumped from 200ms to 2s after {commit}" is fine)
• If too vague: ask for: exact error (copy-paste the message), reproduction steps (commands that trigger it), affected environment (local/CI/prod), and when it started (timestamp or "since merge X")

Step 0c — Reproduction confirmation:

• Attempt to reproduce immediately (run the provided command; trigger the flow)
• Record status in state: confirmed (every time), intermittent (1-in-N with rate), or unreproducible (cannot trigger)
• unreproducible → user batch-question: can they provide fresh logs, a stack trace, or a less-filtered reproduction?
• Still unreproducible after user input → terminate with label Cannot reproduce — investigation blocked and exit cleanly. Do NOT hypothesize against a bug you cannot see.

Step 0d — Symptom locking:

• Extract a ≤200-word symptom statement — the exact error, the surface, when it started, the reproduction command
• Compute symptom_sha256; store in state.json
• This statement is the concept-anchor — every hypothesis is a causal explanation of this symptom. Later phases MUST NOT silently rewrite the symptom to match an emerging hypothesis. symptom_sha256 catches tampering.

Step 0e — Pre-mortem micro-round (blind-spot seeding): Before Phase 2 expansion, spawn 1 Haiku agent with this prompt:

> Given the symptom {symptom}, list 5 concrete ways this debugging investigation could miss the real cause. Cover these angles: > 1. Wrong dimension — the symptom looks like {apparent dimension} but is actually {other dimension} > 2. Measurement artifact — the bug might not exist; we could be reading stale/wrong evidence > 3. Environment assumption — local behavior assumed, production differs > 4. Framework-contract blindness — something assumed guaranteed that actually isn't > 5. Architectural drift — this is symptom N of an architectural problem; fixing here won't hold > Output to deep-debug-{run_id}/premortem.md with one concrete claim per angle.

Each flagged blind-spot becomes an angle with priority: critical and source: premortem in Phase 2.

Step 0f — Pre-run scope declaration:

Deep debug: "{symptom}"
Reproduction: {confirmed | intermittent (1-in-N) | unreproducible-investigation-blocked}
Dimensions: 8 (code-path, data-flow, recent-changes, environment, framework-contract, concurrency-timing, measurement-artifact, architectural-coupling)
Max cycles: {default 3}    | Hard stop: {6} cycles    | Budget cap: ${25.00}
Estimated cost: ~$3-8 per cycle (Sonnet hypothesis + Haiku judge + Haiku probes)
Invocation: {interactive | --auto | --hard-mode}
Continue? [y/N]

Skip gate if --auto.

Print: Starting deep-debug on: {symptom_excerpt} [run: {run_id}]

---

Phase 1: Evidence Gathering

Purpose: Collect the ground-truth artifacts every hypothesis in Phase 2 will be evaluated against. Evidence is tier-ranked per EVIDENCE.md.

Step 1a — Symptom capture (always):

• Exact error text / stack trace / log excerpt — copy verbatim into deep-debug-{run_id}/evidence.md §Symptom
• Timestamp of first occurrence (if known)
• Reproduction command (from Phase 0c)

Step 1b — Recent changes snapshot (always):

• git log --since='7 days ago' --oneline -- {affected-paths} → evidence.md §Recent Changes
• Lock file diffs (package-lock.json / poetry.lock / Gemfile.lock) vs. last-known-good, if known
• Config diffs
• Even if this looks unrelated to the symptom, record it. recent-changes is one of the 8 dimensions — agents will search for the connection.

Step 1c — Boundary instrumentation (when multi-layer):

• If the system has obvious layer boundaries (workflow → build → sign; API → service → DB; frontend → gateway → backend): add instrumentation at each layer to log inputs/outputs
• Run the reproduction once with instrumentation; capture all layer logs into evidence.md §Boundary Instrumentation
• This converts tier-5 stack-position guesses into tier-2 primary-artifact evidence for Phase 2
• Skip if single-layer or instrumentation would take > 30 minutes (add to Phase 4 probe list instead)

Step 1d — Lock evidence:

• evidence.md is append-only after Phase 1. Phase 4 probe results append to §Probe Log; nothing rewrites earlier sections.

---

Phase 2: Hypothesis Generation

Prospective gate (fires BEFORE spawning):

Coordinator outputs:

Cycle {N}: about to spawn 6 hypothesis agents + 1 outside-frame agent (Sonnet tier).
Estimated cost: ~${estimate} (cumulative: ${running_total} of ${budget_cap})
Dimensions queued: {list of angles' dimensions}
Required categories covered so far: {counts from state.json}
Continue? [y/N/redirect:]

Skip if --auto. If --auto AND cumulative budget exceeds 80% of cap: force user confirmation even in auto mode (budget backstop).

Step 2a — Enumerate angles (see DIMENSIONS.md):

For cycle 1:

• Generate 2–4 angles per dimension (8 dimensions × 2–4 = 16–32 angles)
• Generate 2–3 cross-dimensional angles
• Add premortem angles from Phase 0e
• Cap total frontier at 30. Priority-order per DIMENSIONS.md §Priority Ordering.

For cycle 2+:

• Start from graveyard (hypotheses rejected in prior cycle) — what dimensions were under-investigated?
• Add CRITICAL-priority angles for any dimension that had zero hypotheses accepted by the judge
• Add CRITICAL-priority angles for any required category still uncovered

Step 2b — Write data files before spawning:

• deep-debug-{run_id}/known-hypotheses.md — list of all prior-cycle hypothesis IDs and 1-line titles (so critics don't re-propose)
• deep-debug-{run_id}/angles/{angle_id}.md — one file per angle with dimension, question, parent, rationale
• deep-debug-{run_id}/evidence.md is referenced by path; critics read it directly

Step 2c — Spawn hypothesis agents (parallel):

• Pop up to 6 highest-priority angles; assign to spec-derived hypothesis agents
• Spawn 1 additional outside-frame hypothesis agent (slot #7) — seeded from the symptom only, not the current hypothesis list or evidence file narrative
• Before each spawn: write status: "in_progress", spawn_time_iso to state.json
• Spawn each Agent in parallel (subagent_type: general-purpose, model: sonnet) with file paths — NOT inline content
• Hypothesis output path: deep-debug-{run_id}/hypotheses/{angle_id}.md (content-addressed — coordinator CANNOT overwrite)
• If Agent tool returns error: record status: "spawn_failed", do not record as spawned; retry on next round

Quorum: Round complete if ≥ 4 of 6 spec-derived agents return parseable output within timeout (outside-frame tracked separately — its exit does not affect quorum).

Timeout scaling: 180s base. ×1.5 for cycles 2+ (larger evidence file).

Circuit breaker: ≥ 3 consecutive rounds with any critic failures → halt, log SYSTEM_FAILURE_ROUND, notify at turn boundary.

Step 2d — After agents complete:

• For each completed agent: read hypothesis file, extract hypothesis struct, dedup (see STATE.md Tier 1 + Tier 2), update state.json
• Spec-derived agents may suggest ≤ 1 new angle per cycle (immutable once written)
• Outside-frame agent may suggest ≤ 2 new angles (its exemption — it brings novel dimensions)
• Update coverage tracking: which dimensions / required categories are now covered?

---

Phase 3: Independent Hypothesis Judge (two-pass blind)

Purpose: Classify ever

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: npow
• Source: npow/claude-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.