Diagnosing Bugs

Diagnosing Bugs

Use the full diagnosis loop for difficult or uncertain bugs and performance regressions. For a straightforward failure, use the relevant steps and proportionate verification.

When exploring the codebase, read CONTEXT.md (if it exists) to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.

Phase 1 — Build a feedback loop

Build a pass/fail signal that detects the reported symptom. Read relevant source and configuration as needed to construct it; distinguish observed facts from an untested theory. Choose reproduction effort in proportion to the uncertainty and cost.

Ways to construct one — try them in roughly this order

1. Failing test at whatever seam reaches the bug — unit, integration, e2e.
2. Curl / HTTP script against a running dev server.
3. CLI invocation with a fixture input, diffing stdout against a known-good snapshot.
4. Headless browser script (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
5. Replay a captured trace. Save a real network request / payload / event log to disk; replay it through the code path in isolation.
6. Throwaway harness. Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
7. Property / fuzz loop. If the bug is "sometimes wrong output", use seeded inputs and a bounded run sized to the observed failure rate.
8. Bisection harness. If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can git bisect run it.
9. Differential loop. Run the same input through old-version vs new-version (or two configs) and diff outputs.
10. HITL bash script. Last resort. If a human must click, drive them with scripts/hitl-loop.template.sh so the loop is still structured. Captured output feeds back to you.

Use the feedback loop to distinguish causes and verify the fix.

Tighten the loop

Treat the loop as a product. Once you have a loop, tighten it:

• Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
• Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
• Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)

Prefer a fast, deterministic loop, but use slower or statistical evidence when the system requires it.

Non-deterministic bugs

Measure the reproduction rate, then use a bounded experiment suited to it. Repetition, controlled stress, or timing changes may help distinguish causes. Record run counts and failure rates; do not treat a small passing sample as proof of a fix.

When you genuinely cannot build a loop

After reasonable bounded attempts, report what you tried and what remains unverifiable. Continue useful source analysis with conclusions clearly labeled as static evidence or hypotheses. Request a sanitized captured artifact or authorized non-production reproduction only when it would resolve a specific gap. Do not seek production access or add production instrumentation as a default debugging step.

Completion criterion — a tight loop that goes red

When a runnable repro is feasible, record its command and result. The target is a loop that is red-capable: you can name one command — a script path, a test invocation, a curl — that you have already run at least once (paste the invocation and its output), and that is:

• [ ] Red-capable — it drives the actual bug code path and asserts the user's exact symptom, so it can go red on this bug and green once fixed. Not "runs without erroring" — it must be able to catch this specific bug.
• [ ] Repeatable — same verdict for deterministic bugs; for flaky bugs, record controlled conditions and measured failure rates.
• [ ] Fast enough for useful iteration; avoid unnecessary setup.
• [ ] Agent-runnable where feasible — use scripts/hitl-loop.template.sh when a structured human-assisted loop is needed.

Source reading can guide reproduction and hypotheses. Do not present an untested theory as a reproduced cause; use the documented fallback above when execution is unavailable.

Phase 2 — Reproduce + minimise

Run the loop. Watch it go red — the bug appears.

Confirm:

• [ ] The loop produces the failure mode the user described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
• [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
• [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.

Minimise

Once it's red, simplify the repro enough to isolate plausible causes without spending disproportionate effort on perfect minimization. Cut inputs, callers, config, data, and steps one at a time, re-running the loop after each cut — keep only what's load-bearing for the failure.

Why bother: a minimal repro shrinks the hypothesis space in Phase 3 (fewer moving parts left to suspect) and becomes the clean regression test in Phase 5.

Stop reducing when further cuts are unlikely to improve the diagnosis. Preserve the original scenario for final verification.

Phase 3 — Hypothesise

Rank the plausible causes supported by the evidence. Consider alternatives when uncertainty warrants it; do not invent hypotheses to meet a quota. Choose the smallest experiment that distinguishes the leading explanations.

Each hypothesis must be falsifiable: state the prediction it makes.

> Format: "If is the cause, then will make the bug disappear / will make it worse."

If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.

Show the ranked list to the user before testing. They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if the user is AFK.

Phase 4 — Instrument

Each probe must map to a specific prediction from Phase 3. Change one variable at a time.

Tool preference:

1. Debugger / REPL inspection if the env supports it. One breakpoint beats ten logs.
2. Targeted logs at the boundaries that distinguish hypotheses.
3. Never "log everything and grep".

Tag every debug log with a unique prefix, e.g. [DEBUG-a4f2]. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.

Perf branch. For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, performance.now(), profiler, query plan), then bisect. Measure first, fix second.

Phase 5 — Fix + regression test

Write the regression test before the fix — but only if there is a correct seam for it.

A correct seam is one where the test exercises the real bug pattern as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.

If no correct seam exists, that itself is the finding. Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.

If a correct seam exists:

1. Turn the minimised repro into a failing test at that seam.
2. Watch it fail.
3. Apply the fix.
4. Watch it pass.
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.

Phase 6 — Cleanup + post-mortem

Required before declaring done:

• [ ] Original repro no longer reproduces (re-run the Phase 1 loop), or execution limits and remaining uncertainty are reported
• [ ] Regression test passes (or absence of seam is documented)
• [ ] All [DEBUG-...] instrumentation removed (grep the prefix)
• [ ] Throwaway prototypes deleted (or moved to a clearly-marked debug location)
• [ ] The hypothesis that turned out correct is stated in the commit / PR message — so the next debugger learns

Then ask: what would have prevented this bug? If the answer involves architectural change (no good test seam, tangled callers, hidden coupling) hand off to the /improve-codebase-architecture skill with the specifics. Make the recommendation after the fix is in, not before — you have more information now than when you started.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: waffleflopper
• Source: waffleflopper/ai-tools
• License: MIT

Install and usage instructions live in the source repository linked above.