Test Auto Fix

test-auto-fix

Autonomously diagnose and fix failing tests in any project, with hard guardrails so the loop never devolves into "make the red go away".

This SKILL.md is the orchestration index. Load the matching rule file when you need detail — do not preload them.

| Phase | Goal | Required rule | | ----- | ---- | ------------- | | 0 | Resolve the surface (bootstrap or validate) | [rules/bootstrap.md](./rules/bootstrap.md) + [rules/surface-validation.md](./rules/surface-validation.md) | | 1 | Detect which test surface(s) are failing; build fix plan | this file + [templates/plan-artifact.md](./templates/plan-artifact.md) | | 2 | Per failure: classify as test-bug / prod-bug / unsure | [rules/verdicts.md](./rules/verdicts.md) + [rules/self-improvement-loop.md](./rules/self-improvement-loop.md) (read lessons) | | 3 | Per failure: draft the smallest possible fix | this file | | 3.5 | Confidence gate — before any edit | [rules/confidence-gate.md](./rules/confidence-gate.md) | | 4 | Apply + verify single failing test | this file + [rules/anti-patterns.md](./rules/anti-patterns.md) | | 5 | test-provenance-guard (optional companion) | invoke Skill("test-provenance-guard") | | 6 | Outer loop: re-run full surface; regression-detect | [rules/regression-detection.md](./rules/regression-detection.md) + [rules/self-improvement-loop.md](./rules/self-improvement-loop.md) (write lessons) | | 7 | Report (structured exit summary) | [templates/exit-summary.md](./templates/exit-summary.md) + [rules/self-improvement-loop.md](./rules/self-improvement-loop.md) (write on outcome) |

Always read [rules/anti-patterns.md](./rules/anti-patterns.md) first. The hard refusals apply to every phase.

Input

The user provides one of:

• Nothing — auto-run every surface defined in the project surface file.
• A surface name (vitest, unit, integration, etc.) — run only that surface.
• A file path — narrow detection and re-run to that file.
• --surface — override the surface file lookup entirely.
• --plan-only — diagnose and produce the plan artifact; do not write code.
• --max-iterations N — outer-loop cap (default 5).

The argument is: $ARGUMENTS.

Phase 0 — Resolve the surface

Determine the surface file for the current project. Phase 0 is a hard two-branch decision; pick exactly one path and follow it through.

Step 1 — Pick the resolution branch

• If --surface was passed → branch A.
• Otherwise → branch B.

Branch A — --surface override

Use the file at ` directly as the surface. **Do NOT compute the project key, do NOT scan surfaces/`, do NOT run bootstrap.**

Still validate per [rules/surface-validation.md](./rules/surface-validation.md), but tolerate the project-key mismatch warning (it's expected when overriding).

When validation passes, jump straight to Phase 1.

Branch B — Auto-resolve from project key

1. Compute the project key per [rules/project-keying.md](./rules/project-keying.md).

1. Look for surfaces/.md next to this skill file.

The skill's own directory is resolved by following the symlink chain of the loaded SKILL.md: ``bash readlink -f "$(dirname "$0")" ` If that fails (model-invocation context), default to ~/.agents/skills/test-auto-fix/surfaces/`.

1. If no surface file is found → run bootstrap per

[rules/bootstrap.md](./rules/bootstrap.md). Bootstrap detects the stack, proposes a surface diff, waits for user approval, then writes the file. Do NOT proceed to Phase 1 until the surface exists.

1. If a surface file exists → validate it per

[rules/surface-validation.md](./rules/surface-validation.md). If validation fails, propose an update diff and ask once. If the user declines the update, escalate — do not run with an invalid surface.

Phase 1 — Detect failures and build the plan

1. Run the detect command from the surface file:

```bash

`` If --path ` was supplied, append it to narrow the run. If the cache-bust-flag is set in the surface and a prior run was cached-green, append the flag to force a real run.

1. Parse failures using the failure-parser regex from the surface file.

Group by file, then by failure type.

1. If every surface is green: stop immediately. Tell the user. Do not invent work.

1. Write the fix plan to .agent/{branch}/test-auto-fix-plan.md using

[templates/plan-artifact.md](./templates/plan-artifact.md). The plan is read-only documentation of intent. Order: high-confidence test-bug fixes first, prod-bug suspects last.

Phase 2 — Classify each failure (verdict required)

For every failure, apply the rubric in [rules/verdicts.md](./rules/verdicts.md).

Getting this wrong means later phases can corrupt production code to satisfy a broken test, or "fix" a real regression by patching the test.

Emit exactly one verdict per failure:

• test-bug — production code correct; test drifted. Continue to Phase 3.
• prod-bug — production code regressed; test caught it. Escalate. Do not touch the test.
• unsure — confidence ; verdict: ; surface: ; risk: ")

Record the score in the plan artifact.

Full decision matrix: [`rules/confidence-gate.md`](./rules/confidence-gate.md).

At a glance:

| Score | Action |
| ----- | ------ |
| ≥ 90 | Auto-apply. Continue to Phase 4. |
| 80–89 | Show the diff, ask once, apply on approval. |
| 
   ```

3. If still failing: do not patch over. Revert with:
   ```bash
   git restore 
   ```
   Re-classify at Phase 2.

4. If green, and `test-provenance-guard` is installed: invoke
   `Skill("test-provenance-guard")` on the file. If it reports
   tests-by-construction, revert and escalate.

5. Only after both checks pass, mark the failure resolved in the plan.

## Phase 5 — test-provenance-guard (optional)

If `Skill("test-provenance-guard")` is not installed, log:
`companion: test-provenance-guard — not available, continuing`

## Phase 6 — Outer loop

After a batch of fixes:

1. Re-run the full surface using the detect command:
   ```bash
   
   ```
   Regressions hide in untouched files — always re-run the full surface.

2. Compare against the previous failure set per
   [`rules/regression-detection.md`](./rules/regression-detection.md):
   - Same failures → back to Phase 2 (re-classify; do not repeat same fix).
   - Strict subset → continue with remaining failures.
   - New failure → revert last fix, re-plan or escalate.

3. Cap at `--max-iterations` (default 5).

4. On any exit, write a summary section to the plan file.

## Phase 7 — Report

Always end with the structured exit summary from
[`templates/exit-summary.md`](./templates/exit-summary.md).

```text
test-auto-fix run
  Outcome: 
  Resolved:  failures
  Escalated:  failures ()
  Iterations: /
  Surface: 
  Plan: .agent/{branch}/test-auto-fix-plan.md
  Escalation reason:            # if not green

Self-Improvement

/test-auto-fix gets better across runs through a two-tier lessons loop (fast episodic tier + gated promotion), like autonomous-workflow and fix-bug. It reads test-auto-fix-lessons at Phase 2 (biasing the verdict and the Phase 3 fix sub-class) and writes at Phase 6 (regression / same-failure signal) and Phase 7 (outcome retrospective), keyed by stack : failure-pattern : verdict-sub-class. This complements the surface file (which is config — how to run tests here — not learned judgment) rather than duplicating it. Lessons are advisory — they never lower the confidence gate, never turn a prod-bug/unsure escalation into a silent test edit, and never override a refusal in [rules/anti-patterns.md](./rules/anti-patterns.md).

Most value is within a project (catching a recurring verdict misclassification for a failure shape); cross-project leverage is weaker because the feedback is binary and local. A recurring lesson (seen_count >= 3) is promotion-eligible via /create-skill diagnose test-auto-fix. persistent-memory is optional; the loop skips silently if absent. Full contract: [rules/self-improvement-loop.md](./rules/self-improvement-loop.md).

Definition of done

The run is done when ANY of the following is true:

• All surface tests are green AND the structured exit summary has been printed.
• All remaining failures are prod-bug or unsure (escalated; user owns next step).
• The confidence gate scored < 80 and the fix was not written.
• A regression was detected, reverted, and the user owns the next step.
• --max-iterations (default 5) was reached.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mthines
• Source: mthines/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.