Goat Qa

/goat-qa

Shared Conventions

Read .goat-flow/skill-docs/skill-preamble.md before starting. On full-depth, also read .goat-flow/skill-docs/skill-conventions.md.

When to Use

goat-qa is a testing gap analyser: it maps changed code or a codebase area to coverage and outputs prioritized must/should/skip guidance. It does not write tests or run full test commands.

Invoke when:

• Feature branch is ready for testing and you want to know what to focus on
• QA has a test plan and you want to verify it covers the actual code changes
• You're reviewing a PR and want to know what the tests miss
• You want to find manual testing gaps before a release
• You need a QA handoff artifact (flow diagram, risk matrix, manual test plan)

NOT this skill: Run-test requests → run them directly. Test failures or fix verification → /goat-debug. Code quality → /goat-review. Milestones → /goat-plan. Feature briefs → dispatcher. Merge certification → /goat-review plus Proof Gate.

| Excuse | Reality | |--------|---------| | "CI is green so coverage is fine" | Scanner scored 100% while preflight failed with 8 errors. CI tests what was thought of; gap analysis looks for what wasn't. | | "Unit tests cover it" | Structural tests that import and snapshot pass at high coverage but miss every behavioural edge. STRUCTURAL is not BEHAVIOURAL. | | "Coverage report says 80%" | Coverage measures shape, not truth. 20+ content-accuracy failures survived a structural pass that reported high coverage. | | "Doer ran the tests, so we're covered" | Doer-verifier is theater in single-agent context. The verifier must have a context boundary the doer did not cross. |

Coverage Depth

Canonical coverage vocabulary used in Standard, Audit, and cross-skill output.

| Level | Meaning | |-------|---------| | NONE | No matching test file or manual plan | | STRUCTURAL | Imports, constructs, or snapshots only - no behaviour assertion | | PARTIAL-BEHAVIOURAL | Happy path or narrow behaviour only; error/edge paths untested | | BEHAVIOURAL | Meaningful output, side-effect, error-path, or invariant coverage |

Step 0 - Intake

Mode detection - confirm, don't silently decide:

• Changed files + no specific ask → offer standard or audit
• "audit"/"coverage"/"gaps" → Audit mode (full depth)
• "verify coverage"/"what's risky"/"what should I test" or scoped files → Standard mode (quick depth)

Depth mapping: Standard = quick changed-file analysis. Audit = full codebase-area analysis. Dispatcher depth maps quick → Standard, full → Audit.

If mode and scope are clear, state "Running [mode] on [scope]." and proceed. Ask only on ambiguity.

Gather: changed scope, existing test plan (if any), audience. Check the instruction file's Essential Commands section or package.json scripts for test/lint commands.

Footgun check: Use the preamble's learning-loop retrieval on .goat-flow/learning-loop/footguns/, .goat-flow/learning-loop/lessons/, .goat-flow/learning-loop/patterns/, and .goat-flow/learning-loop/decisions/ for the target area. Surface matches or an explicit retrieval miss; do not broad-load any bucket.

PR / issue link (strongly encouraged): ask for PR/issue before Phase 1. Acceptance criteria are the benchmark. If gh is available, use gh pr view + gh pr diff; otherwise note no-intent-spec, which degrades safe to skip confidence.

If arriving from the dispatcher with context already gathered, confirm and proceed.

No existing tests: risk analysis still applies. Mark coverage NONE and state: "This project has no automated tests. Verification falls to human and AI reviewers."

CHECKPOINT: "Analysing [N] changed files against [existing test plan / no test plan]. Audience: [dev/tester/both]." Proceed unless scope, audience, or test plan is ambiguous.

Phase 1 - Change Risk Analysis

Read every changed file. For each, understand WHAT changed and WHY it's risky.

Diff analysis - not just file names. Read the actual diff, not just --stat; one auth line can outrank 200 CSS lines.

Classify each change:

| Risk | What it means | Examples | |------|-------------|---------| | CRITICAL | If this breaks, users are directly affected or security is compromised | Auth logic, payment flow, data mutation, permission checks, API contracts | | HIGH | Business logic or integration that affects correctness | Calculations, state transitions, cross-service calls, database queries | | MEDIUM | Internal logic with limited blast radius | Utilities, validators, formatters, isolated components | | LOW | Cosmetic, config, or changes with no behavioural impact | Styling, copy, constants, type-only changes |

For each CRITICAL/HIGH change, trace the blast radius:

• What depends on this code? (callers, consumers, downstream services)
• What user-visible flows pass through this code path?
• Has this area broken before? (check footguns/lessons)

Output: Change Risk Map

| File | Lines Changed | What Changed (plain English) | Risk | Blast Radius | User-Visible Impact | |------|-------------|---------------------------|------|-------------|-------------------|

CHECKPOINT: "Risk map complete. [N] CRITICAL, [M] HIGH risk changes. Proceeding to gap analysis."

Phase 2 - Gap Analysis

Compare risk vs coverage in both directions:

• If a test plan exists, map cases to CRITICAL/HIGH changes and check reverse coverage.
• If no plan exists, map changed files to automated tests and flag explicit behavior gaps.
• For each changed file, read the matched test file (if any) and classify using Coverage Depth. If tests are unavailable, record tests not read in Verification Integrity.
• Classify gaps as:
• Undertested risk
• Misaligned effort

For CRITICAL items with no coverage, annotate why: new path / missed coverage on existing path / hard-to-test.

Intent vs Reality Diff (when intent spec exists): If a PR, issue, test plan, or user-provided acceptance criteria is available, add:

| Expected Behaviour | Observed Code Behaviour | Gap | Risk |

Map each stated expectation to the code path that implements it. Gaps between intent and code are undertested-risk candidates.

Cross-agent verification: suggest a different agent/model for blind-spot checks.

BLOCKING GATE (auto-released on explicit test-plan intent): Present gap analysis plus Verification Integrity, then stop and ask "Continue to Phase 3, or adjust first?" - unless the invocation already gave explicit "what should I test" / "test plan" intent, in which case treat it as a CHECKPOINT and continue through Phase 3 without pausing. Reserve diagrams for Phase 3; then suggest /goat-plan.

Worked Standard example: Diff touches a terminal launch helper. Read that diff and the smoke tests covering prompt injection plus delayed/fallback delivery. Expected row: HIGH risk runner launch contract; BEHAVIOURAL coverage for prompt injection and delayed/fallback delivery; safe to skip more PTY timing tests unless timer constants changed; proof class STATIC unless executed.

Phase 3 - Targeted Testing Plan

Based on the gaps, produce a focused plan and order by risk.

Must test (CRITICAL gaps): table with what breaks and grounded effort estimate; if effort is unknown, write unknown - needs harness/project context Should test if time allows (MEDIUM gaps): same format, lower priority Safe to skip this round: low-risk or adequately covered areas Misaligned effort: deprioritise plan cases not mapped to current changes

CHECKPOINT: "Targeted testing plan ready. Want a flow diagram for any CRITICAL item?"

Phase 4 - Flow Diagram

For flow diagrams, use Mermaid flowcharts with 8-15 nodes per diagram, happy path first, then branch points for error states and edge cases.

---

Audit Mode

For a codebase area with no recent change. Audit mode analyses existing load-bearing files, coverage depth, and structural-vs-behavioural gaps. It does NOT read a diff; skip Phase 1.

A1 - Scope

Declare the audit boundary explicitly. Supported shapes:

• A directory (e.g. src/payments/) - every source file inside.
• A module (e.g. src/reporting/) - the module's entry point and direct callees.
• A risk class (e.g. "everything touching auth tokens") - files you would need to read to verify the claim.

If unsure, ask the user before A1.5.

A1.5 - Scope-Size Gate

Inventory approximate file count before deep analysis. If too large, present a ranked slice prioritising load-bearing and interface-boundary files. Proceed to A2 only after manageable scope is confirmed.

A2 - Inventory and Risk Ranking

Without any diff, classify each in-scope file by its role, not its recency:

| Role | Examples | |------|----------| | Load-bearing | auth, payments, permission checks, data mutation, migration | | Interface boundary | API routes, CLI commands, public exports | | Integration glue | config loaders, filesystem bridges, external clients | | UI / presentation | views, templates, styling | | Support | types, constants, pure helpers |

Load-bearing + Interface files get CRITICAL or HIGH risk ratings by default.

A3 - Coverage Analysis

For each in-scope file:

1. Does a test file exist? If not → coverage NONE.
2. If yes, read the test. Does it assert behaviour (outputs, side effects, error paths) or only construct the unit?
3. Flag mock-heavy tests (everything mocked = behaviour untested) and integration-only blind spots (suite skips when the external service is unavailable).

Record coverage using the Coverage Depth vocabulary above.

A4 - Gap Report

Rank gaps by Risk × (1 - CoverageLevel) descending - Risk maps CRITICAL=4, HIGH=3, MEDIUM=2, LOW=1; CoverageLevel is a coverage fraction (NONE=0, STRUCTURAL=0.34, PARTIAL-BEHAVIOURAL=0.67, BEHAVIOURAL=1.0), so (1 - CoverageLevel) is the uncovered fraction and a CRITICAL+NONE file ranks top (4.0). Output:

• Blocking gaps - CRITICAL-risk file with NONE or STRUCTURAL coverage. One line per file: missing behaviour + the test the user should add.
• High-value additions - HIGH-risk file with PARTIAL coverage. Describe the untested path.
• Defer - LOW-risk or already well-covered files. Name them explicitly so the user sees what was considered and why.

Worked Audit example: Scope a small audit module; read tests, not filenames - the heuristic misleads both ways. An orchestrator can lack a same-name test yet run behaviourally through an integration suite, so it is PARTIAL-BEHAVIOURAL, not NONE. A content-integrity helper with no unit, integration, or exported-symbol references is genuinely NONE. Expected A4 blocking gap: that content-integrity check, CRITICAL by role, NONE coverage - add a test planting a wrong count and asserting it is flagged. Proof class STATIC.

BLOCKING GATE: Present gap report; wait for human decision before generating a testing plan response. Create no plan file unless separately approved.

Regression Guard Mode

Use after a fix was already verified and the user asks how to keep it from regressing.

1. Cite the prior fix-verification source.
2. Define 1-2 human-readable invariants.
3. Compare each invariant to existing tests/manual coverage.
4. Output only the Regression Guards table and Verification Integrity.

This mode does NOT verify the fix itself.

Constraints

• goat-qa is a testing GAP ANALYSER - it finds mismatches between code (changed or existing) and testing coverage
• MUST compare in-scope code against existing testing coverage (manual plan, automated tests, or neither)
• MUST find gaps in BOTH directions: undertested risks AND misaligned test effort
• MUST produce "must test / should test / safe to skip" tiers with rationale for skips
• MUST include Verification Integrity section
• MUST apply the Proof Gate from skill-preamble.md to every claim made in the gap analysis or testing plan
• MUST tag every finding/claim row with proof class RUNTIME | CONTRACT-GREP | STATIC | NOT-REPRODUCED
• MUST NOT generate test code - hand off to the coding agent
• Universal constraints from skill-preamble.md apply; per-mode MUSTs live in the phase bodies (Phase 1 diff/risk/blast-radius; Audit A2/A4), not restated here.
• If flow diagrams are requested, use Mermaid flowcharts (8-15 nodes, happy path first, annotate gap status per node).
• Regression guard: MUST state invariants as human-readable sentences; MUST cite prior fix-verification source; MUST NOT verify the fix itself
• MUST defend zero-gap results explicitly: state what was checked and why no gaps surfaced. Zero gaps without justification is an error condition, not a clean bill.

Output Format

Output shape depends on the mode declared in Step 0. Pick the template that matches the mode you ran.

Standard mode - Phase 2 output (diff-driven, present at BLOCKING GATE)

## TL;DR  

## Change Risk Map
| File | Lines Changed | What Changed | Risk | Blast Radius | User-Visible Impact | Proof Class |

## Gap Analysis
### Undertested Risks  
| Code Change | Risk | Coverage Depth | Covered By | Gap | Proof Class |

### Misaligned Effort  
| Test Case | Maps to Change | Assessment | Proof Class |

## Verification Integrity
- Intent spec: [PR/issue/test plan URL or `no-intent-spec`]
- Tests read: [list]
- Tests not read / unavailable: [list or `none`]
- Commands discovered: [test/lint commands found]
- Commands run: `none` (goat-qa does not execute tests)
- Runtime execution by others: [who ran what, or `none observed`]
- Coverage claim basis: [OBSERVED | INFERRED | UNVERIFIED]
- Proof classes:  RUNTIME /  CONTRACT-GREP /  STATIC /  NOT-REPRODUCED
- Analysis confidence: [HIGH | MEDIUM | LOW] - [rationale]
- Evidence limit: [diff/files read and any unavailable runtime/tool context]
- Assessed by: [agent]

Standard mode - Phase 3 output (generate only after Phase 2 gate approval)

## Targeted Testing Plan
### Must test before shipping  
### Should test if time allows  
### Safe to skip  

## Verification Integrity

- Changes by: [agent/developer]
- Testing by: [who executes]
- Doer-verifier separation: [FULL / PARTIAL / NONE]

## Regression Guards  
| Invariant | Current Coverage | Recommended Guard | Owner | Proof Class |
## Flow Diagram  

Audit mode (no diff - A1–A4 shape)

## TL;DR  

## Scope

## Inventory and Risk Ranking
| File | Role | Risk | Proof Class |

## Coverage Analysis
| File | Test file | Coverage | Notes | Proof Class |

## Gap Report
### Blocking gaps  
### High-value additions  
### Defer  

## Verification Integrity
- Intent spec: [audit scope rationale or `no-intent-spec`]
- Tests read: [list]
- Tests not read / unavailable: [list or `none`]
- Commands discovered: [test/lint commands found]
- Commands run: `none` (goat-qa does not execute tests)
- Coverage claim basis: [OBSERVED | INFERRED | UNVERIFIED]
- Proof classes:  RUNTIME /  CONTRACT-GREP /  STATIC /  NOT-REPRODUCED
- Analysis confidence: [HIGH | MEDIUM | LOW] - [rationale]
- Assessed by: [agent]
- Would-be testers: [who executes once gaps are filled]

## Flow Diagram  

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: blundergoat
• Source: blundergoat/goat-flow
• License: MIT
• Homepage: https://goat-flow.com

Install and usage instructions live in the source repository linked above.