Quality Playbook

Quality Playbook Generator

Plan Overview — read this first, then explain it to the user

Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information.

Phase 0 (Prior Run Analysis): If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re-runs.

Phase 1 (Explore): Run the documentation intake first. Invocation form: bin/reference_docs_ingest.py does from bin import benchmark_lib at import, so Python's -m bin.reference_docs_ingest resolves ONLY when bin/ is reachable as a top-level package. Resolve the install root (the directory containing this SKILL.md) via the same install-location fallback list this skill uses for references/ / quality_gate.py (the 10 canonical layouts documented below — install-root forms: `, .claude/skills/quality-playbook, .github/skills, .github/skills/quality-playbook, .cursor/skills/quality-playbook, .continue/skills/quality-playbook, .codex/skills/quality-playbook, .windsurf/skills/quality-playbook, .cline/skills/quality-playbook, .aider/skills/quality-playbook). Then, from the target repo root, run PYTHONPATH= python3 -m bin.referencedocsingest . If is empty (QPB self-bootstrap, or a setuprepos.sh install where bin/ is already at the target root) omit the PYTHONPATH= prefix — cwd already provides bin/. The bare python -m bin.referencedocsingest works ONLY for those two layouts; an installskill.py-layout adopter (bin/ inside the marker dir) otherwise gets ModuleNotFoundError: No module named 'bin'. This walks referencedocs/ — cite/ files produce quality/formaldocsmanifest.json records; top-level files are loaded as Tier 4 context via referencedocsingest.loadtier4_context(). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain-knowledge risk analysis, and selected structured exploration patterns. Write all findings to quality/EXPLORATION.md`. This file is the foundation — Phase 2 reads it as its primary input.

Phase 2 (Generate): Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol. (AGENTS.md at the target's repo root is generated by the orchestrator AFTER Phase 6, not by you in Phase 2 — see "File 6" below for the contract.)

Phase 3 (Code Review): Run the three-pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches.

Phase 4 (Spec Audit): Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the Layer-2 semantic citation check — one prompt per reviewer, structured per-REQ verdicts for every Tier 1/2 citation, output to quality/citation_semantic_check.json. Write regression tests for net-new findings.

Phase 5 (Reconciliation): Close the loop — every bug from code review and spec audit is tracked, regression-tested or explicitly exempted. Run TDD red-green cycle. Finalize the completeness report.

Phase 6 (Verify): Run self-check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence.

Phase 7 (Present, Explore, Improve): Present results to the user with a scannable summary table, offer drill-down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system.

Every bug found traces back to a requirement, and every requirement traces back to an exploration finding.

The critical dependency chain: Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost.

MANDATORY FIRST ACTION: After reading and understanding the plan above, print the following attribution banner exactly as written below — do NOT condense, abbreviate, summarize, reformat, or add a version number. Reproduce the full 8-line canonical block including the two ===…=== rules, both taglines, the blank lines, and the license line, byte-for-byte matching bin/_purpose.print_attribution_banner() (the single source of truth). Then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open-ended domain-driven investigation, followed by domain-knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding.

================================================================================
  Quality Playbook -- by Andrew Stellman
  https://github.com/andrewstellman/quality-playbook

  AI code review is good. Quality engineering is better.
  Because code that looks right can still do the wrong thing.

  Licensed under the Apache License, Version 2.0
================================================================================

Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.

How to run this — self-encoded invocation contract

If the operator hands you this skill (or points you at any QPB-installed target) and says "Run the Quality Playbook" — possibly with a hint like "this is a bootstrap run" or "run on itself" or "self-audit" — this section tells you exactly what to do. The operator should not need to provide additional instructions; the canonical invocation, the defaults, the guardrails, and the output contract all live here.

Pick your execution mode

QPB ships in two execution shapes. Pick the one that matches your runtime — the wrong choice produces the codex-on-codex indirection pathology.

| Mode | When this is you | What you do | |------|------------------|-------------| | A. Skill-direct (UI-context) | You are a coding agent (Claude Code, Cursor, Copilot, Codex desktop, etc.) handed this skill in your own chat. Your runtime IS the reasoning loop — you read files, you write files, you decide. | Walk through Phase 1 → Phase 6 yourself using the externalized phase prompts in phase_prompts/. Write artifacts into the target's quality/ directory directly. No subprocess, no runner. | | B. Runner-driven (CLI-automation) | The operator is invoking python3 -m bin.run_playbook deliberately — to batch across multiple targets, drive a headless CI run, or fan out per-phase work to a different model than the one reading this prose. | The orchestrator spawns a CLI agent (claude, copilot, codex, or cursor) per phase. You (or whoever is reading this) are the operator-side control loop, not the per-phase reasoner. |

Both modes use the same phase prompt content — the phase_prompts/*.md files at the repo root are the single source of truth, loaded by bin/run_playbook.py::_load_phase_prompt and read directly by Mode A walkthroughs. The only thing the two modes differ on is WHO drives — you (Mode A) or the orchestrator subprocess-spawning a CLI agent (Mode B).

When in doubt, default to Mode A. If the operator wanted runner-driven invocation they would have run the runner themselves; if they pasted "Run the Quality Playbook" into your chat, they want you to drive. The Mode B section below tells you what to do if the operator explicitly invokes the runner.

Documented Mode A vs Mode B asymmetries — by design. Three behaviors deliberately differ between the modes because the modes serve different purposes (Mode A is the adopter-facing walkthrough on a target repo; Mode B is the QPB-internal runner used for benchmarking + headless CI). The phase prompts are identical (single source of truth — see paragraph above); the divergences are in the driver obligations, not the audited contract:

1. Phase 0 entry contract — install validator (Mode A only). Mode A's mandatory first action is to run the install validator. Invocation form is install-location-aware: resolve the install root (the directory containing this SKILL.md) via the same install-location fallback list this skill uses for references/ / phase_prompts/ (the 10 canonical layouts documented below), then run python3 /bin/qpb_validate.py . For a clone / self-bootstrap (install root is the QPB clone, bin/ reachable from cwd) the bare python3 bin/qpb_validate.py works. For an install_skill-layout adopter (channel or manual install — bin/ inside the marker dir, e.g. .github/skills/quality-playbook/bin/) you MUST use the install-root-prefixed form; running bare bin/qpb_validate.py from the target repo root fails (the validator lives under the marker dir, not at repo root). The adopter has just installed QPB into their target repo and must confirm canonical layout before any phase. Mode B (the QPB clone running its own runner) operates from the canonical install by construction; the validator would always succeed against the QPB clone itself, so the check would be vacuous. Adopters running Mode A get the install correctness gate; Mode B operators have it already.
2. End-of-run archive (Mode B only). Mode B's _finalize_iteration calls archive_lib.archive_run after Phase 6, materializing quality/previous_runs// for benchmark replay + cross-run diffing. Mode A produces artifacts directly into the target's live quality/ for the adopter to inspect, commit, or re-run on top of; there is no archive partition because there is no benchmark loop. Adopters who want longitudinal archives invoke Mode B explicitly (python3 -m bin.run_playbook --full-run ).
3. Phase 6 auditor-prompt sub-agent dispatch (Mode A only). phase_prompts/phase6.md routes Mode A through phase_prompts/phase6_auditor.md via a fresh-context sub-agent spawn (the A-13 hybrid). Mode B's per-phase CLI subprocess is already a fresh context (the structural separation the auditor exists to provide), so phase6.md:5-7 tells Mode B to execute the verification inline and skip the auditor spawn. phase_prompts/phase6_auditor.md is therefore only loaded under Mode A; the file's header documents this routing explicitly.

These asymmetries are pinned by bin/tests/test_mode_a_b_parity_documented.py. If a future change wants to align them, update the pin test, the rationale here, and add the missing behavior to the appropriate side.

Mode A — skill-direct walkthrough (UI-context)

The operator's prompt is just "Run the Quality Playbook" (or "run on itself", "self-audit", etc.). You drive every phase inline.

Mode A means YOU execute the skill. If you are an interactive coding agent (Claude Code, Cursor, Copilot UI, Codex desktop, any tool where the operator is monitoring your chat output in real-time), DO NOT:

• Spawn a sub-skill via your quality-playbook skill-invocation mechanism.
• Spawn a sub-agent via your Task tool / Agent tool / similar.
• Invoke python3 -m bin.run_playbook — that's Mode B, the CLI-automation path.
• Read agents/quality-playbook.agent.md or agents/quality-playbook-claude.agent.md and follow their "spawn sub-agents per phase" pattern — those files are AUTOMATION ONLY (batch CI runs, headless invocations) where per-phase context-window isolation is necessary at the cost of operator chat visibility.

Phase 0 entry contract (MANDATORY first action): run the QPB install validator. Resolve the install root via the install-location fallback list (per the Phase 0 numbered step above), then run python3 /bin/qpb_validate.py — bare python3 bin/qpb_validate.py works only when cwd already contains bin/ (QPB clone / self-bootstrap); install_skill-layout adopters (channel install — bin/ under .github/skills/quality-playbook/, etc.) MUST prefix the install root. Paste every event= line; do not proceed until event=validation_complete status=ok — see AGENTS.md "Mode A entry sequence". Non-negotiable — skipped/fabricated validation and bare-path-from-repo-root invocation are both documented adopter failure modes.

EXCEPTION: Phase 6 verification MUST be delegated to a fresh-context sub-agent. Sub-agent delegation is forbidden for EXECUTION (Phases 1-5) because hiding work from operator chat defeats real-time monitoring. Sub-agent delegation is MANDATED for VERIFICATION (Phase 6) because shared-context executor-as-verifier has empirically fabricated PASS verdicts against failing gates across multiple benchmark runs despite repeated rounds of strengthened prompt-level witness language. The principle is the same as the rule above — operator chat should carry the truth — but the mechanism is opposite: visibility for execution; isolation for verification (a fresh auditor has none of the executor's memory/anchoring/completion-reward bias). See phase_prompts/phase6.md + phase_prompts/phase6_auditor.md for the sub-agent delegation protocol. This exception is scoped strictly to Phase 6 verification — the no-sub-agent rule still holds for all Phase 1-5 execution.

The failure mode that motivated this rule: a shared-context session ran phases 1-6, fabricated the gate-verdict log (quality/results/quality-gate.log was hand-written with a "PASS" tally while the actual gate showed 14 FAIL, GATE FAILED), and reported PASS to the parent. The Phase 6 witness contract (your obligation to quote the gate's verdict lines verbatim) cannot enforce when a sub-skill owns the witness — only the parent's chat carries operator-readable evidence.

YOU read SKILL.md and the phase prompts. YOU execute Phases 1-5 yourself and write artifacts to quality/. Phase 6 verification is delegated to the fresh-context auditor sub-agent per the exception above (the auditor invokes quality_gate.py); you paste its verbatim verdict. Your chat IS the trust trail — execution stays visible, verification stays bias-isolated.

You have shell access — probe before claiming unavailability. Mode A means an interactive coding agent (Claude Code, Cursor, Copilot UI) running commands on the operator's actual machine via your Bash tool — NOT a sandboxed runtime. A common failure is writing "the sandbox can't run X" reflexively, conflating your model-level "I can't natively execute" with the harness Bash tool you DO have. A which / --version probe costs one line; assuming-and-being-wrong forfeits the entire Phase 5 evidence trail.

For each phase 1..6, in order:

1. Load the phase prompt. Read phase_prompts/phaseN.md (resolve via the same install-location fallback list documented for references/ below). For phase1.md, substitute {seed_instruction} (the prelude that says "skip Phase 0/0b" — empty string when seeds are allowed) and {role_taxonomy} (the taxonomy block rendered from the role taxonomy below). For phase2.md through phase6.md, the file is pure-literal — read it verbatim.
2. Execute the phase per the prompt. Read the inputs the prompt names, do the analysis, write outputs into the target's quality/ directory.
3. STOP at the end-of-phase boundary. Every phase prompt ends with an "IMPORTANT: Do NOT proceed to Phase N+1" instruction. Honor it. The operator advances to the next phase by saying so.

You are responsible — without the orchestrator's structural backstop — for the same source-unchanged invariant the runner enforces:

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: andrewstellman
• Source: andrewstellman/quality-playbook
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.