Tooling Strategy

Tooling Strategy

This skill produces the build-plan strategy for Q2, "how do we know if what we have is good?"

The other strategies depend on Q2 being answerable but don't plan how to make it so. /quality-strategy judges the actual state (Q3) using whatever oracles exist — an oracle is anything that can judge whether an output is good — and honestly records where they don't: as Unknowns whose to-resolve notes say nothing can judge them yet. /evaluation-strategy works out, per ility, what better judging and indication would look like — oracles and labelled proxies — and agrees build-shaped moves. /test-strategy plans the investigation and records which questions are blocked on missing instruments — things that exercise or observe the system — or oracles. All of them name gaps; none plans the build. That's this skill's job: collect every "we can't actually answer that yet" from the lanes and turn the pile into one prioritised, sequenced oracle and instrument build plan.

Building oracles is routinely the highest-value quality work an early-stage project can do. A weak oracle gives false confidence; a missing one leaves the most important dimensions permanently Unknown. Yet this work rarely gets planned, because each gap surfaces in a different place and none looks urgent alone. Gather them in one place, weigh each build by what it unblocks, and the priorities are usually obvious.

The demand arrives as requirements — "we must be able to judge X at this fidelity". This skill takes each requirement the rest of the way: into a spec, a source decision (build it, adopt something open-source, buy it, or extend what exists), and a place in a sequenced plan. Each piece of tooling is a mini-product. Its stakeholders are fixed and known — the team, the agents who will run it, the strategies that consume its verdicts — so it needs none of the interview machinery; but the same four questions apply in miniature, and each item needs real answers to them (see step 2).

What this skill is not

It is not an audit, and it does not re-audit. The per-item adequacy verdicts come from the two Q2 audit skills the lanes offer when trust is contested: /oracle-adequacy (from /evaluation-strategy, on the oracles behind the risk map's actual-state claims) and /tooling-adequacy (from /test-strategy, on the means of answering its questions); both also run standalone. This skill consumes their outputs and the lanes' agreed moves. If an input's adequacy is contested and unaudited, send the user to the audit rather than improvising adequacy judgments here.

Resolving file paths — do this first

This skill is part of the quality-strategy plugin. Before anything else, resolve two absolute paths and use them throughout:

• PLUGIN_ROOT — the plugin's install directory: ${CLAUDE_PLUGIN_ROOT} (Claude Code expands this to an absolute path when it loads this file; read it off and note it down). The grounding files this skill reads — PHILOSOPHY.md, skills/tooling-adequacy/SKILL.md — live under it, as does .claude-plugin/plugin.json, whose version field stamps the generated tooling strategy (see "Output").
• PROJECT_DIR — the absolute path of the project whose tooling you're planning (normally the current working directory; confirm with the user if it's ambiguous). The strategy docs normally live under $PROJECT_DIR/quality/ — but /quality-strategy asks at session start where the strategy should be saved, so they may live elsewhere. If $PROJECT_DIR/quality/ is absent, get the docs home instead of assuming: from the orchestrator's brief when this skill was dispatched, else by asking the user; if the path you're given ends in /quality, its parent is the home. From then on treat $PROJECT_DIR as that docs home wherever a path below says $PROJECT_DIR/quality/... — one substitution, made once, before you act on any path.

File references below use the $PLUGIN_ROOT and $PROJECT_DIR placeholders. Substitute the resolved absolute paths before you act on them. The Read tool does no variable expansion and resolves relative paths against the current working directory, not this skill's directory — so an unsubstituted placeholder or a bare relative path will fail.

When to use

• After the lanes — or straight after /quality-strategy when the map is blind. This skill has two natural entry points, and the risk map — the strategy doc's table of quality dimensions, their actual state, and the confidence behind each claim — decides which (Q2 before Q3: you can only investigate what you can judge). When the risk map came out dominated by Unknowns nothing can judge yet, run /evaluation-strategy then this skill immediately — plan the builds that make the project knowable before planning the investigation. A test strategy written against dimensions nobody can judge is mostly "blocked". When the risk map is mostly answerable, let /test-strategy run first and sharpen the demand (its /tooling-adequacy audit surfaces blocked questions), then run this skill on the combined pile.
• When deciding what test or measurement infrastructure to build next — this skill's output is that decision, made with the full demand visible instead of ad hoc.
• Re-run freely. This is the most re-runnable doc in the pack: re-run when build items land (they change what's answerable — see Update protocol in the output), when a lane adds new demand, or when the quality strategy is revised. On any re-run over an existing quality/tooling-strategy.md, archive it first — snapshot to quality/archive/tooling-strategy-.md (suffix -2, -3, … if the name is taken — never overwrite an archive) before changing a word, and mention the archive in the closing summary. Never silently rewrite history: the archive leaves a before/after trail the user can compare and share.

What you need

• Grounding. Read $PLUGIN_ROOT/PHILOSOPHY.md. The disciplines that recur — make confidence visible; record assumptions; push back on vagueness — are load-bearing here. Read step 3 of $PLUGIN_ROOT/skills/tooling-adequacy/SKILL.md for the canonical oracle taxonomy (Specified / Property-or-metamorphic / Differential-or-simulated / Golden-master / Human-or-agent-judge) and the "kill the old-world reflex" move; every build item in the plan names its kind from that taxonomy.
• The quality side. From $PROJECT_DIR/quality/strategy.md: Part 6 (the risk map — Unknowns with their to-resolve notes, above all the ones saying "nothing can judge this yet", and the confidence ratings) and Part 5 (the H/M impact ratings, which weight the value of each build). Part 7 for any build-shaped actions already enumerated (Part 7 may be a recorded deferral — fine; the risk map carries everything this skill needs).
• The evaluation side, if it exists. From $PROJECT_DIR/quality/evaluation-strategy.md (or a prior-era quality/oracle-strategy.md — the lane was renamed): the agreed next moves that are build-shaped, with any /oracle-adequacy verdicts behind them.
• The test side, if it exists. From $PROJECT_DIR/quality/test-strategy.md: agreed moves recorded as blocked on missing instruments or oracles, with their build items. If a lane hasn't run, proceed on what exists and say so in the output — don't guess at demand that hasn't been derived.
• The user. The value calls below are theirs to confirm, and they usually know pains neither strategy captured — the flaky suite nobody trusts, the metric nobody believes. Ask.

If quality/strategy.md doesn't exist, stop: this skill plans the build for gaps the other strategies have found, and has nothing to consume yet. Point the user at /quality-strategy first (or at /oracle-adequacy standalone, if they have an existing strategy doc from elsewhere whose actuals' oracles were never audited).

Session start — itinerary and commit cadence

At the start of every working session (first or resumed), before the next move's work, two quick moves:

• Give the itinerary, in plain words. Five moves, by human name, with rough sizes: "Gather everything the project can't answer about itself yet (quick — mostly collection); spec each build — what it must do, what it unblocks, build-or-adopt (the bulk of the work); weigh value against cost (short, mostly your calls to confirm); sequence it into phases (short); and close the loop — what each build triggers when it lands." On resumption or a re-run, lead with what's done or changed since last time, then the remainder.
• Ask the commit-cadence question (when the docs home is git-managed). First read quality/.scratch/session-config.md — the strategy runs usually already settled the cadence for this docs family; restate it in one line instead of re-asking. Where the note is missing and the folder holding the quality/ docs is git-managed, ask once: "Want me to commit at each move's boundary, commit everything at the end, or leave the commits to you?" Suggest commit-as-we-go as the default (cheap rollback, visible progress); where the docs sit in a repo other people share, say in the same breath that each boundary commit publishes the draft to everyone who pulls. Honour the answer at every boundary and record it in session-config.md.

The work, in order

Write the doc incrementally as each move completes, so what's done is durable; on re-entry, resume from the last completed move. At each move's boundary, give one progress line (where we are, what remains, relative sizes) and the visible exit — what the user already has if they stop here (the demand table alone is a useful artifact, and the doc picks up where it left off). At a boundary it's safe to /clear; say how to resume in the same breath, so resuming is stated, never guessed — "safe to /clear; to pick up, run /tooling-strategy again and it reads quality/tooling-strategy.md (and the strategies it draws from) and continues from the last completed move." The user should never feel the work is unbounded, or have to gamble that re-running the skill works. This is also where the boundary commit lands if the user chose commit-as-we-go.

1. Gather the demand

Collect every place the project currently can't answer "how good is it?":

• From the risk map: every judge-blocked Unknown — its to-resolve note says nothing can judge it yet, or names building the means of judging. An Unknown whose to-resolve note names an action instead — ask a stakeholder, run targeted testing, review a specific area — is not judge-blocked; it routes to the lane that owns that action (stakeholder work back through /quality-strategy, or testing/review through /test-strategy) rather than landing here. This mirrors the same you can only investigate what you can judge handoff /test-strategy's filter states: this skill gathers the judging gaps, not every open Unknown.
• From the oracle strategy (if present): every build-shaped agreed move, with any /oracle-adequacy verdicts (an Over-confident actual — a claim whose oracle can't support it — or a Gated Unknown) behind it.
• From the test strategy (if present): every agreed move blocked on missing instruments or oracles, with its build item.
• From the user: "Anything else you already know you can't measure or judge, that neither doc captured?" Record these honestly as user-reported, not strategy-derived.

Deduplicate across the sides. The same underlying build often appears more than once — the risk map can't establish an actual for the same reason a test-lane question is blocked. Merge them into one item that names every beneficiary — being needed on several sides is exactly what makes it valuable.

2. Spec each item — every tool is a mini-product

The gathered demand is requirements; this step turns each into a spec and a decision. Per item:

• What it must do — instrument (exercise/observe) or oracle (judge), the kind from the taxonomy, and the question(s) it must answer at the fidelity they demand. "Property set: intervals never shrink on a correct recall", not "better testing for the scheduler."
• What it unblocks — the dimension(s) and question(s), by name. An item that unblocks nothing doesn't belong in the plan.
• What "good" means for this tool — one line. The universal dealbreaker is false confidence: a tool that passes wrong things is worse than no tool, because the Unknown was at least visible. Good enough = answers its question at the required fidelity, and cheap enough to actually get run.
• Source: build / adopt / buy / extend — a concrete decision per item, not a default. The asymmetry to expect: instruments often already exist (fault injectors, load generators, fuzzers, coverage, monitors — adopting usually beats building); oracles more often need building, because they encode this project's definition of correct. "Extend" — bolting the missing oracle onto an existing harness — is frequently the cheapest real option. One line of reasoning either way.
• Calibration — the recursive Q2 — how we'll know the tool itself judges correctly. Cheap standard moves: seed known-bad cases and confirm it catches them; diff it against the reference implementation; human-verify its first real verdicts. A tool with no calibration plan is a false-confidence machine waiting to happen.
• Who does the work — agent-buildable now (most reference oracles, property sets, harnesses are), needs human judgment or access (rubrics where a person is the oracle, production telemetry, real-user observation), or a pairing of the two.

3. Weigh value and cost

Propose a value/cost reading per item and confirm it with the user:

• Value comes from what it unblocks: the impact rating of the dimensions (H beats M), the priority of the moves it unblocks, and breadth — one harness that unblocks five questions beats five one-shot builds. Converting an Unknown on one of the risk map's hottest rows is the single highest-value move available.
• Cost under agent economics: most oracle construction is now agent-cheap — don't price it at human cost. Flag the genuinely expensive items (infrastructure, production access, recruiting humans for judgment sessions) rather than letting them silently sink the plan.
• Trustworthiness of the build itself: would the built oracle actually be able to judge the question? A cheap oracle that can't is worse than no oracle — it converts a visible Unknown into invisible false confidence. If you doubt an item can work, say so on the item; building a small probe of it first can be the Phase-1 step.

4. Sequence into a build plan

Order the items into phases, with the reasoning stated:

• Quick wins first — agent-cheap, high-unlock items (typically property sets and simulated/reference oracles on hot dimensions).
• Dependencies — an oracle that judges observations needs its instrument to exist first; shared infrastructure that several items sit on comes before them.
• Group shared work — items that ride the same harness or the same telemetry build belong in one phase.
• Defer with reasons — low-value or speculative items are listed as deferred with a one-line reason, not silently dropped.

5. Close the loop

For each planned item, state what changes when it lands: which risk-map rows to re-assess (a /quality-strategy revision-mode pass over the affected rows), and which blocked moves come unblocked (recorded in the lane's next update). The build plan only pays off because it turns Unknowns into questions we can answer — name that conversion per item, so landing a build triggers the re-assessment instead of leaving the new capability unused.

Then write the Update protocol section: re-run this skill when a phase of builds lands, or when either parent strategy is revised.

Push back when

• Every item is an instrument and none is an oracle (or vice versa). "Build

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: tollens-ai
• Source: tollens-ai/quality-strategy-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.