# Tooling Strategy > Produce the strategy for the second quality question — "how do we know if what we have is good?" Gathers every place the strategy lanes could NOT answer that (risk-map Unknowns nothing can judge yet; build items agreed in /evaluation-strategy; /test-strategy moves blocked on instruments or oracles) and turns them into a prioritised oracle/instrument build plan at quality/tooling-strategy.md. Use… - **Type:** Skill - **Install:** `agentstack add skill-tollens-ai-quality-strategy-skills-tooling-strategy` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [tollens-ai](https://agentstack.voostack.com/s/tollens-ai) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [tollens-ai](https://github.com/tollens-ai) - **Source:** https://github.com/tollens-ai/quality-strategy-skills/tree/main/skills/tooling-strategy ## Install ```sh agentstack add skill-tollens-ai-quality-strategy-skills-tooling-strategy ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Tooling Strategy This skill produces the build-plan strategy for **Q2, "how do we know if what we have is good?"** The other strategies *depend* on Q2 being answerable but don't plan how to make it so. `/quality-strategy` judges the actual state (Q3) using whatever **oracles** exist — an oracle is anything that can judge whether an output is good — and honestly records where they don't: as Unknowns whose to-resolve notes say nothing can judge them yet. `/evaluation-strategy` works out, per ility, what better judging and indication would look like — oracles and labelled proxies — and agrees build-shaped moves. `/test-strategy` plans the investigation and records which questions are blocked on missing **instruments** — things that exercise or observe the system — or oracles. All of them *name* gaps; none plans the build. That's this skill's job: collect every "we can't actually answer that yet" from the lanes and turn the pile into one prioritised, sequenced **oracle and instrument build plan**. Building oracles is routinely the highest-value quality work an early-stage project can do. A weak oracle gives false confidence; a missing one leaves the most important dimensions permanently Unknown. Yet this work rarely gets planned, because each gap surfaces in a different place and none looks urgent alone. Gather them in one place, weigh each build by what it unblocks, and the priorities are usually obvious. The demand arrives as **requirements** — "we must be able to judge X at this fidelity". This skill takes each requirement the rest of the way: into a spec, a source decision (build it, adopt something open-source, buy it, or extend what exists), and a place in a sequenced plan. Each piece of tooling is a mini-product. Its stakeholders are fixed and known — the team, the agents who will run it, the strategies that consume its verdicts — so it needs none of the interview machinery; but the same four questions apply in miniature, and each item needs real answers to them (see step 2). ## What this skill is not It is **not an audit, and it does not re-audit.** The per-item adequacy verdicts come from the two Q2 audit skills the lanes offer when trust is contested: `/oracle-adequacy` (from `/evaluation-strategy`, on the oracles behind the risk map's actual-state claims) and `/tooling-adequacy` (from `/test-strategy`, on the means of answering its questions); both also run standalone. This skill *consumes* their outputs and the lanes' agreed moves. If an input's adequacy is contested and unaudited, send the user to the audit rather than improvising adequacy judgments here. ## Resolving file paths — do this first This skill is part of the `quality-strategy` plugin. Before anything else, resolve two absolute paths and use them throughout: - **PLUGIN_ROOT** — the plugin's install directory: `${CLAUDE_PLUGIN_ROOT}` (Claude Code expands this to an absolute path when it loads this file; read it off and note it down). The grounding files this skill reads — `PHILOSOPHY.md`, `skills/tooling-adequacy/SKILL.md` — live under it, as does `.claude-plugin/plugin.json`, whose `version` field stamps the generated tooling strategy (see "Output"). - **PROJECT_DIR** — the absolute path of the project whose tooling you're planning (normally the current working directory; confirm with the user if it's ambiguous). The strategy docs normally live under `$PROJECT_DIR/quality/` — but `/quality-strategy` asks at session start where the strategy should be saved, so they may live elsewhere. If `$PROJECT_DIR/quality/` is absent, get the docs home instead of assuming: from the orchestrator's brief when this skill was dispatched, else by asking the user; if the path you're given ends in `/quality`, its parent is the home. From then on treat `$PROJECT_DIR` as that docs home wherever a path below says `$PROJECT_DIR/quality/...` — one substitution, made once, before you act on any path. File references below use the `$PLUGIN_ROOT` and `$PROJECT_DIR` placeholders. **Substitute the resolved absolute paths before you act on them.** The Read tool does no variable expansion and resolves relative paths against the current working directory, not this skill's directory — so an unsubstituted placeholder or a bare relative path will fail. ## When to use - **After the lanes — or straight after `/quality-strategy` when the map is blind.** This skill has two natural entry points, and the risk map — the strategy doc's table of quality dimensions, their actual state, and the confidence behind each claim — decides which (Q2 before Q3: you can only investigate what you can judge). When the risk map came out **dominated by Unknowns nothing can judge yet**, run `/evaluation-strategy` then this skill *immediately* — plan the builds that make the project knowable before planning the investigation. A test strategy written against dimensions nobody can judge is mostly "blocked". When the risk map is **mostly answerable**, let `/test-strategy` run first and sharpen the demand (its `/tooling-adequacy` audit surfaces blocked questions), then run this skill on the combined pile. - **When deciding what test or measurement infrastructure to build next** — this skill's output *is* that decision, made with the full demand visible instead of ad hoc. - **Re-run freely.** This is the most re-runnable doc in the pack: re-run when build items land (they change what's answerable — see Update protocol in the output), when a lane adds new demand, or when the quality strategy is revised. **On any re-run over an existing `quality/tooling-strategy.md`, archive it first** — snapshot to `quality/archive/tooling-strategy-.md` (suffix `-2`, `-3`, … if the name is taken — never overwrite an archive) before changing a word, and mention the archive in the closing summary. Never silently rewrite history: the archive leaves a before/after trail the user can compare and share. ## What you need - **Grounding.** Read `$PLUGIN_ROOT/PHILOSOPHY.md`. The disciplines that recur — make confidence visible; record assumptions; push back on vagueness — are load-bearing here. Read step 3 of `$PLUGIN_ROOT/skills/tooling-adequacy/SKILL.md` for the canonical oracle taxonomy (Specified / Property-or-metamorphic / Differential-or-simulated / Golden-master / Human-or-agent-judge) and the "kill the old-world reflex" move; every build item in the plan names its kind from that taxonomy. - **The quality side.** From `$PROJECT_DIR/quality/strategy.md`: Part 6 (the risk map — Unknowns with their to-resolve notes, above all the ones saying "nothing can judge this yet", and the confidence ratings) and Part 5 (the H/M impact ratings, which weight the value of each build). Part 7 for any build-shaped actions already enumerated (Part 7 may be a recorded deferral — fine; the risk map carries everything this skill needs). - **The evaluation side, if it exists.** From `$PROJECT_DIR/quality/evaluation-strategy.md` (or a prior-era `quality/oracle-strategy.md` — the lane was renamed): the agreed next moves that are build-shaped, with any `/oracle-adequacy` verdicts behind them. - **The test side, if it exists.** From `$PROJECT_DIR/quality/test-strategy.md`: agreed moves recorded as blocked on missing instruments or oracles, with their build items. If a lane hasn't run, proceed on what exists and say so in the output — don't guess at demand that hasn't been derived. - **The user.** The value calls below are theirs to confirm, and they usually know pains neither strategy captured — the flaky suite nobody trusts, the metric nobody believes. Ask. If `quality/strategy.md` doesn't exist, stop: this skill plans the build for gaps the other strategies have *found*, and has nothing to consume yet. Point the user at `/quality-strategy` first (or at `/oracle-adequacy` standalone, if they have an existing strategy doc from elsewhere whose actuals' oracles were never audited). ## Session start — itinerary and commit cadence At the start of every working session (first or resumed), before the next move's work, two quick moves: - **Give the itinerary, in plain words.** Five moves, by human name, with rough sizes: *"Gather everything the project can't answer about itself yet (quick — mostly collection); spec each build — what it must do, what it unblocks, build-or-adopt (the bulk of the work); weigh value against cost (short, mostly your calls to confirm); sequence it into phases (short); and close the loop — what each build triggers when it lands."* On resumption or a re-run, lead with what's done or changed since last time, then the remainder. - **Ask the commit-cadence question (when the docs home is git-managed).** First read `quality/.scratch/session-config.md` — the strategy runs usually already settled the cadence for this docs family; restate it in one line instead of re-asking. Where the note is missing and the folder holding the `quality/` docs is git-managed, ask once: *"Want me to commit at each move's boundary, commit everything at the end, or leave the commits to you?"* Suggest commit-as-we-go as the default (cheap rollback, visible progress); where the docs sit in a repo other people share, say in the same breath that each boundary commit publishes the draft to everyone who pulls. Honour the answer at every boundary and record it in `session-config.md`. ## The work, in order Write the doc incrementally as each move completes, so what's done is durable; on re-entry, resume from the last completed move. At each move's boundary, give one progress line (where we are, what remains, relative sizes) and the visible exit — what the user already has if they stop here (the demand table alone is a useful artifact, and the doc picks up where it left off). At a boundary it's safe to `/clear`; say how to resume in the same breath, so resuming is stated, never guessed — *"safe to `/clear`; to pick up, run `/tooling-strategy` again and it reads `quality/tooling-strategy.md` (and the strategies it draws from) and continues from the last completed move."* The user should never feel the work is unbounded, or have to gamble that re-running the skill works. This is also where the boundary commit lands if the user chose commit-as-we-go. ### 1. Gather the demand Collect every place the project currently *can't* answer "how good is it?": - From the risk map: every **judge-blocked Unknown** — its to-resolve note says nothing can judge it yet, or names building the means of judging. An Unknown whose to-resolve note names an action instead — ask a stakeholder, run targeted testing, review a specific area — is not judge-blocked; it routes to the lane that owns that action (stakeholder work back through `/quality-strategy`, or testing/review through `/test-strategy`) rather than landing here. This mirrors the same **you can only investigate what you can judge** handoff `/test-strategy`'s filter states: this skill gathers the *judging* gaps, not every open Unknown. - From the oracle strategy (if present): every **build-shaped agreed move**, with any `/oracle-adequacy` verdicts (an *Over-confident* actual — a claim whose oracle can't support it — or a *Gated* Unknown) behind it. - From the test strategy (if present): every agreed move **blocked on missing instruments or oracles**, with its build item. - From the user: *"Anything else you already know you can't measure or judge, that neither doc captured?"* Record these honestly as user-reported, not strategy-derived. **Deduplicate across the sides.** The same underlying build often appears more than once — the risk map can't establish an actual for the same reason a test-lane question is blocked. Merge them into one item that names every beneficiary — being needed on several sides is exactly what makes it valuable. ### 2. Spec each item — every tool is a mini-product The gathered demand is requirements; this step turns each into a spec and a decision. Per item: - **What it must do** — instrument (exercise/observe) or oracle (judge), the kind from the taxonomy, and the question(s) it must answer at the fidelity they demand. *"Property set: intervals never shrink on a correct recall"*, not *"better testing for the scheduler."* - **What it unblocks** — the dimension(s) and question(s), by name. An item that unblocks nothing doesn't belong in the plan. - **What "good" means for this tool** — one line. The universal dealbreaker is **false confidence**: a tool that passes wrong things is worse than no tool, because the Unknown was at least visible. Good enough = answers its question at the required fidelity, and cheap enough to actually get run. - **Source: build / adopt / buy / extend** — a concrete decision per item, not a default. The asymmetry to expect: *instruments* often already exist (fault injectors, load generators, fuzzers, coverage, monitors — adopting usually beats building); *oracles* more often need building, because they encode *this project's* definition of correct. "Extend" — bolting the missing oracle onto an existing harness — is frequently the cheapest real option. One line of reasoning either way. - **Calibration — the recursive Q2** — how we'll know the tool *itself* judges correctly. Cheap standard moves: seed known-bad cases and confirm it catches them; diff it against the reference implementation; human-verify its first real verdicts. A tool with no calibration plan is a false-confidence machine waiting to happen. - **Who does the work** — agent-buildable now (most reference oracles, property sets, harnesses are), needs human judgment or access (rubrics where a person is the oracle, production telemetry, real-user observation), or a pairing of the two. ### 3. Weigh value and cost Propose a value/cost reading per item and confirm it with the user: - **Value** comes from what it unblocks: the impact rating of the dimensions (H beats M), the priority of the moves it unblocks, and **breadth** — one harness that unblocks five questions beats five one-shot builds. Converting an Unknown on one of the risk map's *hottest* rows is the single highest-value move available. - **Cost** under agent economics: most oracle construction is now agent-cheap — don't price it at human cost. Flag the genuinely expensive items (infrastructure, production access, recruiting humans for judgment sessions) rather than letting them silently sink the plan. - **Trustworthiness of the build itself**: would the built oracle actually be able to judge the question? A cheap oracle that can't is worse than no oracle — it converts a visible Unknown into invisible false confidence. If you doubt an item can work, say so on the item; building a small *probe* of it first can be the Phase-1 step. ### 4. Sequence into a build plan Order the items into phases, with the reasoning stated: - **Quick wins first** — agent-cheap, high-unlock items (typically property sets and simulated/reference oracles on hot dimensions). - **Dependencies** — an oracle that judges observations needs its instrument to exist first; shared infrastructure that several items sit on comes before them. - **Group shared work** — items that ride the same harness or the same telemetry build belong in one phase. - **Defer with reasons** — low-value or speculative items are listed as deferred with a one-line reason, not silently dropped. ### 5. Close the loop For each planned item, state what changes when it lands: which risk-map rows to re-assess (a `/quality-strategy` revision-mode pass over the affected rows), and which blocked moves come unblocked (recorded in the lane's next update). The build plan only pays off because it turns Unknowns into questions we can answer — name that conversion per item, so landing a build triggers the re-assessment instead of leaving the new capability unused. Then write the Update protocol section: re-run this skill when a phase of builds lands, or when either parent strategy is revised. ## Push back when - **Every item is an instrument and none is an oracle** (or vice versa). "Build … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [tollens-ai](https://github.com/tollens-ai) - **Source:** [tollens-ai/quality-strategy-skills](https://github.com/tollens-ai/quality-strategy-skills) - **License:** Apache-2.0 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-tollens-ai-quality-strategy-skills-tooling-strategy - Seller: https://agentstack.voostack.com/s/tollens-ai - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.