Oracle Adequacy

Oracle Adequacy

This skill answers the second of the four quality questions — "How do we know if what we have is good?" — for the actual-state assessment of a quality strategy. It checks whether you can trust how the strategy decided "this dimension is actually at level X".

It is the actual-state counterpart to /tooling-adequacy (which does the same job for the questions /test-strategy wants answered). The two share one oracle core; they differ in what they assess. /tooling-adequacy assesses a learning need (a question the strategy wants testing to answer); /oracle-adequacy assesses a dimension's actual-state claim — the entries Part 6 (Risk Map) records as the project's current level on each H/M dimension.

Judging where a project actually stands on a dimension takes two distinct capabilities, and both can fail independently:

• an instrument — to observe the current state on this dimension (run the thing, inspect the code, read the telemetry, ask the stakeholder);
• an oracle — to judge whether what you observed means the dimension is at the claimed level.

An actual-state claim is only trustworthy if both are adequate. A perfect instrument with no oracle means you can see plenty and still not know if "plenty" is good enough; a perfect oracle with no instrument means you know what "good" looks like but have nothing observed to compare it to.

This skill exists because of two reliable agent failure modes. (1) When "how do we know?" is collapsed into "is it good?", agents claim an actual level (often a comfortable Medium) by deferring to whatever signal happens to exist, never asking whether that signal can actually judge this dimension — so the risk map records confidence the evidence doesn't support. (2) When there's no obvious oracle, agents mark the dimension Unknown and move on as if Unknown were a dead end. But an oracle is usually cheap to build now, and building one turns the Unknown into a knowable actual.

Resolving file paths — do this first

This skill is part of the quality-strategy plugin. Before anything else, resolve two absolute paths and use them throughout:

• PLUGIN_ROOT — the plugin's install directory: ${CLAUDE_PLUGIN_ROOT} (Claude Code expands this to an absolute path when it loads this file; read it off and note it down). The grounding files this skill reads — PHILOSOPHY.md, and skills/tooling-adequacy/SKILL.md for the shared oracle taxonomy — live under it.
• PROJECT_DIR — the absolute path of the project whose actuals you're assessing (normally the current working directory; confirm with the user if it's ambiguous). The strategy docs normally live under $PROJECT_DIR/quality/ — but /quality-strategy asks at session start where the strategy should be saved, so they may live elsewhere. If $PROJECT_DIR/quality/ is absent, get the docs home instead of assuming: from the orchestrator's brief when this skill was dispatched, else by asking the user; if the path you're given ends in /quality, its parent is the home. From then on treat $PROJECT_DIR as that docs home wherever a path below says $PROJECT_DIR/quality/... — one substitution, made once, before you act on any path.

File references below use the $PLUGIN_ROOT and $PROJECT_DIR placeholders. Substitute the resolved absolute paths before you act on them. The Read tool does no variable expansion and resolves relative paths against the current working directory, not this skill's directory — so an unsubstituted placeholder or a bare relative path will fail.

When to use

• From /evaluation-strategy — the evaluation lane's audit engine, offered when the have-already discussion contests whether the oracles behind a dimension's actuals can be trusted. Input: the dimensions in the lane's filter, with their actuals and evidence from the quality strategy's Part 6. Output: an oracle-adequacy assessment plus a list of oracle-build items the lane records against its agreed moves (and /tooling-strategy consumes).
• Standalone — to audit the oracles behind an existing strategy's actuals. Input: the dimensions and claimed actuals from $PROJECT_DIR/quality/strategy.md (Parts 5–6), plus what the team can observe about the codebase.

This skill judges adequacy and names the gaps; it does not plan the build. /tooling-strategy consumes its oracle-build items (together with /tooling-adequacy's, from the test side) and turns them into a prioritised build plan.

What you need

• Grounding. Read $PLUGIN_ROOT/PHILOSOPHY.md. The disciplines that recur — make confidence visible; push back on vagueness; record assumptions; understand the why — are load-bearing here.
• The shared oracle taxonomy. Read step 3 of $PLUGIN_ROOT/skills/tooling-adequacy/SKILL.md. The oracle kinds (Specified / Property-or-metamorphic / Differential-or-simulated / Golden-master / Human-or-agent-judge) and the "kill the old-world reflex" move are the canonical oracle core; this skill applies the same taxonomy to actual-state assessment rather than to test learning needs. The one-line gists are restated in step 3 below so this file is usable on its own, but the fuller treatment lives there.
• The dimensions and their claimed actuals. From /quality-strategy: the H/M dimensions from Part 5 with the proposed actuals and evidence from sub-step 6.2. Standalone: read Parts 5 and 6 of $PROJECT_DIR/quality/strategy.md, and ask the user what they can observe if the doc is thin.
• The project, lightly. Unlike /tooling-adequacy, reading source is allowed here — actual-state assessment is about the system as built, not about preserving an independent testing perspective. But read only enough to judge whether an oracle is feasible; this skill assesses the means of knowing, not the actual state itself (that's 6.2's job).

The work, in order

1. For each dimension, name the instrument and the oracle behind its claimed actual

For each H/M dimension (standalone: each dimension with a claimed actual), state the two things the actual-state claim rests on:

• Instrument — how the current state on this dimension was (or would be) observed. Be specific: "read the error-handling paths in module X", "ran the app and timed cold start", "asked the maintainer."
• Oracle — how anyone decides the observation means the dimension is at the claimed level. Name it explicitly. A claim with an observation but no stated oracle — "reliability is Medium because the code looks careful" — has no oracle yet. That's a finding, not a detail.

For dimensions the strategy marks Unknown, the instrument and oracle are what would be needed to resolve the Unknown. The risk map's "to resolve" note is the starting hypothesis; this skill pressure-tests whether that resolution path actually has an oracle.

2. Assess the instrument — Adequate / Inadequate / Missing

• Adequate — exists and can observe the current state on this dimension at the fidelity the required level demands.
• Inadequate — exists but can't reliably produce the needed observation (the only telemetry is aggregate uptime, but the dimension is tail-latency; a code skim can't judge concurrency-safety).
• Missing — nothing observes this dimension yet; it must be built, run, or asked for.

3. Assess the oracle — Adequate / Inadequate / Missing — and don't accept "there's no oracle, so it's Unknown"

Classify the oracle on the same scale. The oracle is whatever lets you decide an observation means the claimed level. The kinds, from cheapest signal to richest (the canonical treatment is in $PLUGIN_ROOT/skills/tooling-adequacy/SKILL.md step 3):

• Specified — a spec, contract, SLO, or known-correct target states the expected level (*"p99 .md) — hard evidence the audit ran; a claimed-but-missing audit output is a fabrication signal. The orchestrator's brief carries the absolute docs-home path — a sealed dispatch can't ask where the docs live, so write where the brief says, never a path derived from your own working directory. Standalone, surface it in the conversation and offer to write it to $PROJECT_DIR/quality/oracle-adequacy-.md`. Shape:

# Oracle adequacy — actual-state assessment

*Do we actually know where we stand on each dimension — and would we know if a claim were wrong?*

| Dimension | Instrument (observe state) | Oracle (judge level) | Verdict | Oracle-build items |
|---|---|---|---|---|
| … | … — Adequate/Inadequate/Missing | … — Adequate/Inadequate/Missing | Trustworthy / Over-confident / Gated | … |

## Oracle-build items (seed plan of work)

- **** — unblocks the actual-state assessment of . 

(Or: "None — every assessed dimension has an adequate instrument and oracle, with the per-dimension reasons above.")

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: tollens-ai
• Source: tollens-ai/quality-strategy-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.