# Evaluation Strategy

> A deliberately light follow-up to /quality-strategy for the evaluation lane — ingest the release's quality strategy, filter for the ilities where better judging or better indication can make a dent (Unknowns nothing can judge yet, claims resting on signals that couldn't support them, "we wouldn't notice if it broke"), then per ility discuss what quality instruments — oracles and proxies — you hav…

- **Type:** Skill
- **Install:** `agentstack add skill-tollens-ai-quality-strategy-skills-evaluation-strategy`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [tollens-ai](https://agentstack.voostack.com/s/tollens-ai)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [tollens-ai](https://github.com/tollens-ai)
- **Source:** https://github.com/tollens-ai/quality-strategy-skills/tree/main/skills/evaluation-strategy

## Install

```sh
agentstack add skill-tollens-ai-quality-strategy-skills-evaluation-strategy
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Evaluation Strategy

One of the three light follow-ups to `/quality-strategy` — the **evaluation** lane, beside `/test-strategy` (testing) and `/process-strategy` (rules, invariants, and processes). This lane plans how the project will *know where it stands* — per prioritised ility — rather than trusting whatever signal happens to exist.

Its vocabulary is **quality instruments**, which differ in **authority** (the shared definitions live in `$PLUGIN_ROOT/skills/test-strategy/FRAMINGS.md` #8):

- An **oracle** is trusted to **judge** — it tells you whether what you observe is good.
- A **proxy** **indicates** — it correlates with quality, is usually cheap to check, and has known blind spots; satisfying it never proves the ility is met.

Both belong in this lane: a project knows where it stands through a mix of trusted judges and cheap indicators, and the craft is knowing which of your signals is which. (One nearby term to keep distinct: the pack also uses **instrument** in a narrower sense — the tool that lets you *observe* at all, a test rig or telemetry feed, per `/tooling-adequacy`. An observation instrument feeds both oracles and proxies; this lane's umbrella term for oracle-or-proxy is *quality instrument*.)

**This skill is deliberately much lighter than `/quality-strategy` — and that's a design decision, not a shortcut.** The quality strategy is the thorough one: a structured multi-session interview that walks you step by step and refuses to skip substance. This skill assumes that work is done and does something different: it gives you **high-level ideas and good questions**, per ility, and captures what you decide. It will not hold your hand through a step-by-step process, dispatch analysis fan-outs, or re-derive what the quality strategy already established. One or two focused conversations, one short document. When a discussion here needs real depth, the skill points at the deeper tools rather than becoming one: `/oracle-adequacy` to audit whether existing oracles can actually be trusted, `/tooling-strategy` to turn agreed build items into a prioritised build plan.

**Formerly `/oracle-strategy`.** Older strategy docs, deferral notes, and session records may point at `/oracle-strategy` or `quality/oracle-strategy.md` — treat those pointers as pointing here. If a `quality/oracle-strategy.md` exists from an earlier run, it is this lane's prior doc: read it as such, and on update archive it under its own name (`quality/archive/oracle-strategy-.md`) before writing `quality/evaluation-strategy.md` with a one-line note that the lane was renamed.

## Resolving file paths — do this first

This skill is part of the `quality-strategy` plugin. Resolve two paths and use them throughout:

- **PLUGIN_ROOT** — the plugin's install directory: `${CLAUDE_PLUGIN_ROOT}` (Claude Code expands this to an absolute path when it loads this file; read it off and note it down). Grounding files — `PHILOSOPHY.md`, `skills/test-strategy/FRAMINGS.md` (#8 — the shared oracle/proxy definitions), `skills/tooling-adequacy/SKILL.md` (the oracle taxonomy), `.claude-plugin/plugin.json` (whose `version` field stamps the output) — live under it.
- **DOCS_DIR** — where the `quality/` docs live; every `quality/...` path below resolves under it. Normally the current working directory — but `/quality-strategy` asks at session start where the strategy should be saved and records the choice in `quality/.scratch/session-config.md`; this doc joins that same family. If the working directory has no `quality/`, ask the user where the strategy was saved (a path ending in `/quality` means its parent is the home) — never scaffold a fresh `quality/` beside code whose strategy lives elsewhere.

Substitute resolved absolute paths before acting on them — in your own Reads and in any subagent brief; the Read tool does no variable expansion, and a dispatched subagent inherits none of your context.

## Before you start

1. **`$DOCS_DIR/quality/strategy.md` must exist for this release**, completed at least through Part 6 (Risk Map). If it doesn't, stop and direct the user to `/quality-strategy` — this skill filters and discusses that strategy's prioritised ilities; without it there is nothing to filter.
2. **Read `$PLUGIN_ROOT/PHILOSOPHY.md`** (make confidence visible; interview, don't infer; push back on vagueness), **FRAMINGS #8 in `$PLUGIN_ROOT/skills/test-strategy/FRAMINGS.md`** (the oracle/proxy authority distinction and the labelled-proxy-goal rule), and **step 3 of `$PLUGIN_ROOT/skills/tooling-adequacy/SKILL.md`** — the oracle kinds (Specified / Property-or-metamorphic / Differential-or-simulated / Golden-master / Human-or-agent-judge) and the "kill the old-world reflex" move. Those are the idea vocabulary for the whole discussion.
3. **Session choices.** Read `quality/.scratch/session-config.md` and restate the standing choices in one line — *including* where these answers will be recorded and who can read them: this lane's discussion is candid too (what's actually weak, what nobody really follows), and the candor is only safe while the user knows where their words go. Ask only what the note is missing; if there is no note at all, run the save-location ask exactly as `/quality-strategy` → "Session start" defines it before anything is written.

## The session — ingest, filter, then discuss per ility

**1. Ingest the release's quality strategy.** Read `quality/strategy.md`: the header's `Release:` line (this doc inherits it), Part 3's stakeholder bars, Part 5's H/M-rated ilities, Part 6's risk map — especially the actuals whose basis is thin: Unknowns, low-confidence claims, and anything with an honest "no investigation yet". Part 4's non-goals bound everything. Don't re-litigate the strategy; it's the input. Also read `quality/ideas.md` (the ideas ledger) if it exists: ideas the user volunteered spontaneously mid-strategy, in their words, with no role assigned. Consider each for this lane — could it serve as an oracle or a proxy for a dimension's state? — and raise the fits when their ility comes up; annotate an adopted entry in the ledger (*"→ taken up in evaluation-strategy, "*) rather than deleting it, since the same idea may also serve a sibling lane; an idea whose ility this lane's filter drops simply isn't raised here — it stays unannotated in the ledger for whichever lane kept that ility.

**2. Filter — where can better evaluation make a dent?** From the H/M ilities, propose the subset where the *knowing* is the bottleneck: actuals sitting at Unknown because nothing can judge them; confident claims resting on signals that couldn't support them; dimensions where the honest answer to *"would we notice if this broke?"* is no; and ilities tracked **only by an unbounded proxy** — a coverage number or a green CI badge whose blind spots nobody has ever stated. For each kept ility, one line of why; name the left-out ones too, with why not (usually: the evaluation is fine — the gap is testing effort or process, so it belongs to a sibling lane). Confirm the filter with the user before drilling in — it's a proposal, not a verdict.

**3. Per kept ility, in priority order, discuss three questions.** High-level ideas and questions — offer candidates, ask, capture what the user decides. Keep each ility to a few minutes unless the user wants to go deep.

- **What do you have already?** What currently observes this ility (the observation instrument), and what evaluates what you see — and with what authority: which signals are **oracles** (trusted to judge) and which are **proxies** (indicate, with blind spots)? Name each plainly, and for each proxy, say what it misses: *"CI is green"* judges what the suite asserts — what does it assert about *this* ility, and what stays invisible? Authority and coverage are orthogonal: an oracle can be authoritative over only part of the surface (native-speaker review of three locales of twelve is an oracle for those three and nothing for the rest — not a proxy for all twelve); record both what a signal can judge and how much of the surface it sees. If whether an existing oracle can be *trusted* becomes the real question, that's an `/oracle-adequacy` audit — offer it rather than improvising one here, and dispatch it with the resolved absolute `$DOCS_DIR` doc path and a scratch destination (`quality/.scratch/oracle-adequacy-.md`) in the brief; a sealed dispatch can't ask where the docs live.
- **What could be improved?** Where an oracle judges the wrong thing or too coarsely — aggregate uptime standing in for tail latency, a smoke test standing in for data integrity, a dashboard nobody reads. And where a proxy is being *read as more than it is* — the fix there is often not replacing it but **bounding it**: state what it covers and what it misses, so it indicates honestly instead of comforting.
- **What could be added?** Two idea lists. **Oracles**, by kind: a stated property set ("no path loses user data"), a cheap simulated/reference implementation to diff against, a golden master, a defined SLO plus its measurement, a human or agent judge for taste/trust dimensions (name its limits). Under agent costs an oracle is usually cheap to build — *"there's no way to judge this"* is almost always *"we haven't decided how to judge this"*. **Proxies**: cheap leading indicators worth adopting *as labelled proxy goals* — full test coverage, a review run over every code change, clean architecture are all good quality proxies (FRAMINGS #8); adopting one is legitimate — state what it is a proxy *for* and what remains unknown about the ility when it's satisfied. (A proxy like review-per-change is often simultaneously a process rule — flag the overlap to `/process-strategy` rather than duplicating it.)

**Capture as you go**: per ility, a few lines under Have / Improve / Add, plus one to three **agreed next moves** — each naming whether the signal it creates is an oracle or a proxy, and for a proxy, what stays unknown when it's satisfied. An idea the user rejects is recorded as considered-and-set-aside if it sharpened anything; otherwise dropped, not padded.

## Push back when

- Every ility comes back "our current signals are fine" with no per-ility reason — that's the Q2 collapse this lane exists to catch. Ask, per ility: *"what exactly would have told us if this were below the bar last month?"*
- A proxy is treated as the thing itself — *"coverage is at 100%, so correctness is handled."* There is nothing wrong with the proxy goal; the failure is reading proxy satisfaction as quality achieved. Ask: *"when that's green, what do we still not know about this ility?"* — and record the answer on the proxy.
- An Unknown is treated as permanently unknowable. Propose the cheapest oracle kind that could judge it, and record it as a candidate build even if deferred.
- A taste/trust/feel dimension gets handed a purely automated oracle. The human is the oracle there; say so and record the human's role.
- The discussion starts re-deriving required levels or re-rating dimensions. That's the quality strategy's ground — revise it there (`/quality-strategy` revision mode), don't fork it here.

## Output

Write to `$DOCS_DIR/quality/evaluation-strategy.md`, incrementally as ilities are agreed. Shape:

```markdown
# Evaluation Strategy:  — 

*Last updated: *
*Release: *
*Generated by the `evaluation-strategy` skill — quality-strategy-skills (tollens-ai) v · github.com/tollens-ai/quality-strategy-skills*

*A light companion to `quality/strategy.md` — high-level oracle and proxy ideas and decisions per ility, not a step-by-step plan. Depth lives in `/oracle-adequacy` (audit) and `/tooling-strategy` (build plan).*

## The filter — where better evaluation makes a dent

| Ility (from Part 5) | In this lane? | Why / why not |
|---|---|---|

## 

**Have already.** 
**Could improve.** 
**Could add.** 
**Agreed next moves.** 

## Out of the evaluation lane this release

```

On a later update, archive first (`quality/archive/evaluation-strategy-.md` — never overwrite an archive; a prior-era `quality/oracle-strategy.md` archives under its own old name), refresh the header stamp, and record a short `## Since the last cycle` section: what each prior agreed move actually produced, and what's newly worth deciding.

Close with the standing gates the pack shares: **invoke `/effective-comms`** on the doc before calling it final, and offer the onward pointers — `/tooling-strategy` if agreed next moves include real builds, `/oracle-adequacy` standalone if trust in existing oracles stayed contested, and the sibling lanes (`/test-strategy`, `/process-strategy`) for the ilities this filter handed to them.

## Escalation points

- The quality strategy's risk map is missing or all-`?` — stop; this lane needs rated ilities and honest actuals to filter. Direct to `/quality-strategy`.
- The user wants this session to *build* the oracles or wire up the proxies. Building is downstream — capture the agreed moves and point at `/tooling-strategy` (plan) or just do the build after the doc is closed, outside this skill.
- The user asks for the full interview treatment. Offer honesty instead: this lane is deliberately light; if the release genuinely needs deep evaluation work planned end-to-end, that is `/tooling-strategy`'s job with this doc as its demand.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [tollens-ai](https://github.com/tollens-ai)
- **Source:** [tollens-ai/quality-strategy-skills](https://github.com/tollens-ai/quality-strategy-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-tollens-ai-quality-strategy-skills-evaluation-strategy
- Seller: https://agentstack.voostack.com/s/tollens-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.