# Compare Model Sweep

> Use when a developer wants to compare candidate models — any mix of local, gateway, or frontier — on the same eval and see quality, latency, cost, and reliability side by side. "Which model should I use", "sweep these models on my benchmark", "compare Gemma vs the frontier on my eval". To stand up and serve a local candidate first, use run-local-model-lab.

- **Type:** Skill
- **Install:** `agentstack add skill-understudylabs-understudy-agent-tools-compare-model-sweep`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [understudylabs](https://agentstack.voostack.com/s/understudylabs)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [understudylabs](https://github.com/understudylabs)
- **Source:** https://github.com/understudylabs/understudy-agent-tools/tree/main/skills/compare-model-sweep

## Install

```sh
agentstack add skill-understudylabs-understudy-agent-tools-compare-model-sweep
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Compare Model Sweep

Use this worker when the question is not "can one model pass this eval?" but
"which candidate sits on the useful frontier for this workload?" The skill runs
one frozen harness across a candidate matrix, records each run, and emits a
small Pareto report an agent can use for route decisions.

Prefer this after [`../understand-workload/SKILL.md`](../understand-workload/SKILL.md),
[`../optimize-agentic-workload/SKILL.md`](../optimize-agentic-workload/SKILL.md), or
[`../run-local-model-lab/SKILL.md`](../run-local-model-lab/SKILL.md) has already
identified a resettable eval and a first local candidate.

## Safety Gates

Get explicit approval before provider spend, remote gateway calls, benchmark
submissions, uploads, or new model downloads. Local cached model runs are fine
when the user has already approved the local harness. Never mix private traces or
customer data into a public sweep report; use synthetic, anonymized, or local-only
artifacts.

Do not claim a model is cheaper, faster, or better unless the sweep used the same
rows, harness, metric, tool-access mode, prompt, seed, and state reset for every
candidate.
If those differ, label the result as an ablation or diagnostic, not a Pareto
comparison.

## Flow

1. **Freeze the comparison contract.** Record workload id, harness command,
   split/row ids, metric, prompt, tool-access mode, seed, timeout, concurrency,
   and budget.
   Write it to `.understudy/model-sweeps//sweep-plan.json`.

2. **Normalize candidate routes.** Include each model id, route type
   (`local`, `understudy-managed-catalog`, `understudy-routed`,
   `byo-provider`, or `hosted-other`), endpoint/base URL, loader/runtime,
   local-vs-remote boundary, max tokens, reasoning effort, pricing basis, and
   whether the model is cached or requires download.

   For remote Understudy candidates, run `understudy models list --json`;
   prefer managed-catalog ids unless BYO provider-direct behavior is required.
3. **Run a smoke row first.** Each candidate must complete one row without
   connection errors, empty responses, or bad model aliases. For local MLX, prefer
   verified filesystem paths or Understudy snapshot aliases over arbitrary
   Hugging Face ids.

4. **Run the frozen matrix.** Call the same harness once per candidate with the
   same rows, split, tool-access mode, prompt, seed, and export path. Keep each
   export under `.understudy/model-sweeps//candidate-runs//`,
   with per-row results in the required `understudy.eval_result.v1` eval-evidence format ([schema](../../schemas/understudy.eval_result.v1.schema.json)).

5. **Summarize at the same grain.** Build `summary.csv` with candidate, route,
   route type, requested model, effective model, model family, tool-access mode,
   task count, pass rate, partial credit, total tokens, cost/task, run seconds,
   errors, empty responses, and caveats. If per-task latency is unavailable, use
   run-level duration and say so.

   For Understudy gateway runs, capture `x-understudy-mode`,
   `x-understudy-route`, and `x-understudy-effective-model`; exclude rows where
   requested and effective model disagree unless that is the test.

   **Caching parity.** Cost columns must state each candidate's caching basis.
   If the incumbent runs cache-warmed in production (e.g. a cached primer),
   measure same-provider candidates cache-warmed too; when a candidate has no
   equivalent cache layer (a different provider or gateway can't share the
   incumbent's cache pool), measure it uncached and label it — what you
   measured is what they'd pay. Note that batching or longer cache TTLs shift
   absolute costs on the cached side but should not change the ratios. To
   structure the harness itself for cache hits (and avoid the
   parallel-fan-out all-miss trap), see
   [`optimize-workload/references/prompt-cache-optimization.md`](../optimize-workload/references/prompt-cache-optimization.md).

   **Pairwise option.** When the workload has no programmatic metric
   (open-ended generation), score quality as a pairwise preference against the
   incumbent instead of an absolute rubric: same row, two outputs, an LLM
   judge picks A/B/tie — run **twice with the order swapped** and count a win
   only when both passes agree. Report the debiased win-rate with N. Dry-run
   the judge on a few rows first, budget-gate the live judge like any provider
   spend, and keep judge-scored quality as its own column — never silently
   blended with a programmatic metric.

6. **Compute the frontier.** A candidate is dominated when another candidate has
   equal or better quality and equal or lower cost and latency, with no worse
   error rate or safety result. Write `pareto.json` with dominated reasons.

7. **Report the decision.** Write `report.md` with the top frontier candidates,
   the cheapest acceptable model at the agreed quality floor, and the next action:
   ship route, build retrieval/tooling, run GEPA, climb local model, or use remote.

## Harness Pattern

Use the workload harness's JSON exports directly. Keep the command concrete in
`sweep-plan.json`, but avoid changing anything except the candidate route:

```sh
"$HARNESS_CMD" \
  --model "$MODEL" \
  --endpoint "$ENDPOINT" \
  --rows "$ROWS" \
  --tool-access "$TOOL_ACCESS" \
  --export-json ".understudy/model-sweeps/$RUN/candidate-runs/$ID/results.json"
```

For keyless managed-catalog sweeps, use one cleared/no-route workload and vary
only request-body `model`. Do not use a traffic split unless the non-routed
passthrough share has a managed provider credential or BYO key.

For API-workflow sweeps, keep tool-access interpretation explicit:

- Broad or production tool access is the deployable baseline.
- Curated, narrow, or oracle tool access is diagnostic unless a retriever or
  advisor is being evaluated and scored separately.

## Output Standard

End with the sweep path, candidate count, split size, quality/cost/latency axes,
which candidates are on the frontier, which are dominated, and the recommended
next action.

When the sweep informs a real route decision, also write the report as a
**decision memo** the developer can paste to their team unedited:

- a results table: candidate, pass rate against the production validator,
  total cost, cost per unit of work the business counts (per deal, per ticket,
  per call — not per token), and the cost ratio vs the incumbent;
- caching basis per row (see step 5) and any other comparability caveats;
- where the residual failures cluster, as named patterns with the cheapest fix
  per pattern (prompt line, decode/format fix, decomposition, route fallback);
- a staged recommendation: the drop-in safe swap, the parallel pilot, and the
  cheap iteration, rather than one all-or-nothing verdict;
- a scope line: what fraction of the workload's total LLM cost this step
  represents, and what's next if the approach extends;
- one headline the finance owner can multiply by volume ("saves ~$X per
  ; multiply by monthly volume").

If the verdict supports changing production traffic, hand off to
[`../ramp-and-verify/SKILL.md`](../ramp-and-verify/SKILL.md): add the provider,
set the route, and ramp staged traffic gated by the same production validator
the sweep used.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [understudylabs](https://github.com/understudylabs)
- **Source:** [understudylabs/understudy-agent-tools](https://github.com/understudylabs/understudy-agent-tools)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-understudylabs-understudy-agent-tools-compare-model-sweep
- Seller: https://agentstack.voostack.com/s/understudylabs
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.