Understudy

Understudy

Understudy is agent improvement infrastructure: it helps a coding agent improve its developer's LLM system from real traces. This skill is the orchestrator — it gives your agent the loop and routes each stage to exactly one worker skill. It does not do the work inline.

The loop is local-first: it needs no registration, auth, provider keys, account, or hosted gateway to start. Begin from files the developer already has, produce auditable local artifacts, and only cross into upload, hosted execution, provider spend, or model downloads after explicit approval in the current thread.

The improvement loop

1. Understand the codebase — find where LLM calls happen and the current

model / provider / harness / routing / eval setup.

1. Understand the objective (cost, speed, quality, reliability, compliance, or a

weighted mix).

1. Understand the constraints (what must not be violated).
2. Understand the workload — inspect prompts in situ, trace the request/response

code path, profile the dataset/traces, and confirm the task meaning with the user.

1. Capture or locate real traces.
2. Build or improve a small, meaningful eval harness; rerun the incumbent

baseline.

1. Run local optimization against eval failures.
2. Compare candidate vs baseline on the objective.
3. Recommend the most efficient route — harness, model, supplier, gateway/

inference-layer route, deployment approach.

1. Implement the selected route safely (smallest viable change).
2. Produce an Understudy Agent Improvement Report the developer can review.

Frame every job

Keep these six separate and explicit — say them back before acting:

• Objective — what are we optimizing for?
• Constraints — what are we not allowed to violate?
• Evidence — what traces / evals / prices / measurements do we have?
• Workload — what task does the prompt/data/code path actually represent?
• Route — what harness / model / supplier / deployment path?
• Action — what does the agent actually change?
• Verification — how do we prove the change helped?

Default mode

• Local-first.
• Inspect before changing.
• Measure before optimizing.
• Optimize before routing.
• Compare before deploying.
• Ask before spending money or uploading data.

Show partial findings early ("found 3 LLM call sites"; "app uses LiteLLM, so gateway insertion is low-risk"; "no evals — I'll build a small harness first"; "stated ZDR constraint blocks hosted upload unless approved").

Engagement & pacing

Keep the user oriented and engaged across the loop's slow steps. Full doctrine in [../../docs/engagement-and-pacing.md](../../docs/engagement-and-pacing.md); the rules:

• Estimate before you start. Say how long a step takes and what it costs

before running it (e.g. "100-task baseline ≈ 15–20 min, ~$31"). Measure a small sample first if you don't know the rate.

• Background the slow thing — first. Downloads, baselines, and sweeps run in

the background, started before the interactive part so they finish during it.

• Fill the wait. While a long run goes, cost-model the candidate models at

the user's real volume and pull their benchmarks so the comparison is ready the moment scores land.

• Plan up front; show where they are. Open with a concrete plan; each turn,

state the loop stage (capture → baseline → optimize/compare → validate → decide), re-derived from artifacts on disk.

• Meet them where they are. Read ~/.understudy/profile.json and set

vocabulary, coaching depth, and opinion strength to match. Also read ~/.understudy/agent-card.json when the user asks whether their Understudy is active or how to talk to it; this card is the agent-facing runtime source of truth for model endpoint, tmux session, companion status, and the exact local chat command. No profile yet → run [../onboard/SKILL.md](../onboard/SKILL.md) first.

Inspect before you ask

Answer from the repo before asking the developer. Read package files, provider/ model wrappers, env vars, LLM SDK usage, agent frameworks, prompt files, eval and test dirs, tracing/logging, deploy and CI config, and any compliance/README/ architecture docs. The repo-inspection and call-site checklist lives in [../capture-evidence/SKILL.md](../capture-evidence/SKILL.md) and its reference.md.

Ask only what you genuinely cannot infer:

1. What are we optimizing for? (cost / speed / quality / reliability / compliance

/ weighted mix)

1. What is the target task? (e.g. support triage, tool-use, classification,

extraction, retrieval QA)

1. Which constraints matter? (SOC 2, ZDR, approved providers, no data leaves the

machine, max cost/latency, required model family, region)

1. Which environment? (local-only, Understudy hosted, private gateway, hybrid)

See [reference.md](reference.md) for the intake, objective menus, constraints, route-selection taxonomy, fresh-pricing rule, the report template, anti-patterns, and worked examples.

Route to one worker

Identify the developer's current stage and load exactly one:

• Anthropic / Claude bill reduction — the developer says "lower my

Anthropic bill", "audit my Claude spend", "why is cache_read zero", asks to find Anthropic calls, or arrives through the --lower-my-ant-bill installer path → [../lower-anthropic-bill/SKILL.md](../lower-anthropic-bill/SKILL.md) first. It audits call sites, tokenizer risk, cache structure, batchability, and route candidates before any code edits or provider calls.

• Billing email / browser source inspection — the developer wants to inspect

Anthropic billing emails, receipts, invoices, usage exports, or an authenticated billing website for spend hotspots → [../inspect-billing-sources/SKILL.md](../inspect-billing-sources/SKILL.md). This is optional evidence for lower-anthropic-bill and requires exact read scope approval before email or browser access.

• First run / new user / no profile yet — the developer is new, just

installed the plugin, or asks "where do I start?" and ~/.understudy/profile.json is missing → [../onboard/SKILL.md](../onboard/SKILL.md) (backgrounds a small model, profiles the machine, interviews, writes the profile).

• Product explanation / onboarding copy — the developer asks what

Understudy is, how Desktop, local serving, Fusion sidekick, candidate results, rollout lab, or training workflows work → [../product-knowledge/SKILL.md](../product-knowledge/SKILL.md).

• Workload unclear — the repo has prompts, traces, datasets, eval rows, or

LLM call sites, but the task, data shape, request/response path, or success criteria have not been explained and confirmed → [../understand-workload/SKILL.md](../understand-workload/SKILL.md). Do this before head-to-head comparisons or optimization.

• Codebase / evidence not yet pinned down — LLM call sites, current model/

harness, traces, metric, splits, or incumbent baseline are missing, ambiguous, or stale → [../capture-evidence/SKILL.md](../capture-evidence/SKILL.md) (also owns repo inspection + eval-harness discovery/build).

• Traces already exist — a bucket of captures, a provider log export, or

gateway capture files are in hand and the developer wants them turned into local eval sets, profiled for where the spend goes, or triaged in bulk by content ("which runs failed", "label these by failure mode") → [../ingest-traces/SKILL.md](../ingest-traces/SKILL.md) and its fleet lenses (references/profile-captures.md for structure/cost, references/lotus-semantic-triage.md for content-level bulk triage).

• Inference / routing / capture / auth / deploy — Understudy inference,

gateway trace capture, project/key management, model A/B via route traffic %, understudy run, or registering an improved route → [../use-understudy-gateway/SKILL.md](../use-understudy-gateway/SKILL.md).

• Frontier access choice — a local-vs-frontier comparison, installer, or

agent run needs to decide between the Understudy managed catalog, BYO .env provider keys, or skipping remote frontier calls → [../use-understudy-gateway/SKILL.md](../use-understudy-gateway/SKILL.md) and its frontier-keys lens (references/frontier-keys.md).

• Production ramp / rollback — a route decision exists and live traffic

must move safely: "ramp this route", "move 25% of traffic to the new model", "did the route change regress anything", "roll this back", "prove the savings are real" → [../ramp-and-verify/SKILL.md](../ramp-and-verify/SKILL.md).

• Routing diagnostics / provider health — the developer asks "is

Understudy causing my errors", "which workloads are routed", "check provider health", "are there 500s", "what's our error rate", or wants self-service diagnostics without asking the team → [../check-routing-health/SKILL.md](../check-routing-health/SKILL.md).

• Share savings / leaderboard receipt — the developer wants to share how

much money they saved, submit an anonymous lower-Anthropic-bill result, or send metrics back for the coming leaderboard → [../share-savings/SKILL.md](../share-savings/SKILL.md). Require the metrics-only payload review and explicit approval before posting.

• Acquire / cache / organize local models — download a model, see what's

already cached, free up model disk, pick which Gemma/Nemotron to pull, or explain how open weights work → [../manage-local-models/SKILL.md](../manage-local-models/SKILL.md).

• Local model experiment / local-only route — evaluate a local or

workstation-hosted model through the same frozen workload/eval, choose a runtime, compare local vs remote, or satisfy ZDR / no-upload constraints → [../run-local-model-lab/SKILL.md](../run-local-model-lab/SKILL.md).

• Multi-model sweep / Pareto comparison — the developer has a frozen eval and

wants to run the same rows across several local, gateway, or frontier models to compare quality, latency, cost, and reliability → [../compare-model-sweep/SKILL.md](../compare-model-sweep/SKILL.md). When the sweep verdict supports changing production traffic, don't stop at the report — the answer to "how do I get this running in prod?" is [../ramp-and-verify/SKILL.md](../ramp-and-verify/SKILL.md) (add provider → set route → staged ramp gated by the production validator).

• Token-level logprob comparison — the developer has two same-family eval

runs with stored logprob sidecars and wants to see the exact moment a smaller, quantized, or weaker model diverged from a larger/full-precision sibling on the same row → [../compare-trajectories/SKILL.md](../compare-trajectories/SKILL.md) and its token-logprob lens (references/logprob-lens.md).

• Tool-trace failure diagnosis — an environment-backed or tool-calling run

failed and the developer asks whether the model lacked evidence, failed to retrieve visible evidence, trusted the wrong authority, acted too early, missed exact format, failed ID resolution, or exposed a scorer/harness issue → [../understand-workload/SKILL.md](../understand-workload/SKILL.md) and its tool-trace forensics lens (references/tool-trace-forensics.md).

• Pedagogical learning / privileged-context self-teaching — the developer has

answer keys, execution feedback, verifier traces, oracle tool labels, or canonical solutions and wants the local model to learn from trajectories that are both correct and learnable before SFT, GRPO, or hosted RL → [../local-distillation-lab/SKILL.md](../local-distillation-lab/SKILL.md) and its pedagogical arm (references/pedagogical-arm.md).

• RLM pedagogical training / verifiers / Prime RL — the developer wants to

train or evaluate a Recursive Language Model policy over stateful trajectories, use privileged context as training/scoring signal, measure on-policy state coverage or surprise concentration, or prepare a Prime Intellect verifiers / prime-rl path → [../recursive-language-model/SKILL.md](../recursive-language-model/SKILL.md) and its pedagogical-training lens (references/pedagogical-training.md).

• Hosted run planning — the developer wants to run a hosted job (fine-tune,

RL run, big eval) and asks where to run it, how long it takes, or what it costs → [../plan-hosted-run/SKILL.md](../plan-hosted-run/SKILL.md).

• Single-output optimization — fresh artifacts exist and the developer wants

to validate, optimize (GEPA), compare candidates, or claim readiness → [../optimize-workload/SKILL.md](../optimize-workload/SKILL.md).

• Agentic / tool-use optimization — multi-turn agent that calls tools, from

read-only search loops to multi-step REST/API workflows that mutate state and must pass final-state validators; evaluate, A/B-compare models, or optimize its prompt/route → [../optimize-agentic-workload/SKILL.md](../optimize-agentic-workload/SKILL.md) (verifiers env or resettable sandbox, model A/B through the gateway, prompt-GEPA for the cheap model).

• RL / stateful-policy training handoff — local RLM/distillation rungs are

insufficient and the agent must learn multi-step behavior in external or hosted training → [../prepare-verifier-handoff/SKILL.md](../prepare-verifier-handoff/SKILL.md) (decide → author env → package → hand off).

Multi-turn or tool-use alone is NOT a handoff: agentic evaluation, A/B, and prompt/route optimization stay local in optimize-agentic-workload. RLM policy training routes to recursive-language-model (pedagogical training) first; only external/hosted RL handoffs route to prepare-verifier-handoff. When in doubt, route to capture-evidence — optimizing without a current harness/metric/split/baseline creates false progress.

Local specialization sequencing

When the goal is "train an understudy" / "can a local model do this?", the orchestrator owns the ladder — sequence workers, don't reimplement them:

1. Smallest reasonable rung first. Pick the smallest local model plausibly

reasonable for the task class (easy classification/extraction → Gemma 4 E2B first; coding/structured generation → E2B feel-test then E4B/12B; tool-use → E2B only if the tool surface can be bounded; long-context/high-recall → expect hybrid or remote). Never start at the smallest model blindly, and never evaluate *-assistant drafter checkpoints as standalone candidates. Use [../manage-local-models/SKILL.md](../manage-local-models/SKILL.md) and [../run-local-model-lab/SKILL.md](../run-local-model-lab/SKILL.md).

1. Optional frontier head-to-head for calibration — use

[../ladder/SKILL.md](../ladder/SKILL.md) when the developer needs a no-data local-vs-frontier feel test. Prefer the Understudy managed catalog when the frontier id is listed, so the comparison uses one gateway key and no provider key handling. Preference is a gap signal, never a claim.

1. Freeze the eval (capture-evidence; for workflow/tool tasks,

design-simulated-environment), with frontier as the incumbent baseline and local as the candidate. Then write a gap report and let the gap pick the rung:

• model too weak, harness sane → climb the Gemma/Nemotron ladder;
• prompt or output contract weak → GEPA / prompt repair

(optimize-workload);

• workflow/tool state matters → seeded env

(design-simulated-environment);

• small model drowns in one giant prompt → RLM decomposition

(recursive-language-model);

• local handles easy cases only → hybrid / local-as-router

(run-local-model-lab + use-understudy-gateway);

• local cannot meet quality → stay remote and record the revisit

trigger (new runtime, new weights, larger hardware, better env feedback).

1. Rerun until the claim is real: local beats the incumbent on the agreed

metric for the frozen slice, or the route stays remote — measur

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: understudylabs
• Source: understudylabs/understudy-agent-tools
• License: MIT

Install and usage instructions live in the source repository linked above.