# Optimize Workload

> Use when a developer has a measured eval and wants the prompt or route improved without retraining — "optimize my prompt against my eval", "close the quality gap with the cheaper model", "squeeze cost without losing quality". Keeps holdout sealed, runs GEPA (automatic prompt evolution) on train/dev only, and requires a claim packet before any savings statement.

- **Type:** Skill
- **Install:** `agentstack add skill-understudylabs-understudy-agent-tools-optimize-workload`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [understudylabs](https://agentstack.voostack.com/s/understudylabs)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [understudylabs](https://github.com/understudylabs)
- **Source:** https://github.com/understudylabs/understudy-agent-tools/tree/main/skills/optimize-workload

## Install

```sh
agentstack add skill-understudylabs-understudy-agent-tools-optimize-workload
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Optimize Workload

Use this worker only after the workload has fresh local artifacts from
`capture-evidence`. Validation and optimization must be evidence-led and
split-safe.

## Safety Gates

Default to the cheapest path that still reaches an optimization outcome — not to
zero spend (a skipped improvement has real opportunity cost). Get the
developer's explicit approval before any upload, hosted run, or provider spend.

Do not run live provider calls, hosted jobs, model downloads, uploads,
benchmark submissions, or training without a named surface, capped spend or
download size, exact data class, reviewed dry-run or local plan, and visible
output path under `.understudy/`. Follow the repo public boundary in
[`../../docs/privacy-and-data-boundaries.md`](../../docs/privacy-and-data-boundaries.md)
for prompts, completions, traces, labels, datasets, repo paths, secrets, and
private notes.

## Refusal Gate

Refuse to optimize unless all required artifacts are present and fresh:

```text
.understudy/capture-evidence/harness.json
.understudy/capture-evidence/metric.json
.understudy/capture-evidence/splits.json
.understudy/capture-evidence/baseline.json
```

Fresh means generated for the same workload, metric, split contract, and
incumbent baseline in the current task context. Freshness is hash-bound:
`baseline.json` must include `harness_sha256`, `metric_sha256`, and
`splits_sha256`, and those values must match the current `harness.json`,
`metric.json`, and `splits.json`. If freshness is ambiguous or hashes do not
match, route back to
[`../capture-evidence/SKILL.md`](../capture-evidence/SKILL.md) instead
of optimizing.

## Split Rules

- GEPA is train/dev-only.
- Prompt, route, parser, renderer, and candidate selection changes may use
  train and dev only.
- Never mutate holdout rows, labels, validators, thresholds, or sampling after
  optimization begins.
- Holdout is only for final validation after the candidate is frozen.
- If holdout is touched accidentally, mark the result contaminated and create a
  new split contract before claiming progress.

## Before You Optimize

Confirm these before spending GEPA budget — full detail, model defaults, and
validator kinds in [`reference.md`](reference.md):

- **Headroom** — `baseline.json` must show failing-but-promptable rows. No
  incumbent failures → nothing to optimize. A strong model fails them too →
  task beyond frontier; stop.
- **Models** — student = a cheap candidate; `reflection_lm` = a strong frontier
  model (optional, but a weak one caps quality). **Inference defaults to
  Understudy** after explicit approval:
  `understudy login --email `, then
  `understudy run -- `. Fall back to the developer's own
  provider keys only if they choose BYO. See reference.md → Inference.
- **Scope boundary** — optimizing the prompt, route, or parser of a workload's
  policy model is in scope, including an *agentic / tool-use* policy. Treat one
  full agent rollout as the unit and the rubric as the metric. Only RL
  *trajectory/policy training* and stateful verifier environments are out of
  scope; that is the handoff in
  [`../prepare-verifier-handoff/SKILL.md`](../prepare-verifier-handoff/SKILL.md).
  For agentic eval/optimization, set up the rollout harness and rubric in
  [`../optimize-agentic-workload/SKILL.md`](../optimize-agentic-workload/SKILL.md)
  first, then return here for the GEPA prompt pass.
- **Stopping rule** — if the scorer saturates to 1.0 fast, the surface is too
  easy; strengthen the metric, don't claim. If GEPA stalls with headroom left,
  recommend the next rung (SFT/distillation); this skill does not train.

## Flow

1. Inspect the required artifacts and confirm they describe the same workload.
2. Re-state the metric, validator, split boundary, incumbent score, latency
   basis, cost basis if available, and failure taxonomy.
3. Select the cheapest intervention that matches the observed failure mode:
   prompt repair, parser/schema repair, context trimming, route change,
   candidate model comparison, or GEPA. When the complaint is cost (not
   quality) and inputs dominate the bill, check prompt-cache structure first —
   it's often the cheapest lever of all:
   [`references/prompt-cache-optimization.md`](references/prompt-cache-optimization.md).
   Pick the cheapest *target* that matches
   the failure too — see [`reference.md`](reference.md) → Optimization-Target
   Menu for the full list. For an agentic workload, treat latency
   and cost per rollout as first-class objectives alongside the rubric score —
   tool-call count, redundant calls, and wasted context are common, optimizable
   failure modes, not just quality misses.
4. For GEPA/DSPy execution, use a small local `uv` environment only after
   explicit approval. Do not vendor GEPA/DSPy or depend on a full private
   runtime. The CLI owns a registry-backed adapter wrapper:
   `optimize-workload adapter run --adapter  ... --execute`.
   `eval-input-gepa` runs upstream GEPA locally without provider calls unless a
   model-backed path is explicitly selected. `adapter run --adapter dspy-gepa
   --execute` resolves the Understudy API key, passes it to the child process
   through environment only, and runs train/dev rows through the gateway. GEPA's
   edge is natural-language feedback: the metric must return a diagnosis of
   *why* each failing row failed and what to change, not a bare score — bland
   feedback wastes the optimizer. For an agentic workload this means the
   per-criterion rubric must emit, per failing row, a short natural-language
   note tied to the rollout — e.g. "called search 6 times for a fact on the
   first result page; tighten the stop condition" or "answer omitted the date
   filter the question required; add it to the query plan" — rather than a bare
   `0`.
5. Keep deterministic work in the TypeScript CLI and this skill's templates. Follow
   [`../../docs/optimize-workload-contract.md`](../../docs/optimize-workload-contract.md)
   for adapter, metric feedback, and claim packet details.
6. When GEPA is available and explicitly approved, run train/dev-only and
   record the command, model/deployment, metric-call budget, seed, selected
   candidate, rejected variants when available, and whether provider calls were
   made through Understudy.
7. Freeze the candidate before any holdout validation.
8. Run holdout only once the candidate is frozen, and record score, failures,
   latency basis, cost basis, fallback route, demotion trigger, and caveats.

The home of record for an optimization run is the **active experiment**
directory (see the Experiment section of
[`../understudy/SKILL.md`](../understudy/SKILL.md) for the record shape):

```text
.understudy/experiments//      # experiment.json, candidate.json, claim.json
```

Open or reuse an experiment before optimizing with `understudy experiments new`
(it pins the current baseline); `experiments/active` names it. On
`optimize-workload adapter run --execute` the produced candidate is frozen into
the active experiment directory automatically; `understudy experiments freeze`
does it explicitly and also freezes a `--claim-from `. The optimizer's
working artifacts (`eval-input-candidate.json`, `proof-packet.json`, adapter
outputs) stay under `.understudy/optimize-workload/`. Run `understudy next` to
see the current loop step.

The previous Python helper scripts have been removed with the Python CLI
prototype. Use the TypeScript CLI gates first, and still inspect artifacts
directly when a workload has unusual validator, split, or claim-boundary shape.
Block on stale hashes, missing metric feedback, unapproved provider calls, or
proxy-only validation.

Use the CLI guide before creating a local optimizer env:

```bash
understudy skills --search gepa
understudy optimize-workload adapter run --repo . --adapter eval-input-gepa --manifest eval-input-manifest.json --execute
```

If approved, keep Python isolated under ignored local runtime state:

```bash
uv venv .understudy/venvs/optimize
uv pip install --python .understudy/venvs/optimize/bin/python 'gepa>=0.0.27,=3.0.0'
```

## Claim Rules

Do not claim savings without:

```text
.understudy/experiments//claim.json
```

`claim.json` must cite `harness.json`, `metric.json`, `splits.json`,
`baseline.json`, and the frozen candidate artifact. It must include the same
`harness_sha256`, `metric_sha256`, and `splits_sha256` values from the
baseline contract, plus `baseline_sha256` and the frozen candidate hash. It
must also include sample size, split used, score delta, latency basis, cost
basis, price assumptions, request-volume assumption, confidence level, caveats,
fallback route, and demotion trigger. For an agentic workload, latency and cost
basis are per-rollout and mandatory, not optional: report the delta in tool-call
count and end-to-end rollout cost alongside the rubric delta, so a quality gain
bought with more calls or higher latency is visible rather than hidden.

Per-row eval evidence behind any claim — the baseline rerun, dev-set
comparisons, and the frozen-candidate holdout table — must be recorded as
`understudy.eval_result.v1` rows
([`schemas/understudy.eval_result.v1.schema.json`](../../schemas/understudy.eval_result.v1.schema.json)).
The row-level `provenance.harness_sha256` and `provenance.split_sha256` fields
carry the same hash chain the claim packet cites, a `score` of 0 is a scored
failure (never a missing value), and `unscored` rows are excluded from
averages rather than counted as 0.

No claim may imply replacement readiness, production readiness, or recurring
savings unless those fields are present and the holdout evidence supports the
statement. If the evidence is train/dev-only, call it an optimization lead, not
a win.

## Output Standard

End with:

- required artifacts inspected and freshness status;
- validation or optimization run;
- split used and whether holdout remained untouched;
- candidate status and whether `claim.json` exists;
- result type: validation, optimization, heldout, or blocked;
- one recommended next local command or action.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [understudylabs](https://github.com/understudylabs)
- **Source:** [understudylabs/understudy-agent-tools](https://github.com/understudylabs/understudy-agent-tools)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-understudylabs-understudy-agent-tools-optimize-workload
- Seller: https://agentstack.voostack.com/s/understudylabs
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.