# Capture Evidence

> Use when a developer wants to build an eval from their real LLM app before changing anything — "measure how my app is doing today", "build an eval from my workload", "we have no baseline", "is my current model actually good". Turns the workload into auditable local artifacts (harness, metric, frozen splits, baseline); has a public-benchmark on-ramp when no traces exist.

- **Type:** Skill
- **Install:** `agentstack add skill-understudylabs-understudy-agent-tools-capture-evidence`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [understudylabs](https://agentstack.voostack.com/s/understudylabs)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [understudylabs](https://github.com/understudylabs)
- **Source:** https://github.com/understudylabs/understudy-agent-tools/tree/main/skills/capture-evidence

## Install

```sh
agentstack add skill-understudylabs-understudy-agent-tools-capture-evidence
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Capture Evidence

Use this worker when the developer has not yet produced current local evidence
for the workload, or when any core artifact is missing, stale, ambiguous, or
untrusted.

The OSS loop does not require registration, auth, provider keys, an Understudy
account, or hosted gateway access. Do the smallest local pass that turns the
workload into auditable artifacts.

## Safety Gates

Default to the cheapest path that still reaches an optimization outcome — not to
zero spend (a skipped improvement has real opportunity cost). Get the
developer's explicit approval before any upload, hosted run, or provider spend.

Follow the repo public boundary in
[`../../docs/privacy-and-data-boundaries.md`](../../docs/privacy-and-data-boundaries.md).
Prefer metadata, paths, hashes, counts, schemas, and redacted examples. Do not
read, print, commit, upload, or transmit raw prompts, completions, traces,
labels, datasets, repo paths, secrets, or private notes without explicit
approval for that exact data class and action.

## Goal

Create or refresh these artifacts under `.understudy/capture-evidence/`:

```text
workload-profile.md
harness.json
environment.json
metric.json
splits.json
baseline.json
```

Each artifact must include a creation timestamp, source refs or path refs, and
enough provenance for another agent to repeat the step without guessing.

## Required Checks

0. Confirm the workload profile.
   If `.understudy/capture-evidence/workload-profile.md` is missing or stale,
   route to [`../understand-workload/SKILL.md`](../understand-workload/SKILL.md)
   before building metrics. The profile should summarize the task purpose, data
   or trace distribution, prompt/request structure, code path, tool/action
   surface, output contract, failure taxonomy, and user-confirmed success
   criteria. It may cite local paths and hashes, but should not contain raw
   private payloads unless explicitly approved.
1. Attach the harness.
   Capture the local runner, command, fixture path, entrypoint, timeout,
   dependency notes, input schema, output schema, and validator invocation in
   `harness.json`.
2. Attach the environment.
   Record language/runtime versions, package manager, relevant lockfile status,
   model/provider route used by the incumbent, local hardware notes when
   relevant, and required env var names without values in `environment.json`.
   If the harness needs a local proxy, bootstrap repair, or provider-key
   presence check, route to the existing public setup skill for that recovery
   path before claiming the baseline is runnable.
   Model preflight: record whether the intended candidate/student model
   supports the workload's required capabilities (tool-calling,
   structured-output, vision, reasoning toggle) and whether its context window
   fits the workload's longest input. A context-window mismatch is a silent
   failure mode that later surfaces as confusing zero scores.
3. Confirm the scoring metric and validator (the load-bearing step).
   Write `metric.json` with the primary metric, pass/fail threshold,
   tie-breakers, validator, failure taxonomy, and `approved: true` only after a
   human confirms it. The metric is the real game: optimizing a *proxy* metric
   instead of the real validator is how prior runs scored 0/12. Record the
   validator `kind` and follow its rule:
   - `unit-test` / `golden` / `custom-command` — runs a deterministic check; the
     feedback is the assertion or diff that failed.
   - `schema` (e.g. Zod/JSON-schema `safeParse`) — keep `schema_pass` separate
     from `quality_pass`; a valid-shape, valid-enum output must not be failed
     merely for not matching a teacher trace verbatim. Two grounding checks a
     shape-only schema misses, both observed in real workloads: (a) **verbatim
     evidence** — when a field claims to quote a source (transcript, doc,
     log), verify the quote appears verbatim in the source; smaller open
     models hallucinate correctly-formatted quotes; (b) **conditional
     requireds** — when an optional signal (date, flag, risk) is present, its
     evidence subfield must be present too; models that include the signal but
     drop its evidence are the dominant residual failure and usually
     prompt-fixable.
   - `rubric` — a confirmed criteria list (each criterion: id, description,
     review type); auto-generated rubrics need human approval.
   - `llm-judge` — must debias position with a swapped two-pass score
     (`(r_ab − r_ba + 2) / 4`); never single-pass.
   - `human-review` — a blind, order-randomized packet; report judge-vs-human
     agreement separately from candidate preference.
   Whatever the kind, the metric must emit **natural-language feedback that
   diagnoses why** an output failed and what to change — not just a scalar.
   For API workflow benchmarks, record final-state correctness, policy
   compliance, data accuracy, endpoint discovery, required-write completion,
   forbidden-write avoidance, unnecessary calls/retries, schema validity, and
   recoverable errors as separate axes before collapsing to an overall score.
   If the metric or validator is unclear, stop and ask one concrete question.
4. Freeze splits.
   Write `splits.json` with train/dev/holdout names, sizes, source refs,
   deterministic split seed or frozen row ids, and an explicit "no holdout
   mutation" note.
5. Rerun the incumbent baseline.
   Use the frozen harness, metric, validator, and splits to rerun the current
   incumbent route. Write `baseline.json` with command, timestamp, split used,
   sample size, score, latency basis, cost basis if available, failures, and
   caveats. It must also include `harness_sha256`, `metric_sha256`, and
   `splits_sha256` for the exact artifacts used by the rerun.
   Record the per-row (or per-cluster) pass/fail set, not just an aggregate
   score, so the next step can see whether optimization **headroom** exists —
   i.e. rows the incumbent fails that a stronger model could fix.
   Record each per-row result as an `understudy.eval_result.v1` row — the
   required row format for eval evidence across every Understudy surface
   ([`schemas/understudy.eval_result.v1.schema.json`](../../schemas/understudy.eval_result.v1.schema.json)):
   `run_id`, `task_id`, `split`, `score` (0..1 or null — a 0 is a scored
   failure, never a missing value), `status` (`ok`/`error`/`skipped`/`unscored`),
   model, route, cost/tokens/latency when known, and a `provenance` block whose
   `harness_sha256`/`split_sha256` carry the same hash chain as `baseline.json`.

## Flow

Inspect the repo first to find where LLM calls happen and the current
model/provider/harness/eval state, then surface that inventory before building
anything. The inventory includes evidence that already exists outside the
repo: the Understudy desktop app exports benchmark comparison packets
(`understudy.fusion_benchmark_comparison.v1`) under `~/.understudy/exports/`,
each carrying `understudy.eval_result.v1` rows plus a packet-level
`provenance` block (rows in a sibling JSONL file; verify
`shasum -a 256 ` equals
`provenance.eval_results_sha256` before admitting it). Surface any verified
packets in the inventory so the developer isn't asked to re-measure what the
app already measured — the admission checklist lives in
[`../ramp-and-verify/SKILL.md`](../ramp-and-verify/SKILL.md). If the request/response path, dataset/trace shape, prompt purpose, or
success criteria are not already clear, route to
[`../understand-workload/SKILL.md`](../understand-workload/SKILL.md) first and
use its workload profile as the narrative source of truth. The deep inspection
checklist (call sites by SDK family, env vars, tracing, CI) and the eval-harness
discover-then-build playbook live in [`reference.md`](reference.md). For
cross-cutting objective/constraint framing, read
[`../understudy/reference.md`](../understudy/reference.md). For multi-turn /
tool-use / agentic workloads — both read-only search loops and multi-step
REST/API workflows that mutate state — route the eval to
[`../optimize-agentic-workload/SKILL.md`](../optimize-agentic-workload/SKILL.md)
instead of building a single-output harness; its state-mutating lens records
reset/seed state, API schemas, policy docs, request logs, and final-state
validators as part of the harness.

Start from the real local workload:

- app route, eval suite, trace export, benchmark fixture, prompt set, dataset,
  report, or existing runner;
- **no traces yet?** Start from a public benchmark — the golden-path ladder
  (AutomationBench, Harvey LAB) in
  [`references/public-benchmark-path.md`](references/public-benchmark-path.md)
  runs this same evidence loop against public fixtures;
- otherwise, create only a synthetic fixture and label it clearly as synthetic.

Do not optimize, tune prompts, choose replacement models, mutate splits, or
claim savings in this worker. Its job is to make the next validation step
possible.

If any artifact cannot be created, write down the missing input, the attempted
local command or inspection, and the next smallest action.

## Output Standard

End with:

- workload source inspected;
- workload profile status and whether the task understanding was confirmed;
- artifact paths created or refreshed;
- metric, validator, split boundary, and incumbent baseline status;
- result type: evidence-capture or blocked;
- one recommended next local command or action.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [understudylabs](https://github.com/understudylabs)
- **Source:** [understudylabs/understudy-agent-tools](https://github.com/understudylabs/understudy-agent-tools)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-understudylabs-understudy-agent-tools-capture-evidence
- Seller: https://agentstack.voostack.com/s/understudylabs
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.