Compare Trajectories

Compare Trajectories

Use this worker when [../compare-model-sweep/SKILL.md](../compare-model-sweep/SKILL.md) has already told you that model A and model B score differently on a frozen eval, and the next question — the one that decides whether to spend RL budget — is HOW their trajectories differ, step by step, on the same tasks. A pass-rate delta is a scalar; it cannot tell you whether the stronger model wins because it tries longer, because it knew the right endpoint, or because the weaker model's tool-calls were malformed. Those three answers point at three different fixes — RL, retrieval/world-knowledge, or decoding/prompt — and only a per-task trajectory diff separates them.

This skill aligns two (or more) run exports by task id, builds the outcome-delta matrix, computes behavioral divergence on shared tasks, classifies each reachable-gap task, and counts how many clean warm-start trajectories the comparison yields. It reads trajectory JSON that already exists on disk; it does not run models. To produce the trajectories, run [../run-local-model-lab/SKILL.md](../run-local-model-lab/SKILL.md) or [../compare-model-sweep/SKILL.md](../compare-model-sweep/SKILL.md) first.

Safety Gates

• Local-first, no upload. Trajectory exports contain full message bodies and

tool I/O. Keep the diff, the matrix, and the report under .understudy/; do not paste trajectories or task content into external services.

• Flag small N. A reachable-gap set of 3 tasks is an anecdote, not a

capability claim. State the shared-task count and the per-class count on every verdict, and refuse to generalize below a stated floor (default N/`.

Repeat-replay stability

A single run per model can mistake sampling noise for a behavioral gap. Before classifying gaps — and always before a candidate graduates toward live traffic — re-run the candidate on the same frozen rows N times (default 3): all repeats match → stable; some → borderline; none → stochastic (exclude from gap classification and the warm-start yield). Above ~5% unstable, fix decoding (temperature, seed) or downgrade verdicts to directional-only. The dispositions feed the pre-ramp gate in [../ramp-and-verify/SKILL.md](../ramp-and-verify/SKILL.md) (stable → serve, borderline → shadow, stochastic → incumbent fallback).

Token-logprob lens

When two same-family runs (e.g. quantized vs full-precision) diverge on the same row and you need the token-level why — which token the weaker model got wrong and how confident it was — see [references/logprob-lens.md](references/logprob-lens.md). It documents the renderer in [scripts/render-logprob-compare.mjs](scripts/render-logprob-compare.mjs) and the private logprob-sidecar artifacts it requires; nothing public in this repo produces those artifacts yet, so treat it as an optional lens, not a step.

Output Standard

End with the diff path and: the two run ids + harness/tool-access mode; shared-task count and N caveat; the outcome matrix (A✓B✓ / A✗B✗ / A✗B✓ reachable-gap / A✓B✗ regressions) with named id lists; the behavioral divergence summary (steps-to-done, finish-reason buckets, recovery events, median first-divergence step); the gap-class breakdown (persistence/recovery vs knowledge vs format/parsing over the reachable-gap set); the warm-start yield (clean B-passes of A-failures); and an explicit "is this gap RL-shaped?" verdict — RL-shaped when persistence/recovery dominates, not RL-shaped when knowledge or format dominates — stated as a hypothesis with the N caveat and any holdout segregation. If regressions (A✓B✗) exist, surface them; a stronger model that loses tasks is not strictly better.

References

• [../compare-model-sweep/SKILL.md](../compare-model-sweep/SKILL.md) — the

scalar Pareto this skill complements; run it first for the frontier.

• [../run-local-model-lab/SKILL.md](../run-local-model-lab/SKILL.md) — produces

the local-model trajectory exports this skill diffs.

• [../ladder/SKILL.md](../ladder/SKILL.md) — quick local-vs-frontier feel check before committing to a full trajectory diff.
• [../curate-trajectories/SKILL.md](../curate-trajectories/SKILL.md) — supplies

hash-stamped, holdout-segregated selections to diff.

• [../local-distillation-lab/SKILL.md](../local-distillation-lab/SKILL.md) —

consumes the warm-start yield (clean B-passes of A-failures).

• [../understand-workload/SKILL.md](../understand-workload/SKILL.md) — adjacent

vocabulary for tool-class and step decomposition.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: understudylabs
• Source: understudylabs/understudy-agent-tools
• License: MIT

Install and usage instructions live in the source repository linked above.