Intake

intake — collect inputs, scaffold the project

The first phase. Its job is to turn a vague wish ("make this agent better at X") into a concrete, runnable project: a filled capevolve.yaml, an adapter ready to implement, and every NEEDED input resolved before any budget is spent. Intake is cheap; a botched intake is not — an unresolved input discovered three phases later means a wasted optimization run and a meaningless number.

Inputs / outputs (manifest tokens)

• needs: (nothing) — intake is the pipeline entry point.
• provides: project (the scaffolded .capevolve/project/) and tasks (the

evaluation dataset, resolved either to a path or to the adapter's tasks()).

Downstream, implement-and-check consumes project; baseline consumes project + tasks. If intake under-delivers either token, the hard gate in implement-and-check fails loudly rather than silently optimizing against a stub.

What it does

1. Interview (driven by this SKILL.md): pick the capability skill (what is

optimized), the optimizer (which coding agent proposes edits), the algorithm (the search loop), the dataset, the splits, and the budget.

1. Scaffold .capevolve/project/ from the template (scripts/run.py):

adapter stub, inputs/, capevolve.yaml, PROJECT.md, and the optimizer-prompt template optimizer/INSTRUCTIONS.md (the whole templates/project/ tree is copytree'd verbatim, so this file is already in place — confirm it exists).

1. Resolve inputs per inputs/INPUTS.md — the contract below.
2. Wire trajectories + scoring into the adapter. From the trajectories path

and metric extraction / scoring source inputs:

• implement adapter.trajectories(split) to RETURN the runner's native trajectory

directory for the last eval of split (any structure/format; it is copied verbatim into the optimizer's ./trajectories/). Return None only if there is genuinely no separate native store (cap-evolve then falls back to its per-rollout JSON) — note that choice in PROJECT.md.

• implement score() to extract the OBJECTIVE metric from a rollout, matching the

benchmark's own scoring source; verify it reproduces the benchmark's number.

• Make score()'s feedback ARGUMENT-LEVEL — it IS the learning signal. A

tool-name-only signal ("action X was wrong") is too coarse for the optimizer to localize a fix; it pattern-matches to prose rules and the run plateaus. For EACH failing check, the feedback must point at the specific argument/value/step that was wrong: name the wrong ARGUMENT key and the agent's OWN wrong value (not the gold value), name the wrong target id, and for communication/omission misses name the value or field the agent failed to state when it is derivable from the agent's own state (e.g. an un-stated computed total). This is gold-SAFE: derive everything from the agent's own messages/tool-calls/observed state (and the user's own profile/db state the agent saw) — use the gold record ONLY to learn WHICH check/argument failed (key names are safe; gold VALUES must never be read or printed). When a piece is not safely derivable, fall back to the coarser tool-name message. Keep score() deterministic (the check gate requires it).

1. **Author the optimizer instructions for THIS benchmark — SCOPED TO THE SELECTED

CAPABILITIES. Customize the scaffolded .capevolve/project/optimizer/INSTRUCTIONS.md. Keep the {{...}} placeholders intact ({{FOCUS_SUMMARY}}, {{FAILURES}}, {{CAP_BRIEF}}, {{ALGO_BRIEF}}, {{BENCH_REPO}} — the harness fills them per iteration). Keep the authored static guidance short on meta-narration, explicit and DEMANDING on iteration depth, and make it capability-scoped**: include guidance, skill references, and edit-space ONLY for the capabilities actually listed in capevolve.yaml: capabilities.

• DEPTH MANDATE — address ALL failure clusters each iteration. The authored

instructions must demand a substantial multi-root-cause pass. Produce this target snippet: > "Each iteration is a substantial, multi-root-cause pass. Diagnose ALL clusters > and fix as many as possible in ONE candidate — improve multiple tools' code, > validation, and return values/errors; add new tools; sharpen many tool docs; > and fix the prompt — together. Scope each fix to protect passing tasks; do NOT > trade breadth for caution. A single small edit is an under-used iteration."

• State the GOAL up front: maximize the eval score — make the largest

improvement you can this iteration, grounded in the trajectories.

• The authored INSTRUCTIONS MUST encode all three of these (generic, capability-scoped):

1. STEP-0 reading mandate. Before diagnosing, the optimizer must READ

./guidance//SKILL.md (for EACH selected capability) and the optimizer features reference under ./guidance/optimizer/. State this as an explicit first step.

1. The EXISTING-tool-code mandate (when tools is selected). Demand: convert

violated textual rules into in-code checks across MANY EXISTING tool bodies — most violated rules govern a tool that already exists, so the fix is an in-body guard there, not a new tool. State plainly: a docstring-only iteration (or one that only adds a single new tool + rewords docstrings, leaving rules as prose) is under-used.

1. The explicit TWO-PHASE subagent pattern. Require: Phase 1 — diagnose

fan-out (one read-only subagent per trajectory-group → tight issue list; main dedups into clusters); Phase 2 — implement fan-out (one edit-subagent per ISSUE, each in its own worktree, each PREFERRING to edit the EXISTING tool's code body to enforce its rule); then the main agent MERGES all edits into ONE candidate. Point at ./guidance/optimizer/.md for the agent's concrete trigger phrasing.

1. The NON-OVERFITTING guardrail. Demand that every prompt/tool edit encode

a GENERAL rule/policy/validation that generalizes across the whole class of inputs — NEVER hardcode a specific task's id/value/date/name/answer. A guard must fire on the general condition (e.g. "id not in the user's profile"), not match a literal value (NOT if id == ""). A literal special-case that only helps one task is forbidden — it overfits, fails the held-out gate, and hurts other tasks. Per-task specifics are for understanding the failure CLASS only; the fix must be general.

1. EXPLOIT ground-truth/eval present in the trajectories (diagnosis only).

Tell the optimizer that when ./trajectories/ include ground-truth / expected actions / a reward breakdown, it should USE them during diagnosis to localize the exact defect (expected vs actual action/argument/value) — and if not present, infer from the traces + feedback. State plainly that ground truth informs the failure class only; the resulting edit must still be GENERAL (guardrail 4) and never copy a gold value.

• Capability-scoping (the key rule): reference ./guidance//SKILL.md

and present the editable artifacts for the selected caps only. If only tools is selected, do NOT include any prompt-editing guidance, do NOT reference the system-prompt skill, and do NOT present the prompt/policy file as editable. If only system-prompt is selected, do not surface the tools file as editable. Each capability's "What you can change here" lives in its ./guidance//SKILL.md — point the optimizer there rather than restating it.

• Always include (capability-agnostic): READ the four cross-iteration files in

the working dir FIRST — ./LEDGER.md (framework facts: each iteration's outcome + tasks broken/fixed), the whole ./JOURNAL.md (the optimizer's append-only handover across the run), and ./RUNMAP.md + ./prior_iterations// (every prior iteration's PROCESS.md + capability diff) — and never re-propose an approach the journal/ledger shows was rejected as implemented (a better-designed version may still work — not a permanent ban); READ and USE the diagnose skill ./guidance/diagnose/SKILL.md (incl. its KNOWLEDGE / BEHAVIORAL / CAPABILITY-GAP tags), the optimizer features reference ./guidance/optimizer/.md, this step's ./trajectories/, and any ./guidance/sources/ data-model files. Each iteration the optimizer MUST: fill ./PROCESS.md (the required explainability template — ranked issues + tags, every edit + class, verify-the-fix, subagents/features used, what to preserve, what was skipped) and APPEND its entry to ./JOURNAL.md below the marker (what was tried / worked / regressed / refuted / plateau-signal / focus-next). Ship MULTIPLE edit classes and ADD a new code-bearing tool whenever a CAPABILITY-GAP/stall cluster is present.

1. Set the spec keys in capevolve.yaml:

• runner_repo_path — the benchmark/runner source, surfaced read-only to the

optimizer.

• optimizer_instructions_file — point at the customized template (default

optimizer/INSTRUCTIONS.md).

• capability_sources — the benchmark's data-model / types source files that the

tools import (resolved relative to the project dir; copied verbatim into the optimizer's ./guidance/sources/), so the optimizer can write correct code against the real types. Set this whenever a selected capability's code imports a shared types/data-model module; leave the default [] when there is none.

Ask-the-user-if-missing (mandatory — the core discipline)

Read inputs/INPUTS.md. It classifies every input as NEEDED or RECOMMENDED. For each NEEDED input that is not already present:

> ASK THE USER. Quote (a) the exact path where it is expected, (b) the > command or option that produces it, and (c) any alternatives. Then wait.

Never invent a NEEDED input. Fabricating a dataset, a scorer, or a gold answer does not unblock the run — it produces a number that measures nothing and hides that fact. A missing tasks file is a question for the user, not a gap for you to paper over. This is the single most important behavior of this phase.

RECOMMENDED inputs have sane defaults and may be skipped — but log every skip in PROJECT.md (e.g. "num_trials defaulted to 1 — scores will be single-trial, so the significance gate will correctly reject marginal gains"), so the honesty cost of each default is visible at report time.

Block on a missing NEEDED input (never fabricate)

The action when a NEEDED input is absent depends on the run mode:

• Interactive / chat mode — ASK THE USER and wait: quote what is needed, why

it is needed (what breaks without it), and how to provide it (the exact path / command / option / alternatives from INPUTS.md). Do not proceed past the missing input.

• Non-interactive mode (cap-evolve run / orchestrate, no human to ask) — do NOT

fabricate. WRITE a clearly delimited section into PROJECT.md: BLOCKED: — why it is needed — how to provide it, then STOP with a non-zero exit. A blocked-but-honest stop is correct; a green run on a guessed input is not.

This extends the ask-if-missing discipline above — it is the same rule, with an explicit non-interactive fallback so a headless run fails loud and recorded instead of silently inventing a dataset, scorer, trajectories path, or scoring source.

Why a contract, not a guess

The classic failure mode of "auto-optimize my agent" tooling is to start running with whatever it can find and backfill assumptions. That yields a green run and a worthless result. Splitting inputs into NEEDED (blocking → ask) vs RECOMMENDED (default → log) makes the only legitimate way to proceed-without-an-input an explicit, recorded default — never a silent fabrication. Treat INPUTS.md as the spec; this SKILL.md is just the procedure for honoring it.

Dual-mode

This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:intake (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.

How to run

python scripts/run.py --base .capevolve        # scaffold .capevolve/project

The script is purely mechanical: it copies the template and prints the next steps. The judgment — interviewing, choosing components, and running the ask-if-missing loop — is yours, driven by this SKILL.md and inputs/INPUTS.md.

Then implement adapters/adapter.py, fill capevolve.yaml, and proceed to implement-and-check. Together, intake → implement-and-check is the full integration: scaffold → implement the 4 adapter methods → cap-evolve check green, before any budget is spent. The using-agent (e.g. the chosen optimizer) can run this whole integration autonomously.

> Worked example (onboard a new benchmark from a prompt): see examples/ for > an end-to-end onboarding. The intake/onboarding step installs the benchmark > (clones + installs it) and the optimizer agent wires the adapter from the > stub until cap-evolve check passes, then optimizes the selected capability. The > example's setup.sh is the executable transcript of that onboarding; run.sh > runs the full optimization with the live dashboard.

What good vs bad intake looks like

• Good: every NEEDED input resolved to a real path or "adapter"; splits and

budget chosen deliberately; each defaulted RECOMMENDED input logged in PROJECT.md; capevolve.yaml fully filled; the user answered every blocking question before the scaffold was declared done.

• Bad: a tasks file that "looked plausible" was synthesized; the scorer leaks

the gold answer into feedback; test == train with no note; budget left at a default that cannot possibly find a gain; the run proceeded past a missing NEEDED input "to keep moving".

• Bad: authored INSTRUCTIONS that let an iteration pass by adding one tool +

rewording docstrings (leaving violated rules as prose) — or that omit the STEP-0 reading mandate, the existing-tool-code mandate, or the explicit two-phase (diagnose fan-out → implement fan-out → merge) subagent pattern.

References

• references/concepts.md — the inputs contract, NEEDED vs RECOMMENDED

rationale, the four adapter methods, and split/trial/budget guidance with sources.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: skillberry-ai
• Source: skillberry-ai/cap-evolve
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.