# Finalize

> Score the best candidate on the held-out TEST split exactly once and seal the run. Use as the last evaluation step, after optimization stops. The run dir enforces the seal — a second finalize raises an error — so the headline number is produced once on data the optimizer never saw, the way an honest benchmark result must be.

- **Type:** Skill
- **Install:** `agentstack add skill-skillberry-ai-cap-evolve-finalize`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [skillberry-ai](https://agentstack.voostack.com/s/skillberry-ai)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [skillberry-ai](https://github.com/skillberry-ai)
- **Source:** https://github.com/skillberry-ai/cap-evolve/tree/main/skills/phases/finalize

## Install

```sh
agentstack add skill-skillberry-ai-cap-evolve-finalize
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# finalize — the one honest number

Optimization hill-climbs on val: every accept decision consumed the val split as
a tuning signal, so by the end of search val is *optimistic* — it has been
selected against. The number you **report** must come from data nothing was
tuned against. finalize scores the run's best candidate on the sealed `test`
split exactly once and writes `final.json`. That single number is the result.

## Inputs / outputs (manifest tokens)
- **needs:** `candidate` — the run's best candidate (selected on val).
- **provides:** `report` — `final.json` with the test reward, `stderr`, and pass^k.

## The seal (why "exactly once")
`cap_evolve` flips a `test_used` flag the first time test is scored; any second
attempt raises `TestSealError`. This is non-negotiable. The instant test informs
*any* choice — picking between two finalists, "double-checking" a low number,
re-running with more trials until it looks better — it stops being held out. Each
peek is a selection event that leaks information from test into the decision, and
the reported number drifts from an unbiased estimate toward an optimistic fit
metric. Selecting the best of several test scores is exactly the best-of-noise
inflation the gate exists to prevent, now applied to the one split that was
supposed to be clean. The seal makes that mistake impossible rather than merely
discouraged.

Corollary: **all selection happens before finalize.** Choose the single best
candidate on val, *then* finalize it. If you genuinely need to compare two
finalists, compare them on val (or a fresh held-out slice) — never on test.

## Dual-mode
This phase runs two ways from the **same** SKILL.md: standalone as the slash command `/cap-evolve:finalize` (the `argument-hint` shows its run.py args), and orchestrator-callable — `cap-evolve run` / the `orchestrate` skill invokes the same `scripts/run.py` headlessly and threads the run dir between phases.

## How to run
```
python scripts/run.py --run-dir .capevolve/run_XXXX --project .capevolve/project --n-trials 3
```
Use multiple trials so the headline carries an honest `stderr` and a pass^k
reliability figure, not a single noisy point. If the split was configured with no
holdout (test == train/val), the number is a *fit* metric and the report must
flag it as such — it is not a held-out result.

## What good vs bad looks like
- **Good:** one best candidate chosen on val, scored once on test with ≥3 trials;
  `final.json` carries reward + stderr + pass^k; test ≈ val (the val gain
  generalized).
- **Bad:** finalizing several candidates and keeping the best test score;
  re-running finalize "to confirm"; reporting a single-trial test number with no
  uncertainty; presenting a no-holdout fit number as if it were held out.

## References
- `references/concepts.md` — held-out sealing, why each peek biases the estimate,
  the selection-before-finalize rule, and how this maps to benchmark protocol,
  with sources.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [skillberry-ai](https://github.com/skillberry-ai)
- **Source:** [skillberry-ai/cap-evolve](https://github.com/skillberry-ai/cap-evolve)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-skillberry-ai-cap-evolve-finalize
- Seller: https://agentstack.voostack.com/s/skillberry-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.