# Harnessgym > Iterative agent harness improvement: run a coding agent on a hard task, generate the reusable tooling it was missing, qualify it, and replay fresh sessions with it activated. Works with Codex and Claude Code. - **Type:** MCP server - **Install:** `agentstack add mcp-patrick-toulme-harnessgym` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [patrick-toulme](https://agentstack.voostack.com/s/patrick-toulme) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [patrick-toulme](https://github.com/patrick-toulme) - **Source:** https://github.com/patrick-toulme/harnessgym - **Website:** https://harnessgym.com ## Install ```sh agentstack add mcp-patrick-toulme-harnessgym ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # HarnessGym 📖 **Documentation: [harnessgym.com](https://harnessgym.com)** — the docs are built from `docs/` with MkDocs Material and deployed on every push to `main`. HarnessGym is an open-source framework for iterative agent harness improvement. It runs a coding agent on a hard task, reflects in the same session on which reusable harness artifact would have helped most, builds that single artifact under `.harnessgym/`, and starts the next iteration in a fresh session with the accumulated registry context. The package is alpha software for developers evaluating agent workflows. The core package has no third-party runtime dependencies; runner backends shell out to the agent CLI you choose. Codex and Claude Code are supported, and the deterministic `fake` runner works offline for smoke tests and demos. ## Install ```bash python -m pip install harnessgym harnessgym --help ``` For a source checkout with tests and the bundled examples: ```bash git clone https://github.com/patrick-toulme/harnessgym.git cd harnessgym python -m pip install -e ".[dev]" python -m pytest ``` Runner prerequisites: - `--runner exec` uses the `codex` CLI, configurable with `--codex-bin`. - `--runner claude` uses the `claude` CLI, configurable with `--claude-bin`. - `--runner fake` is deterministic and does not require an agent account. - Some examples below require additional local tooling such as NumPy, a C/C++ compiler, PyTorch, Triton, or access to a remote GPU. ## Quickstart Create or choose a task file in an existing workspace, then run HarnessGym against that workspace: ```bash harnessgym run \ --task task.md \ --workspace . \ --iterations 3 \ --attempt-timeout 45m \ --build-timeout 20m \ --runner exec ``` For an offline smoke test from a source checkout, use the bundled numerical debugging demo: ```bash harnessgym run \ --task examples/numerical_debug_task/task.md \ --workspace examples/numerical_debug_task \ --iterations 2 \ --attempt-timeout 10s \ --build-timeout 10s \ --runner fake ``` ## CLI Optimization tasks can stop on an objective score instead of only `status: solved`: ```bash harnessgym run \ --task task.md \ --workspace . \ --iterations 2 \ --attempt-timeout 5m \ --build-timeout 5m \ --runner exec \ --stop-score 2.0 \ --score-key best_ms ``` For time-boxed optimization tasks, use post-attempt scoring so HarnessGym independently verifies the workspace after every attempt, even if the runner process is killed before it updates `result.json`: ```bash harnessgym run \ --task task.md \ --workspace . \ --iterations 5 \ --attempt-timeout 5m \ --runner exec \ --optimization-mode \ --score-key best_cycles \ --stop-score 1 \ --post-attempt-command "python3 benchmark.py --json --mode final" \ --post-attempt-score-key best_cycles ``` In optimization mode, `summary.json` records the baseline score, best score, relative improvement/reduction, and per-iteration harness usage. This avoids losing a real improvement when the attempt times out after mutating the workspace but before writing its final result JSON. By default, HarnessGym also checkpoints the best independently scored task workspace and restores it after the run, while leaving `.harnessgym`, `.codex`, `.agents`, and `.claude` activation state intact. Use `--no-restore-best` if you want to inspect the final attempted state instead. Use `--attempt-timeouts 5m,8m` for per-iteration attempt budgets. Use `--build-after-solve` when you want a solved run to still reflect and build reusable harness artifacts for later replay experiments. HarnessGym defaults to `--harness-depth deep`, which steers reflection/build toward capability-building harnesses instead of lightweight notes: multi-tool MCP servers with executable inspection and automation, such as compiler/IR analysis, assembly signals, benchmark history, variant sweeps, trace/layout tools, rollback-safe experiment ranking, and comprehensive self-tests. Use `--harness-depth standard` for smaller focused artifacts. By default, generated artifacts are qualified before they are promoted to the next fresh attempt. After each build phase HarnessGym copies the pre-run task workspace into `.harnessgym/runs//qualification/`, copies in only the reusable `.harnessgym/` artifact bundle, activates generated skills/MCPs there, and runs MCP self-tests. If qualification fails, HarnessGym sends the exact failure report back to the same runner session and allows `--artifact-repair-attempts` repair builds, default `1`. Artifacts that still fail are marked quarantined in `registry.json`, hidden from future attempt prompts and activation, and preserved for repair evidence. Use `--no-qualify-artifacts` only when debugging the framework itself. Task state has two modes: ```bash # Compound task progress across iterations. This is the default. harnessgym run --task task.md --workspace . --task-state continue # Restore task files before each new iteration, while keeping accumulated .harnessgym artifacts. harnessgym run --task task.md --workspace . --task-state reset ``` Use `continue` when you want the agent to keep improving the same working tree. Use `reset` when you want each fresh session to face the original task with only the generated harness artifacts carried forward. Claude Code can be used instead of Codex: ```bash harnessgym run \ --task task.md \ --workspace . \ --iterations 3 \ --attempt-timeout 45m \ --build-timeout 20m \ --runner claude ``` Useful Claude-specific flags: - `--claude-bin`: path to the Claude Code executable, default `claude`. - `--claude-model`: optional model alias or full model name, such as `sonnet` or `opus`. - `--claude-permission-mode`: defaults to `bypassPermissions` for autonomous HarnessGym runs. - `--claude-max-budget-usd`: optional per-phase spend cap for Claude Code print mode. - `--claude-extra-arg`: repeatable escape hatch for newer Claude Code CLI flags. Inline task text is also supported: ```bash harnessgym run \ --task-text "Fix the failing tests and verify them." \ --workspace . \ --iterations 2 ``` Replay A/B comparisons are supported with `compare`. This copies a clean workspace template for each trial, runs attempt-only replays with and without a generated `.harnessgym/` artifact bundle, optionally runs a final JSON benchmark command, and writes `compare_report.json`. Failed post commands are recorded as invalid worst-case outcomes with `post_valid: false`, `post_invalid_reason`, and invalid counts in the summary instead of being silently treated as missing scores. Harnessed trials also require at least one generated MCP tool to activate by default; otherwise the trial is marked invalid for comparison so a broken artifact bundle cannot be mistaken for a harness win. Use `--no-require-active-harness` for smoke tests that intentionally copy only docs/scripts. HarnessGym records concrete generated MCP `tools/call` telemetry to `.harnessgym/mcp_calls.jsonl` and surfaces call counts, called tool names, and compact samples in `compare_report.json`. Claude records this through its MCP bridge. Codex records this through the Content-Length MCP telemetry proxy used by its generated MCP config and by the workspace-local helper `python3 .harnessgym/runtime/mcp_call.py --server --tool --arguments ''`. Use `--require-harness-tool-use` when you want a harnessed trial to count only if the agent actually called at least one generated MCP tool, not merely activated it. ```bash harnessgym compare \ --workspace-template examples/c_flash_attention_optimization_task \ --task task.md \ --artifact-source tmp/c_flash_attention_5iter_real/.harnessgym \ --output-dir tmp/c_flash_attention_compare \ --trials 2 \ --attempt-timeout 5m \ --runner exec \ --post-command "python3 benchmark.py --json --mode final" \ --post-score-key best_cycles ``` The repo also includes a committed tensor-layout harness bundle generated by a previous qualified run. It is kept outside the workspace template so plain trials do not see the harness artifacts: ```bash harnessgym compare \ --workspace-template examples/tensor_layout_pipeline_task \ --task task.md \ --artifact-source examples/tensor_layout_harness_artifacts/.harnessgym \ --output-dir tmp/tensor_layout_claude_compare_final \ --trials 1 \ --iterations 1 \ --attempt-timeout 5m \ --runner claude \ --score-key best_cycles \ --stop-score 1 \ --task-state continue \ --post-command "python3 benchmark.py --json --mode final" \ --post-score-key best_cycles \ --post-timeout 2m \ --require-harness-tool-use \ --overwrite ``` The same replay is available as a helper script that first checks whether Claude Code can make model calls, then audits the resulting compare report for valid plain/harnessed trials, valid post scores, and the expected active tensor-plan MCP tool count. The helper defaults to `REQUIRE_HARNESS_TOOL_USE=1`, so a harnessed Claude replay must record at least one generated MCP tool call: ```bash examples/tensor_layout_harness_artifacts/run_claude_compare.sh ``` For quota-window validation, set `WAIT_FOR_CLAUDE=1` to retry the preflight for up to `PREFLIGHT_MAX_WAIT_SECONDS` before running the compare. The repo also includes a real H100-backed Triton task and the generated HarnessGym artifact bundle from a verified run: ```bash rm -rf tmp/h100_triton_real mkdir -p tmp/h100_triton_real cp -R examples/triton_rmsnorm_h100_task/. tmp/h100_triton_real/ HARNESSGYM_GPU_HOST= \ HARNESSGYM_GPU_PORT= \ HARNESSGYM_GPU_KEY=~/.ssh/id_ed25519 \ PYTHONPATH=src \ python3 -m harnessgym.cli run \ --task tmp/h100_triton_real/task.md \ --workspace tmp/h100_triton_real \ --iterations 2 \ --attempt-timeout 5m \ --reflection-timeout 3m \ --build-timeout 5m \ --post-attempt-command 'python3 remote_h100.py --workspace h100_triton_real_post -- python3 verifier.py --json --mode final --warmup 10 --repeats 20' \ --post-attempt-score-key best_us \ --post-attempt-timeout 3m \ --score-key best_us \ --stop-score 90 \ --optimization-mode \ --runner exec ``` The recorded experiment improved the H100 final score from `150.016 us` to a best checkpoint of `103.328 us`, generated a skill plus MCP server, repaired a qualification failure, and confirmed 10 generated MCP calls in the next fresh Codex session. See `docs/experiments/h100-triton-rmsnorm.md` and `examples/triton_rmsnorm_h100_harness_artifacts/`. A longer four-iteration follow-up started from that generated harness bundle and improved the independently verified H100 score from `142.848 us` to `99.744 us`. That run expanded the committed H100 MCP to 17 active tools, including remote health checks, exact/approximate SiLU source sweeps, joint source-plus-launch search, repeated scoring, and next-experiment ranking. See `docs/experiments/h100-triton-rmsnorm-long.md`. ## Workflow Each iteration: 1. Starts a fresh runner session for the attempt phase. 2. Attempts the primary task until solved, blocked, failed, or timed out. 3. Reflects in the same session on the highest-leverage missing skill, MCP server, verifier, fixture, script, docs, or tool. 4. Builds the single selected improvement in the same session. 5. Qualifies generated artifacts in a clean replay workspace, repairs failed harness artifacts in the same session when possible, and quarantines anything still failing. 6. Stores reusable, promoted artifacts under `.harnessgym/` and records them in `.harnessgym/registry.json`. 7. Starts the next iteration with a fresh session and the accumulated registry context. HarnessGym stops early when an iteration writes `result.json` with `status: "solved"` and `verified: true` or a passed verification object. With `--task-state reset`, HarnessGym snapshots the non-`.harnessgym` workspace before the run and restores it before iterations 2..N. `.harnessgym` remains intact, so skills, MCP servers, tools, fixtures, and registry entries accumulate. ## Artifacts Run-specific logs are stored under: ```text .harnessgym/runs//iterations// ``` Each iteration directory contains: - `result.json` - `.prompt.txt` - `.stdout.txt` - `.stderr.txt` - `.transcript.txt` Reusable artifacts live under: ```text .harnessgym/skills/ .harnessgym/mcp/ .harnessgym/tools/ .harnessgym/verifiers/ .harnessgym/fixtures/ .harnessgym/tests/ .harnessgym/docs/ .harnessgym/scripts/ ``` The registry file `.harnessgym/registry.json` is synchronized from those directories after each build phase. Generated tooling is expected to carry its own tests. For MCP servers, manifests can declare `self_test`; `true` means HarnessGym will run the server command with `--self-test`, while a command string/list/object can point at a separate test runner. For numerical, kernel, compiler, or benchmark tooling, the generated tests should include known toy cases, tolerance checks, fixed-seed randomized cases, and fast/dev plus final/held-out modes when applicable. Failed artifacts remain on disk but are quarantined in `registry.json` with qualification metadata. Quarantined artifacts are not advertised in attempt prompts and are not injected into runner MCP config; the qualification report path gives the next repair build the concrete failure evidence. ## Agent Activation HarnessGym keeps generated artifacts repo-local under `.harnessgym/`, then activates runner-native pieces before each fresh attempt: - Skills from `.harnessgym/skills//SKILL.md` are symlinked into `.agents/skills/` for Codex and `.claude/skills/` for Claude Code. - MCP manifests from `.harnessgym/mcp/**/{mcp.json,server.json,harnessgym-mcp.json}` are written into project-local `.codex/config.toml`. - MCP servers are smoke-checked during activation with `initialize` and `tools/list`; failures are recorded as warnings instead of silently advertising a broken server. - MCP servers must expose a non-empty tool inventory and pass a manifest `self_test` during activation before they are injected into a fresh runner session. `self_test: true` means HarnessGym runs the server command with `--self-test`; a command string/list/object can point at a separate test runner. - The `exec` runner converts passed activation records into `codex exec -c mcp_servers...` overrides, because `codex exec` loads the user config by default and should not require mutating global Codex config for repo-local generated MCPs. Generated MCP servers are launched through `harnessgym.mcp_telemetry_proxy`, which preserves Content-Length MCP framing while logging tool calls. Activation also writes `.harnessgym/runtime/mcp_call.py`; Codex exec workers should use this helper when native MCP callables are not visible in the session instead of writing ad hoc JSON-RPC clients. - The `claude` runner writes passed activation records into `.harnessgym/claude_mcp_config.json`, launches Claude Code with `--strict-mcp-config --mcp-config `, and grants generated MCP tools with Claude's server-level permission token `--allowedTools=mcp__`. Because Claude Code 2.1.133 sends newline-delimited MCP JSON over stdio while HarnessGym-generated Codex MCP servers use Content-Length framing, the runner wraps each generated server with `harnessgym.claude_mcp_bridge`. The bridge response timeout follows each manifest's MCP tool timeout. - The Codex MCP telemetry proxy and Claude MCP bridge both log each generated MCP `tools/call` as compact JSONL under `.harnessgym/mcp_calls.jsonl`, including server name, tool name, argument-key summary, duration, status, and result size. Compare reports include `mcp_telemetry`, `mcp_call_count`, and `mcp_called_tools`. - MCP servers with failed smoke checks, missing/failing self-tests, or empty tool inventories are not injected into runner attempts. - Generated MCP servers are expected to use Content-Length framed stdio JSON-RPC. Newline-delimited JSON helpers may be useful as local scripts, but they will fail Codex and Claude MCP activation. - Activation details, … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [patrick-toulme](https://github.com/patrick-toulme) - **Source:** [patrick-toulme/harnessgym](https://github.com/patrick-toulme/harnessgym) - **License:** Apache-2.0 - **Homepage:** https://harnessgym.com Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-patrick-toulme-harnessgym - Seller: https://agentstack.voostack.com/s/patrick-toulme - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.