Harnessgym

HarnessGym

📖 Documentation: harnessgym.com — the docs are built from docs/ with MkDocs Material and deployed on every push to main.

HarnessGym is an open-source framework for iterative agent harness improvement. It runs a coding agent on a hard task, reflects in the same session on which reusable harness artifact would have helped most, builds that single artifact under .harnessgym/, and starts the next iteration in a fresh session with the accumulated registry context.

The package is alpha software for developers evaluating agent workflows. The core package has no third-party runtime dependencies; runner backends shell out to the agent CLI you choose. Codex and Claude Code are supported, and the deterministic fake runner works offline for smoke tests and demos.

Install

python -m pip install harnessgym
harnessgym --help

For a source checkout with tests and the bundled examples:

git clone https://github.com/patrick-toulme/harnessgym.git
cd harnessgym
python -m pip install -e ".[dev]"
python -m pytest

Runner prerequisites:

• --runner exec uses the codex CLI, configurable with --codex-bin.
• --runner claude uses the claude CLI, configurable with --claude-bin.
• --runner fake is deterministic and does not require an agent account.
• Some examples below require additional local tooling such as NumPy, a C/C++ compiler, PyTorch, Triton, or access to a remote GPU.

Quickstart

Create or choose a task file in an existing workspace, then run HarnessGym against that workspace:

harnessgym run \
  --task task.md \
  --workspace . \
  --iterations 3 \
  --attempt-timeout 45m \
  --build-timeout 20m \
  --runner exec

For an offline smoke test from a source checkout, use the bundled numerical debugging demo:

harnessgym run \
  --task examples/numerical_debug_task/task.md \
  --workspace examples/numerical_debug_task \
  --iterations 2 \
  --attempt-timeout 10s \
  --build-timeout 10s \
  --runner fake

CLI

Optimization tasks can stop on an objective score instead of only status: solved:

harnessgym run \
  --task task.md \
  --workspace . \
  --iterations 2 \
  --attempt-timeout 5m \
  --build-timeout 5m \
  --runner exec \
  --stop-score 2.0 \
  --score-key best_ms

For time-boxed optimization tasks, use post-attempt scoring so HarnessGym independently verifies the workspace after every attempt, even if the runner process is killed before it updates result.json:

harnessgym run \
  --task task.md \
  --workspace . \
  --iterations 5 \
  --attempt-timeout 5m \
  --runner exec \
  --optimization-mode \
  --score-key best_cycles \
  --stop-score 1 \
  --post-attempt-command "python3 benchmark.py --json --mode final" \
  --post-attempt-score-key best_cycles

In optimization mode, summary.json records the baseline score, best score, relative improvement/reduction, and per-iteration harness usage. This avoids losing a real improvement when the attempt times out after mutating the workspace but before writing its final result JSON. By default, HarnessGym also checkpoints the best independently scored task workspace and restores it after the run, while leaving .harnessgym, .codex, .agents, and .claude activation state intact. Use --no-restore-best if you want to inspect the final attempted state instead.

Use --attempt-timeouts 5m,8m for per-iteration attempt budgets. Use --build-after-solve when you want a solved run to still reflect and build reusable harness artifacts for later replay experiments.

HarnessGym defaults to --harness-depth deep, which steers reflection/build toward capability-building harnesses instead of lightweight notes: multi-tool MCP servers with executable inspection and automation, such as compiler/IR analysis, assembly signals, benchmark history, variant sweeps, trace/layout tools, rollback-safe experiment ranking, and comprehensive self-tests. Use --harness-depth standard for smaller focused artifacts.

By default, generated artifacts are qualified before they are promoted to the next fresh attempt. After each build phase HarnessGym copies the pre-run task workspace into .harnessgym/runs//qualification/, copies in only the reusable .harnessgym/ artifact bundle, activates generated skills/MCPs there, and runs MCP self-tests. If qualification fails, HarnessGym sends the exact failure report back to the same runner session and allows --artifact-repair-attempts repair builds, default 1. Artifacts that still fail are marked quarantined in registry.json, hidden from future attempt prompts and activation, and preserved for repair evidence. Use --no-qualify-artifacts only when debugging the framework itself.

Task state has two modes:

# Compound task progress across iterations. This is the default.
harnessgym run --task task.md --workspace . --task-state continue

# Restore task files before each new iteration, while keeping accumulated .harnessgym artifacts.
harnessgym run --task task.md --workspace . --task-state reset

Use continue when you want the agent to keep improving the same working tree. Use reset when you want each fresh session to face the original task with only the generated harness artifacts carried forward.

Claude Code can be used instead of Codex:

harnessgym run \
  --task task.md \
  --workspace . \
  --iterations 3 \
  --attempt-timeout 45m \
  --build-timeout 20m \
  --runner claude

Useful Claude-specific flags:

• --claude-bin: path to the Claude Code executable, default claude.
• --claude-model: optional model alias or full model name, such as sonnet or opus.
• --claude-permission-mode: defaults to bypassPermissions for autonomous HarnessGym runs.
• --claude-max-budget-usd: optional per-phase spend cap for Claude Code print mode.
• --claude-extra-arg: repeatable escape hatch for newer Claude Code CLI flags.

Inline task text is also supported:

harnessgym run \
  --task-text "Fix the failing tests and verify them." \
  --workspace . \
  --iterations 2

Replay A/B comparisons are supported with compare. This copies a clean workspace template for each trial, runs attempt-only replays with and without a generated .harnessgym/ artifact bundle, optionally runs a final JSON benchmark command, and writes compare_report.json. Failed post commands are recorded as invalid worst-case outcomes with post_valid: false, post_invalid_reason, and invalid counts in the summary instead of being silently treated as missing scores. Harnessed trials also require at least one generated MCP tool to activate by default; otherwise the trial is marked invalid for comparison so a broken artifact bundle cannot be mistaken for a harness win. Use --no-require-active-harness for smoke tests that intentionally copy only docs/scripts.

HarnessGym records concrete generated MCP tools/call telemetry to .harnessgym/mcp_calls.jsonl and surfaces call counts, called tool names, and compact samples in compare_report.json. Claude records this through its MCP bridge. Codex records this through the Content-Length MCP telemetry proxy used by its generated MCP config and by the workspace-local helper python3 .harnessgym/runtime/mcp_call.py --server --tool --arguments ''. Use --require-harness-tool-use when you want a harnessed trial to count only if the agent actually called at least one generated MCP tool, not merely activated it.

harnessgym compare \
  --workspace-template examples/c_flash_attention_optimization_task \
  --task task.md \
  --artifact-source tmp/c_flash_attention_5iter_real/.harnessgym \
  --output-dir tmp/c_flash_attention_compare \
  --trials 2 \
  --attempt-timeout 5m \
  --runner exec \
  --post-command "python3 benchmark.py --json --mode final" \
  --post-score-key best_cycles

The repo also includes a committed tensor-layout harness bundle generated by a previous qualified run. It is kept outside the workspace template so plain trials do not see the harness artifacts:

harnessgym compare \
  --workspace-template examples/tensor_layout_pipeline_task \
  --task task.md \
  --artifact-source examples/tensor_layout_harness_artifacts/.harnessgym \
  --output-dir tmp/tensor_layout_claude_compare_final \
  --trials 1 \
  --iterations 1 \
  --attempt-timeout 5m \
  --runner claude \
  --score-key best_cycles \
  --stop-score 1 \
  --task-state continue \
  --post-command "python3 benchmark.py --json --mode final" \
  --post-score-key best_cycles \
  --post-timeout 2m \
  --require-harness-tool-use \
  --overwrite

The same replay is available as a helper script that first checks whether Claude Code can make model calls, then audits the resulting compare report for valid plain/harnessed trials, valid post scores, and the expected active tensor-plan MCP tool count. The helper defaults to REQUIRE_HARNESS_TOOL_USE=1, so a harnessed Claude replay must record at least one generated MCP tool call:

examples/tensor_layout_harness_artifacts/run_claude_compare.sh

For quota-window validation, set WAIT_FOR_CLAUDE=1 to retry the preflight for up to PREFLIGHT_MAX_WAIT_SECONDS before running the compare.

The repo also includes a real H100-backed Triton task and the generated HarnessGym artifact bundle from a verified run:

rm -rf tmp/h100_triton_real
mkdir -p tmp/h100_triton_real
cp -R examples/triton_rmsnorm_h100_task/. tmp/h100_triton_real/

HARNESSGYM_GPU_HOST= \
HARNESSGYM_GPU_PORT= \
HARNESSGYM_GPU_KEY=~/.ssh/id_ed25519 \
PYTHONPATH=src \
python3 -m harnessgym.cli run \
  --task tmp/h100_triton_real/task.md \
  --workspace tmp/h100_triton_real \
  --iterations 2 \
  --attempt-timeout 5m \
  --reflection-timeout 3m \
  --build-timeout 5m \
  --post-attempt-command 'python3 remote_h100.py --workspace h100_triton_real_post -- python3 verifier.py --json --mode final --warmup 10 --repeats 20' \
  --post-attempt-score-key best_us \
  --post-attempt-timeout 3m \
  --score-key best_us \
  --stop-score 90 \
  --optimization-mode \
  --runner exec

The recorded experiment improved the H100 final score from 150.016 us to a best checkpoint of 103.328 us, generated a skill plus MCP server, repaired a qualification failure, and confirmed 10 generated MCP calls in the next fresh Codex session. See docs/experiments/h100-triton-rmsnorm.md and examples/triton_rmsnorm_h100_harness_artifacts/.

A longer four-iteration follow-up started from that generated harness bundle and improved the independently verified H100 score from 142.848 us to 99.744 us. That run expanded the committed H100 MCP to 17 active tools, including remote health checks, exact/approximate SiLU source sweeps, joint source-plus-launch search, repeated scoring, and next-experiment ranking. See docs/experiments/h100-triton-rmsnorm-long.md.

Workflow

Each iteration:

1. Starts a fresh runner session for the attempt phase.
2. Attempts the primary task until solved, blocked, failed, or timed out.
3. Reflects in the same session on the highest-leverage missing skill, MCP server, verifier, fixture, script, docs, or tool.
4. Builds the single selected improvement in the same session.
5. Qualifies generated artifacts in a clean replay workspace, repairs failed harness artifacts in the same session when possible, and quarantines anything still failing.
6. Stores reusable, promoted artifacts under .harnessgym/ and records them in .harnessgym/registry.json.
7. Starts the next iteration with a fresh session and the accumulated registry context.

HarnessGym stops early when an iteration writes result.json with status: "solved" and verified: true or a passed verification object.

With --task-state reset, HarnessGym snapshots the non-.harnessgym workspace before the run and restores it before iterations 2..N. .harnessgym remains intact, so skills, MCP servers, tools, fixtures, and registry entries accumulate.

Artifacts

Run-specific logs are stored under:

.harnessgym/runs//iterations//

Each iteration directory contains:

• result.json
• .prompt.txt
• .stdout.txt
• .stderr.txt
• .transcript.txt

Reusable artifacts live under:

.harnessgym/skills/
.harnessgym/mcp/
.harnessgym/tools/
.harnessgym/verifiers/
.harnessgym/fixtures/
.harnessgym/tests/
.harnessgym/docs/
.harnessgym/scripts/

The registry file .harnessgym/registry.json is synchronized from those directories after each build phase.

Generated tooling is expected to carry its own tests. For MCP servers, manifests can declare self_test; true means HarnessGym will run the server command with --self-test, while a command string/list/object can point at a separate test runner. For numerical, kernel, compiler, or benchmark tooling, the generated tests should include known toy cases, tolerance checks, fixed-seed randomized cases, and fast/dev plus final/held-out modes when applicable.

Failed artifacts remain on disk but are quarantined in registry.json with qualification metadata. Quarantined artifacts are not advertised in attempt prompts and are not injected into runner MCP config; the qualification report path gives the next repair build the concrete failure evidence.

Agent Activation

HarnessGym keeps generated artifacts repo-local under .harnessgym/, then activates runner-native pieces before each fresh attempt:

• Skills from .harnessgym/skills//SKILL.md are symlinked into .agents/skills/ for Codex and .claude/skills/ for Claude Code.
• MCP manifests from .harnessgym/mcp/**/{mcp.json,server.json,harnessgym-mcp.json} are written into project-local .codex/config.toml.
• MCP servers are smoke-checked during activation with initialize and tools/list; failures are recorded as warnings instead of silently advertising a broken server.
• MCP servers must expose a non-empty tool inventory and pass a manifest self_test during activation before they are injected into a fresh runner session. self_test: true means HarnessGym runs the server command with --self-test; a command string/list/object can point at a separate test runner.
• The exec runner converts passed activation records into codex exec -c mcp_servers... overrides, because codex exec loads the user config by default and should not require mutating global Codex config for repo-local generated MCPs. Generated MCP servers are launched through harnessgym.mcp_telemetry_proxy, which preserves Content-Length MCP framing while logging tool calls. Activation also writes .harnessgym/runtime/mcp_call.py; Codex exec workers should use this helper when native MCP callables are not visible in the session instead of writing ad hoc JSON-RPC clients.
• The claude runner writes passed activation records into .harnessgym/claude_mcp_config.json, launches Claude Code with --strict-mcp-config --mcp-config , and grants generated MCP tools with Claude's server-level permission token --allowedTools=mcp__. Because Claude Code 2.1.133 sends newline-delimited MCP JSON over stdio while HarnessGym-generated Codex MCP servers use Content-Length framing, the runner wraps each generated server with harnessgym.claude_mcp_bridge. The bridge response timeout follows each manifest's MCP tool timeout.
• The Codex MCP telemetry proxy and Claude MCP bridge both log each generated MCP tools/call as compact JSONL under .harnessgym/mcp_calls.jsonl, including server name, tool name, argument-key summary, duration, status, and result size. Compare reports include mcp_telemetry, mcp_call_count, and mcp_called_tools.
• MCP servers with failed smoke checks, missing/failing self-tests, or empty tool inventories are not injected into runner attempts.
• Generated MCP servers are expected to use Content-Length framed stdio JSON-RPC. Newline-delimited JSON helpers may be useful as local scripts, but they will fail Codex and Claude MCP activation.
• Activation details,

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: patrick-toulme
• Source: patrick-toulme/harnessgym
• License: Apache-2.0
• Homepage: https://harnessgym.com

Install and usage instructions live in the source repository linked above.