Harness Trace

ABOUTME: Structured execution trace capture and token baselining for claude-forge

ABOUTME: Extracts traces from session JSONL, counts tokens with tiktoken, generates baselines

---

name: harness-trace description: "Execution trace capture and token baselining. Use when user says trace, extract trace, count tokens, token baseline, or /harness-trace. Not for harness optimization (use harness-mechanic)." compatibility: "Python >=3.11, uv" ---

Harness Trace

Structured execution trace capture and token baselining for the claude-forge harness. Based on Meta-Harness (arxiv 2603.28052): measure before optimizing.

Quick Start

cd skills/harness-trace

# Install
uv sync

# Extract traces from a session JSONL
uv run -- harness-trace extract ~/.claude/projects/PROJECT/SESSION.jsonl

# Count tokens in harness files
uv run -- harness-trace count-tokens rules/

# Generate full baseline
uv run -- harness-trace baseline --base-dir /path/to/claude-forge

Commands

| Command | Purpose | Output | |---------|---------|--------| | extract | Parse session into trace JSONL | Trace entries (stdout or file) | | count-tokens | Count tokens per file | Token counts per file | | baseline --base-dir | Full harness token baseline | TSV with tier classification |

Extraction strategy

extract uses two signal sources, in order of precision:

1. Tool-use blocks (primary, high precision). Walks every tool_use block in the assistant messages. The mapping below converts concrete tool invocations into trace entries with confidence:

| Tool / pattern | Emits | |----------------|-------| | Agent(subagent_type=*-reviewer) | ROUTE + REVIEW | | Agent(subagent_type=*) (non-reviewer) | ROUTE | | Bash matching pytest\|make test\|go test\|npm test\|cargo test\|rspec | VERIFY | | Bash matching ast-grep\|sg --pattern | BLAST_RADIUS | | Edit / Write / MultiEdit | counted into IMPLEMENT aggregate | | WebFetch to arxiv\|docs | RESEARCH | | AskUserQuestion | counted into safety_compliance.hitl_gates_hit |

1. Text regex (fallback). Applied only to messages whose tool stream did not already surface the step, and only for steps that are typically pure text (SCORE, FIX, LOOP, UAT, plus VERIFY/REVIEW when no tool signal exists). This avoids the v1 problem where prose like "tests pass" in a chat triggered fake VERIFY entries.

SUMMARY is always synthesized last from JSONL aggregates: duration_min (lastts - firstts), files_changed (Edit+Write count), and the 6-dimension metrics object (paper §5.2.1). No SUMMARY is emitted for sessions with zero other entries.

Trace Schema (v2)

One JSONL line per orchestrator step. v2 adds rejected_alternatives (top-level, attaches to any step), two new step types (PERMISSION_EVENT, ROUTE), and an end-of-task metrics mini-report inside SUMMARY. Based on "Code as Agent Harness" (arxiv 2605.18747 §3.5.1, §5.2.1).

Step taxonomy. step mixes two kinds intentionally:

• Lifecycle phases (REFINE, RESEARCH, ..., SUMMARY): at most once per round, sequential.
• Cross-cutting events (PERMISSION_EVENT, ROUTE): fire opportunistically, possibly many per phase. Consumers that care about the distinction should filter by step name.

{"v":2,"session":"slug","ts":"ISO8601","step":"STEP","data":{...},"rejected_alternatives":[...]}

| Step | Data fields | |------|-------------| | REFINE | ambiguitiesfound, questionsasked | | RESEARCH | complexity, sourcesconsulted | | LOCALIZE | filesplanned, filesproposed, filesactuallychanged, precision, recall, mismatches | | REPRODUCE | script, failsbeforefix, passesafterfix | | IMPLEMENT | agents, fileschanged, subtaskcount, localizationprecision | | DRIFTCHECK | subtaskid, verdict, deviations | | VERIFY | testspass, lintclean, buildok, retries, reproductionconfirmed | | REVIEW | agents, findings (CRITICAL/MAJOR/MINOR), reviewvalidity | | FIX | findingsaddressed, deviations | | BLASTRADIUS | triggered, triggerreason, filesscanned, contradictions | | SCORE | score, threshold, gate | | LOOP | round, totalrounds, exitreason | | UAT | performed, items, passed, failed | | SUMMARY | tokensin/out, model, duration, finalscore, metrics (v2) | | PERMISSIONEVENT (v2) | tool, action, outcome (granted/denied/deniedbysettings/autoapproved/error/timeout/bypassed), reason. Callers must redact secrets in action. | | ROUTE (v2) | router, target, alternativesconsidered, decision_basis |

v2 cross-cutting fields

• rejected_alternatives (top-level on TraceEntry): list of {description, reason_rejected, cost_estimate?}. Captures paths the agent considered but discarded. Useful for diagnosing decision quality without rerunning the agent.

v2 SUMMARY.metrics (6 harness dimensions, paper §5.2.1)

Each dimension is an optional dict; populate only what was measured. None = not measured (do not infer 0).

| Dimension | Suggested keys | |-----------|----------------| | trajectoryefficiency | toolcalls, tokens, edits, executions, activemin (gap-clamped working time, paired with SUMMARY.durationmin calendar span) | | verificationstrength | testcoveragepct, oraclescount, falseacceptrate | | recoveryability | failures, recovered, escalations | | stateconsistency | memoryreposynced, driftdetected | | safetycompliance | permissiondenials, hitlgateshit, sandboxused | | replayability | fulltracecaptured, artifacts_persisted |

Token Baseline TSV

file	tokens	words	lines	tier	loaded
rules/orchestrator-protocol.md	1247	450	67	rule	always
skills/golang/SKILL.md	2890	1050	145	skill	on-demand

Tiers: rule (always-on), skill/skill-ref (on-demand), agent (on-demand), config (always-on).

Architecture

src/harness_trace/
  models.py          trace JSONL (heuristic parser)
  token_counter.py  <- tiktoken counting, directory scanning, TSV generation
  cli.py            <- CLI entry point (extract, count-tokens, baseline)
tests/              <- pytest suite

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: maroffo
• Source: maroffo/claude-forge
• License: MIT

Install and usage instructions live in the source repository linked above.