# Harness Trace

> A Claude skill from maroffo/claude-forge.

- **Type:** Skill
- **Install:** `agentstack add skill-maroffo-claude-forge-harness-trace`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [maroffo](https://agentstack.voostack.com/s/maroffo)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [maroffo](https://github.com/maroffo)
- **Source:** https://github.com/maroffo/claude-forge/tree/main/skills/harness-trace

## Install

```sh
agentstack add skill-maroffo-claude-forge-harness-trace
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# ABOUTME: Structured execution trace capture and token baselining for claude-forge
# ABOUTME: Extracts traces from session JSONL, counts tokens with tiktoken, generates baselines

---
name: harness-trace
description: "Execution trace capture and token baselining. Use when user says trace, extract trace, count tokens, token baseline, or /harness-trace. Not for harness optimization (use harness-mechanic)."
compatibility: "Python >=3.11, uv"
---

# Harness Trace

Structured execution trace capture and token baselining for the claude-forge harness. Based on Meta-Harness (arxiv 2603.28052): measure before optimizing.

## Quick Start

```bash
cd skills/harness-trace

# Install
uv sync

# Extract traces from a session JSONL
uv run -- harness-trace extract ~/.claude/projects/PROJECT/SESSION.jsonl

# Count tokens in harness files
uv run -- harness-trace count-tokens rules/

# Generate full baseline
uv run -- harness-trace baseline --base-dir /path/to/claude-forge
```

## Commands

| Command | Purpose | Output |
|---------|---------|--------|
| `extract ` | Parse session into trace JSONL | Trace entries (stdout or file) |
| `count-tokens ` | Count tokens per file | Token counts per file |
| `baseline --base-dir ` | Full harness token baseline | TSV with tier classification |

## Extraction strategy

`extract` uses two signal sources, in order of precision:

1. **Tool-use blocks** (primary, high precision). Walks every `tool_use` block in the assistant messages. The mapping below converts concrete tool invocations into trace entries with confidence:

   | Tool / pattern | Emits |
   |----------------|-------|
   | `Agent(subagent_type=*-reviewer)` | `ROUTE` + `REVIEW` |
   | `Agent(subagent_type=*)` (non-reviewer) | `ROUTE` |
   | `Bash` matching `pytest\|make test\|go test\|npm test\|cargo test\|rspec` | `VERIFY` |
   | `Bash` matching `ast-grep\|sg --pattern` | `BLAST_RADIUS` |
   | `Edit` / `Write` / `MultiEdit` | counted into `IMPLEMENT` aggregate |
   | `WebFetch` to `arxiv\|docs` | `RESEARCH` |
   | `AskUserQuestion` | counted into `safety_compliance.hitl_gates_hit` |

2. **Text regex** (fallback). Applied only to messages whose tool stream did not already surface the step, and only for steps that are typically pure text (`SCORE`, `FIX`, `LOOP`, `UAT`, plus `VERIFY`/`REVIEW` when no tool signal exists). This avoids the v1 problem where prose like "tests pass" in a chat triggered fake VERIFY entries.

`SUMMARY` is always synthesized last from JSONL aggregates: `duration_min` (last_ts - first_ts), `files_changed` (Edit+Write count), and the 6-dimension `metrics` object (paper §5.2.1). No `SUMMARY` is emitted for sessions with zero other entries.

## Trace Schema (v2)

One JSONL line per orchestrator step. `v2` adds `rejected_alternatives` (top-level, attaches to any step), two new step types (`PERMISSION_EVENT`, `ROUTE`), and an end-of-task `metrics` mini-report inside `SUMMARY`. Based on "Code as Agent Harness" (arxiv 2605.18747 §3.5.1, §5.2.1).

**Step taxonomy.** `step` mixes two kinds intentionally:
- **Lifecycle phases** (`REFINE`, `RESEARCH`, ..., `SUMMARY`): at most once per round, sequential.
- **Cross-cutting events** (`PERMISSION_EVENT`, `ROUTE`): fire opportunistically, possibly many per phase. Consumers that care about the distinction should filter by step name.

```jsonl
{"v":2,"session":"slug","ts":"ISO8601","step":"STEP","data":{...},"rejected_alternatives":[...]}
```

| Step | Data fields |
|------|-------------|
| REFINE | ambiguities_found, questions_asked |
| RESEARCH | complexity, sources_consulted |
| LOCALIZE | files_planned, files_proposed, files_actually_changed, precision, recall, mismatches |
| REPRODUCE | script, fails_before_fix, passes_after_fix |
| IMPLEMENT | agents, files_changed, subtask_count, localization_precision |
| DRIFT_CHECK | subtask_id, verdict, deviations |
| VERIFY | tests_pass, lint_clean, build_ok, retries, reproduction_confirmed |
| REVIEW | agents, findings (CRITICAL/MAJOR/MINOR), review_validity |
| FIX | findings_addressed, deviations |
| BLAST_RADIUS | triggered, trigger_reason, files_scanned, contradictions |
| SCORE | score, threshold, gate |
| LOOP | round, total_rounds, exit_reason |
| UAT | performed, items, passed, failed |
| SUMMARY | tokens_in/out, model, duration, final_score, **metrics** (v2) |
| **PERMISSION_EVENT** (v2) | tool, action, outcome (granted/denied/denied_by_settings/auto_approved/error/timeout/bypassed), reason. Callers must redact secrets in `action`. |
| **ROUTE** (v2) | router, target, alternatives_considered, decision_basis |

### v2 cross-cutting fields

- `rejected_alternatives` (top-level on TraceEntry): list of `{description, reason_rejected, cost_estimate?}`. Captures paths the agent considered but discarded. Useful for diagnosing decision quality without rerunning the agent.

### v2 SUMMARY.metrics (6 harness dimensions, paper §5.2.1)

Each dimension is an optional dict; populate only what was measured. `None` = not measured (do not infer 0).

| Dimension | Suggested keys |
|-----------|----------------|
| trajectory_efficiency | tool_calls, tokens, edits, executions, active_min (gap-clamped working time, paired with SUMMARY.duration_min calendar span) |
| verification_strength | test_coverage_pct, oracles_count, false_accept_rate |
| recovery_ability | failures, recovered, escalations |
| state_consistency | memory_repo_synced, drift_detected |
| safety_compliance | permission_denials, hitl_gates_hit, sandbox_used |
| replayability | full_trace_captured, artifacts_persisted |

## Token Baseline TSV

```
file	tokens	words	lines	tier	loaded
rules/orchestrator-protocol.md	1247	450	67	rule	always
skills/golang/SKILL.md	2890	1050	145	skill	on-demand
```

Tiers: `rule` (always-on), `skill`/`skill-ref` (on-demand), `agent` (on-demand), `config` (always-on).

## Architecture

```
src/harness_trace/
  models.py          trace JSONL (heuristic parser)
  token_counter.py  <- tiktoken counting, directory scanning, TSV generation
  cli.py            <- CLI entry point (extract, count-tokens, baseline)
tests/              <- pytest suite
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [maroffo](https://github.com/maroffo)
- **Source:** [maroffo/claude-forge](https://github.com/maroffo/claude-forge)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-maroffo-claude-forge-harness-trace
- Seller: https://agentstack.voostack.com/s/maroffo
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.