Weak Agent Test

Adversarial review — weak-agent harness for docx-cli

This harness answers one question: can weak agents (Haiku) actually use docx-cli to get real work done, and what should we fix first? It runs the weak-agent-test workflow (.claude/workflows/weak-agent-test.js), which fans out one Haiku agent per scenario, renders every output with Microsoft Word, grades each against ground-truth criteria, and synthesizes a prioritized improvement report.

The test corpus is bundled with this skill under scenarios/, one folder per scenario, named after its key (scenarios/mnda/, scenarios/loi/, …). Each scenario folder is self-describing and holds everything that scenario needs:

• task.md — the AGENT-FACING request, written as a human delegating the work:

the goal, the data, the intent — and no tool vocabulary (no docx commands, locators, or OOXML terms), because discovering which features deliver the outcome is part of what's measured,

• criteria.md — the JUDGE-ONLY grading rubric (the precise, tool-specific checks).

The stage step withholds it from the agent's run workspace, and the judge reads it from the pristine source — the agent never sees the answer key,

• the fixture .docx to work on (edit scenarios only; authoring scenarios create

their output fresh),

• assets/ — any additional inputs (data files, images; empty for most edit

scenarios).

The workflow's SCENARIOS manifest holds only the per-scenario routing metadata (key, bucket label, edit/author kind, the doc filename, whether to render a baseline); the actual request/criteria/fixture/assets all live in the folder. The skill is therefore self-contained and travels with its test corpus. To change what a scenario tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx, renders, reviews, the report — are dumped to ./tmp/docx-weak-agent-test//, never into the repo.)

Each run produces, under the timestamped run dir, one result folder per scenario (named after its key) plus the run-level report and metrics:

/
  REPORT.md            ← synthesized report (+ appended measured metrics)
  haiku-metrics.md     ← measured per-Haiku-agent tokens/time/tool split
  haiku-metrics.json
  benchmark.json       ← self-reported tool economy
  /               ← one per scenario; the worked-on copy lives here
    task.md  assets/   ← (criteria.md is withheld from this copy — judge-only)
    .docx         ← the edited/authored document
    renders/output/    ← the Word-rendered PNGs the judge looked at
    renders/baseline/  ← pristine before-render (baseline scenarios only, e.g. psa)
    review.md          ← the judge's saved review for this task (written in-run)
    metrics.json       ← this task's measured tokens/time/tool split (written post-run by haiku-metrics)

Steps

Run these in order from the repo root. Do NOT skip the build — the global docx on PATH is a stale binary; the harness must test the CURRENT working tree.

1. Preflight — ALWAYS rebuild (mandatory gate)

The whole harness is meaningless if it tests a stale binary, so the build is a hard gate, not an optional step. Always run bun run build:binary, even if dist/docx already exists — never reuse a prior build. Abort the whole run if any check below fails.

REPO="$(git rev-parse --show-toplevel)"
cd "$REPO"
SCENARIOS_DIR="$REPO/.claude/skills/weak-agent-test/scenarios"   # this skill's bundled corpus (one folder per scenario)

# Word must be installed (this harness renders with Word, not LibreOffice).
test -d "/Applications/Microsoft Word.app" || echo "WARNING: Microsoft Word not found — render phase will fail."

# (1) Build the CURRENT working tree into a fresh standalone binary. Abort on failure.
bun run build:binary || { echo "BUILD FAILED — abort"; exit 1; }
BINARY="$REPO/dist/docx"

# (2) Hard gate: the fresh binary must match package.json's version AND have `render`.
# `--version` prints "docx X.Y.Z"; take the 2nd space-delimited field. NOTE: use `cut`,
# NOT an awk field reference — a literal dollar-N positional token gets clobbered by
# slash-command positional-arg substitution when this skill runs with arguments, mangling
# the gate. Keep this whole block free of dollar-N tokens for the same reason.
EXPECTED="$(bun -e 'console.log(require("./package.json").version)')"
GOT="$("$BINARY" --version | cut -d' ' -f2)"
echo "built docx $GOT (package.json: $EXPECTED)"
[ "$GOT" = "$EXPECTED" ] || { echo "VERSION MISMATCH ($GOT != $EXPECTED) — build is stale, abort"; exit 1; }
"$BINARY" render --help >/dev/null 2>&1 || { echo "render MISSING — build stale/broken, abort"; exit 1; }
echo "preflight OK: fresh $GOT binary with render"

If the version mismatches or render is missing, the build did not reflect the working tree — stop and fix it before running. Do not proceed on a stale binary.

> First-run note: Word-for-Mac rendering triggers a one-time macOS Automation > permission prompt for the controlling terminal. If the render phase fails on a > fresh machine, grant it under System Settings → Privacy & Security → Automation and > re-run.

2. Make an isolated run workspace (under ./tmp/)

Create an empty timestamped ./tmp/ run dir. Do NOT copy the scenarios here — the workflow's Stage phase copies only the active scenarios' folders from the pristine $SCENARIOS_DIR into $RUN_DIR, one subfolder per scenario ($RUN_DIR//), so originals stay untouched, the repo stays clean, and a single-scenario run doesn't drag the whole corpus along:

TS="$(date +%Y.%m.%d-%H%M%S)"
RUN_DIR="./tmp/docx-weak-agent-test/$TS"
mkdir -p "$RUN_DIR"   # empty; the workflow's Stage phase seeds one subfolder per active scenario from $SCENARIOS_DIR
echo "RUN_DIR=$RUN_DIR"

3. Launch the workflow

Invoke the Workflow tool with scriptPath pointing at the workflow file and pass the absolute paths as args:

Workflow({
  scriptPath: "/.claude/workflows/weak-agent-test.js",
  args: {
    runDir: "",
    binary: "",
    scenariosDir: "",
    only: 
  }
})

only restricts the run to a subset of scenarios (omit it to run all 6). To run a single task, pass its key as a plain string — only: "mnda". It also accepts an array (only: ["mnda", "loi"]) or a comma/space-separated string (only: "mnda,loi"); all three are normalized to the same list, so use whichever is convenient. The keys are the folder names under $SCENARIOS_DIR (run ls "$SCENARIOS_DIR" if you need to confirm them); unknown keys abort the run with a "No scenarios matched" error listing the valid ones.

> Use scriptPath, NOT name: "weak-agent-test". Launching by name resolves to a > copy cached at session start, so any edit to the workflow made during the session is > ignored; scriptPath always reads the current file from disk. (The workflow also > tolerates args arriving as a JSON string — the runtime stringifies it — so passing > a plain object is fine.)

When the tool returns, note the Transcript dir: path it prints — call it TRANSCRIPT_DIR (it looks like …/subagents/workflows/wf_). You need it in step 4 to measure per-agent tokens and time.

Scenario keys (omit only to run all 6): mnda, invoice, resume, contract-markup, contract-finalize, eliot-journal.

If the user passed scenario keys as arguments to this skill (e.g. /weak-agent-test mnda loi or mnda,loi), parse them into the only array. Otherwise run everything.

The run is heavy (6 Haiku agents, serial Word rendering, 6 judges + a synthesis pass); it can take many minutes. Watch live progress with /workflows.

4. Save the report + measure the Haiku metrics

When the workflow completes, its return value is { report, runDir, binary, exercises, verdicts, benchmark }. The report already contains the scoreboard, per-task merits/demerits, and the self-reported tool economy. Two more steps add the measured numbers:

1. Write report (Markdown) to "$RUN_DIR/REPORT.md" and benchmark (JSON) to

"$RUN_DIR/benchmark.json" with the Write tool.

1. Run the metrics pass over the run's transcripts to append the **accurate

per-Haiku-agent tokens + wall-clock time + docx/non-docx tool split** (the workflow can't measure these itself — the runtime gives the script no token API and bans clocks, so this reconstructs them from the transcripts). TRANSCRIPT_DIR is the path you noted in step 3: ``bash bun "$REPO/.claude/skills/weak-agent-test/scripts/haiku-metrics.ts" \ "" "$RUN_DIR" "$BINARY" "haiku" >> "$RUN_DIR/REPORT.md" ` The 4th argument is the exercise model the run used (the workflow's args.model, default haiku). **If you ran the workflow with a non-default args.model (e.g. sonnet), pass that same value as the 4th arg** — otherwise the metrics default to haiku, match no exercise agents, and emit an empty table (the script warns on stderr when 0 transcripts match the filter). This appends the measured "Haiku tool & cost economy" table to REPORT.md, writes $RUNDIR/haiku-metrics.json (run-level), and drops each scenario's measured row into $RUNDIR//metrics.json so every per-task folder is self-contained. Token cost is reported as **effective input** (cache-weighted: fresh input + cache write ×1.25 + cache read ×0.1) plus **output**, kept separate — NOT a single "total tokens", because cache reads are ~10× cheaper than fresh input and lumping them in overstates cost. The raw cache split is in the Totals table and haiku-metrics.json`.

1. Present in chat: the Executive summary, the per-task merits/demerits, and

the Haiku metrics (total docx vs other calls + docx share, effective input + output tokens, total time, and the per-scenario outliers). Tell the user where the artifacts live:

• /REPORT.md — findings + scoreboard + per-task merits/demerits + measured metrics table
• /haiku-metrics.json and benchmark.json — the raw numbers
• // — one folder per scenario, each holding that task's worked-on

.docx, its renders/ (the Word PNGs the judge looked at), review.md (the judge's saved review), and metrics.json (that task's measured tokens/time/tool split)

Notes

• This harness is re-runnable: each invocation rebuilds the binary (mandatory),

stages a fresh ./tmp/ run dir, and never mutates the bundled scenarios/.

• The headline benchmark metric is the Haiku tool economy — docx-cli calls vs

other tool calls (self-reported per agent, aggregated by the workflow). A high non-docx share, or a lot of docx calls for a simple task, is a friction signal.

• The weak agents invoke the binary at an allowlisted absolute path

(dist/docx), so they should not hit permission prompts for the CLI itself. The benign shell commands they and the render step use (mkdir, cp, ls, cat) are NOT yet allowlisted — if you get prompted, add them via the update-config skill or run with edits allowed. See .claude/settings.local.json.

• To add a scenario, create a folder under this skill's scenarios// holding

task.md (the agent-facing request, in human voice — NO tool vocabulary, so the agent must discover the features), criteria.md (the judge-only grading rubric — withheld from the agent's run workspace, read by the judge from the pristine source), the fixture .docx (edit scenarios only), and an assets/ folder, then add a routing entry to SCENARIOS in the workflow (.claude/workflows/weak-agent-test.js): { key, bucket, kind, doc, baseline }. To change what an existing scenario tests, edit the files in its folder — the request/criteria/fixture/assets all live there, not in the workflow.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kklimuk
• Source: kklimuk/docx-cli
• License: MIT

Install and usage instructions live in the source repository linked above.