Audit Runs

audit-runs

MANDATORY: READ ENTIRE FILE before composing user-facing prose: $PWD/skills/shared/readability-style.md.

Audit recently-merged larch run logs for the selected skill (--skill=design|implement) for anomalies. The current implement registry covers EXON/OOS/missing-file/NS-retry/self-deploying checks; the current design registry is intentionally narrower and ships cache-freshness plus guideline-assessment coverage. Always file a chain-of-history audit-report issue; record bug-issue candidates as proposals at scan time and act on them only after explicit user direction in chat.

This is a dev-only operator skill (.claude/skills/). It is NOT shipped with the plugin.

Usage

/larch:audit-runs --skill  [] [--repo owner/name] [--allow-concurrent]

(Plugin slash-alias: /audit-runs … may also resolve to this skill depending on marketplace wiring; prefer /larch:audit-runs when unsure.)

Args

• --skill (required): which skill’s run logs to audit. Closed enum: design or implement. Reject missing or out-of-enum values before any side effect (no gh call, no tmpdir). Parse from argv before preflight.
• ` (optional positional): when present, describes which PRs to audit. When omitted or empty, treat as since last audit` (same error paths as that form). Supported forms:
• last N PRs — N most-recently-merged PRs targeting main
• since last audit — PRs merged after the prior audit report's audited_pr_range.last; error if no prior report exists or no new PRs have merged (do NOT file an empty report)
• since — PRs merged after that instant (same interpretable forms as GitHub mergedAt: …Z or explicit ±HH:MM offset). This filter is not tied to the Pacific wall-clock convention used for audit report titles and audit_timestamp.
• #N or PR #N — exactly one PR
• Default when empty: treat as since last audit (requires a prior audit-report issue; continues to error if no prior report or malformed frontmatter — same as the explicit since last audit form)
• --repo : target repo. Default: character-ai/larch
• --allow-concurrent: override the shared 5-minute audit-report concurrency guard (not skill-scoped)
• Removed flag: --no-fix-issues is not supported. If any token in the skill argv is exactly --no-fix-issues, refuse immediately with a clear usage error (flag removed); do not proceed or silently ignore it.

Pre-flight

After parsing and validating --skill into $SKILL:

PREFLIGHT_OUT=$(python3 "$PWD/python/cli.py" audit-runs preflight \
  --skill "$SKILL" --repo "" [--allow-concurrent])

Read PREFLIGHT_OK and REASON from stdout. Fail-fast when PREFLIGHT_OK=false; print REASON to the user. Contract: python/cli.py audit-runs preflight.

Verbal-Description Resolution

RESOLVE_OUT=$(python3 "$PWD/python/cli.py" audit-runs resolve-prs \
  --skill "$SKILL" --repo "" [--verbal-description ""])

Read PR_LIST, PR_COUNT, IMPLICIT_SINCE_LAST_AUDIT, PRIOR_REPORT_NUMBER, RESOLVED_ECHO, and ERROR from stdout. Fail-fast when ERROR is non-empty; print it to the user. Print RESOLVED_ECHO before scanning. Contract: python/cli.py audit-runs resolve-prs.

Scan Registry

The scan list is externalized in .claude/skills/audit-runs/scans-$SKILL.tsv (one row per scan: name, type, pattern, expected_outcome, severity). JSON emission, category filtering, and scan implementations live in python/audit_runs.py behind python/cli.py audit-runs scan-run; offline coverage lives in python/test_audit_runs.py. Adding a scan requires coordinated updates: (1) a new scans-$SKILL.tsv row in the relevant per-skill registry, (2) matching python/audit_runs.py scan-run logic, (3) any counter wiring in python/audit_runs.py compute_counters_main when the scan feeds cumulative totals, (4) this SKILL.md scan table if the operator-facing baseline changes, and (5) hermetic python/test_audit_runs.py coverage for the new NDJSON shape and counter path. Plan fidelity: substantive changes to that surface (new counters, new cumulative YAML keys, or registry-wide behavior) should be tracked in their own issue/PR when they go beyond a routine scan-row + test update. Routine changelog-rebase-conflicts / changelog_rebase_conflicts / ns_retries_cursor_specialist wiring is part of this skill’s maintained baseline, not an ad-hoc add-on. Operator parity with run-log audit-title hygiene on main: audit-title and search-exclusion work assumes run logs and issue titles stay aligned with the same ^\[(Run Logs Audit |Implement Run Logs Audit |Design Run Logs Audit ).* Report\] title regex used by the audit-report writer; the pre-lock python3 "$PWD/python/cli.py" git check-main-sync probe uses the locally cached origin/main ref (no fetch). If the probe fails with SYNC_STATUS=probe-error, operators must git fetch origin main before locking. This is the same freshness requirement as the audit-report title migration. Treat main-sync as a first-class preflight gate next to audit-title hygiene, not an undocumented side effect.

Read the registry at runtime:

SCANS_TSV="$PWD/.claude/skills/audit-runs/scans-$SKILL.tsv"

Scans (baseline — see scans-implement.tsv / scans-design.tsv for the machine-readable registry)

Implement currently uses the full table below. Design currently uses cache-freshness and guideline-assessment unless/until scans-design.tsv grows additional rows.

| Scan | What | Where | |---|---|---| | Required-file presence | Compare against docs/run-logs-required-files.tsv (NDJSON result is pass / fail / skip / error) | run-log root | | EXON misclassification | \| FINDING_.* \| 0 \| 0 \| [1-9]+ \|.*\| rejected \| | round-*/voting-tally.md | | OOS category mangle | plan-review accepted rows only: non-empty category not in {code-quality, risk-integration, correctness, architecture, security} (code-review accepted prose categories are ignored by design) | review-findings-full.jsonl | | NS-retry sidecars | reviewer_signals[].ns_retry_reason when present; legacy *-ns-retry* file fallback when the concise carrier is missing (result:"skip" only when both are absent) | round-*/round-meta.json (+ legacy round-*/) | | Cursor CI stall causes | cursor-ci-stall-*.json sidecars: informational histogram of .channel values (pass when none) | round-*/ | | Codex round-1 adherence | round 2+ panel-manifest should not contain tool=codex | round-N/panel-manifest.ndjson | | Codex generalist waste | reviewer_signals entry for codex-generalist-output.txt is NO_ISSUES_FOUND only AND timing > 120s (result:"skip" when carrier missing) | round-1/round-meta.json + timing-report.json | | Execution-issues categories | non-Warnings entries in execution-issues.ndjson | execution-issues.ndjson | | Cache freshness | manifest.json::larch_version vs latest plugin version (result: informational when the run lags current; empty larch_version remains fail; other rows may emit skip/error) | manifest.json | | Guideline assessment | committed Gate C guideline assessment is present and non-empty when the artifact exists (result:"informational" when absent) | architectural-guideline-assessment.md | | Changelog rebase/conflicts (heuristic) | execution-issues.ndjson bodies mentioning changelog + rebase/conflict | execution-issues.ndjson | | Coder tool | CODER_TOOL field | round-*/coder.env | | Trailing-content NOISSUESFOUND | reviewer_signals[].first_pass_trailing_content == true (result:"skip" when carrier missing; legacy *-first-pass.txt no longer primary) | round-*/round-meta.json | | OOS silent drop | accepted non-security ### OOS_ blocks vs filed GitHub URLs, Inline-triage commit lines, and rejected-OOS markers in oos-issues.ndjson | oos-accepted-*.md, oos-issues*.ndjson, oos-issues-created.md, git log on run-log repo root |

Scanning

# lint-consecutive-bash: ok map output feeds per-run scan loop
# Map each PR to its run-log directory
RUN_MAP_TSV=$(python3 "$PWD/python/cli.py" audit-runs map-runs \
  --skill "$SKILL" --pr-list "$PR_LIST" --repo "")
# TSV: pr_numberrun_idstarted_atlarch_versioncloses_issue

Then for each PR row in the TSV:

python3 "$PWD/python/cli.py" audit-runs scan-run \
  --skill "$SKILL" \
  --run-dir "larch-logs/$SKILL/" \
  --pr  \
  --scans-tsv "$SCANS_TSV" \
  --required-files-tsv "$PWD/docs/run-logs-required-files.tsv" \
  --current-version "" \
  > "$TMPDIR/scan-results-.ndjson"

Read scan-results-*.ndjson files as NDJSON (one JSON object per scan per line). Each line’s result is not limited to pass/fail: treat informational, skip, and error as first-class outcomes when writing the report (for example cache-freshness behind current vs missing inputs vs manifest/registry drift). Contract: python/cli.py audit-runs scan-run.

Cross-cutting checks (NDJSON + operator judgment): the synthetic cross-cutting object (and cache-freshness / manifest fields) flags manifest integrity — empty ended_at / pr_number, and manifest_pr_number_mismatch_with_audited_pr / legacy self_deploying_gap when manifest.json’s pr_number disagrees with the audited PR (run-log vs merge skew / self-deploying version gaps). When run_version --search "" --json number,title,state,closedAt ``

1. Open matches (including [IMPLEMENTING] … titles): proposed_augmentations. In ## Open issues snapshot, when an augmented issue’s title starts with [IMPLEMENTING], note that the finding recurred in this batch (pre-fix) for that issue number.
2. Closed matches only (no open match for this finding): apply the version-window check before proposing proposed_new_issues:

• Resolve fix merge time: prefer gh pr list --state merged --search "closes #" --repo --json number,mergedAt,title,body and pick one merged PR you attribute to the fix (see PR disambiguation below). If that query returns no candidates, fall back to gh issue view --json closedAt,createdAt for timing only and set matched_pr / merge metadata to unknown in your notes / version_window_checks rationale.
• PR disambiguation (normative): When multiple merged PRs match the search, prefer the PR whose body/title contains an explicit closing reference for # (closes, fixes, resolved, case-insensitive). If still tied, prefer the PR with mergedAt closest after the issue createdAt (smallest positive delta). If no candidate has mergedAt strictly after createdAt, use the candidate with the latest mergedAt (ISO-8601 timestamps sort lexicographically). If still ambiguous, do not silently suppress: treat as in-scope (decision: propose, in_scope: true) and record both PR numbers plus a one-line operator rationale in the audit-report prose (or in the finding slug row’s implied narrative). When the search returns zero PRs, use issue closedAt only; keep fix_shipped_in: unknown unless you can attribute a merge elsewhere.
• Find the next plugin version shipped after that instant (file-shaped contract, not subject-line guessing alone):

``bash git log --oneline --grep="Bump version" --after="" --reverse -- .claude-plugin/plugin.json | head -1 ` Let BUMPSHA be that commit hash (first field of the line). Read the shipped plugin version from the tree at that commit: `bash git show "$BUMP_SHA:.claude-plugin/plugin.json" | jq -r '.version // empty' ` **Normalize** the returned dotted version for comparisons: strip a single leading v, trim ASCII whitespace, require three **integer** components MAJOR.MINOR.PATCH, then compare **numerically per component** (so 1.10.0 is greater than 1.9.999; do **not** use naive string sort on the dotted token). If **no** bump commit exists after that instant **or** .version is empty, set fixshipped_in: unknown` — do not skip the proposal solely for missing bump metadata (treat as in-scope for recurrence unless other reasoning applies).

• If either fix_shipped_version or an audited larch_version fails that three-integer parse (for example 34.0.0-rc1, extra dotted segments, or odd strings), treat that side as unknown for the inequality: you cannot prove the fix is strictly newer than every audited run, so do not apply closed-only suppression on that basis alone — propose and record the parse gap in version_window_checks.
• Compare fix_shipped_version (parsed semantic version, or unknown) against each audited run’s manifest.json::larch_version from this batch’s run-map / scan inputs (normalize each batch version the same way).
• If fix_shipped_version is known and strictly greater than every audited larch_version, the fix post-dates all audited runs → do not propose a new issue for this closed-only match.
• If fix_shipped_version is unknown, or fix_shipped_version ≤ any audited larch_version, the closed fix was in scope for at least one run → propose proposed_new_issues (recurrence).

1. Record each closed-issue evaluation in version_window_checks (see Frontmatter). Use version_window_checks: [] when no closed issue was evaluated for any finding.

Precedence: any open match for the finding → proposed_augmentations only (even if older closed duplicates exist).

Post-report user prompt

After the audit report issue is filed and prior reports are handled per Close Prior Reports:

1. Print the full audit-report body verbatim to chat (the same markdown submitted as the issue body), then print the audit-report URL.
2. Zero-findings short-circuit: if proposed_new_issues and proposed_augmentations are both empty, state No findings — no bug issues to file. and exit — do not ask the 3-way question.
3. Otherwise, ask the operator a 3-way question: (1) file/augment all, (2) discuss specific findings first, or (3) skip filing. Act on the response:

• File/augment all: file new issues via /larch:issue (dedup ON); post augmentation comments with gh issue comment --repo "" --body-file "$TMPDIR/audit-augment-.md" (write the Augmentation comment shape markdown to that file first — same --body-file pattern as issue create-one; do not pass multi-line tables through an inline --body string).
• Discuss first: wait for operator direction; file or augment per finding only as approved.
• Skip filing: exit cleanly; the audit report already captures proposed findings for the historical record.

1. Post-report session summary (audit-report issue only): only when an audit-report issue was actually filed (issue create-one returned a non-empty issue number / AUDIT_REPORT_NUMBER) and you did not end the run at step 2’s zero-findings short-circuit (that path exits immediately after the chat message — no 3-way walkthrough, no session-summary). After step 3’s per-finding walkthrough completes (filed, augmented, skipped, or mixed), compose $TMPDIR/session-summary.md and post it as a single comment on that audit-report issue (supplementary history). Skip this entire step whenever no audit-report issue exists — for example the zero-PR since last audit short-circuit (no issue create-one call), preflight/resolve failures before filing, or any other path that never yields an audit-report issue number (there is nothing to comment on).

```markdown ## Post-report session summary

3-way decision:

Per-finding actions:

| Finding | Decision | Filed as | URL | |---|---|---|---| | ... | filed-as-drafted \| modified \| skipped | #N or — | url or — |

Augmentations:

| Target issue | Action | Comment URL | |---|---|---| | #N | posted \| skipped | url or — |

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: character-ai
• Source: character-ai/larch
• License: MIT

Install and usage instructions live in the source repository linked above.