# Audit Runs

> Use when auditing recently merged larch run logs for anomalies, filing the chain-of-history audit issue, and proposing user-approved bug follow-ups.

- **Type:** Skill
- **Install:** `agentstack add skill-character-ai-larch-audit-runs`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [character-ai](https://agentstack.voostack.com/s/character-ai)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [character-ai](https://github.com/character-ai)
- **Source:** https://github.com/character-ai/larch/tree/main/.claude/skills/audit-runs

## Install

```sh
agentstack add skill-character-ai-larch-audit-runs
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# audit-runs

**MANDATORY: READ ENTIRE FILE before composing user-facing prose: `$PWD/skills/shared/readability-style.md`.**

Audit recently-merged larch run logs for the selected skill (`--skill=design|implement`) for anomalies. The current implement registry covers EXON/OOS/missing-file/NS-retry/self-deploying checks; the current design registry is intentionally narrower and ships cache-freshness plus guideline-assessment coverage. Always file a chain-of-history audit-report issue; record bug-issue candidates as proposals at scan time and act on them only after explicit user direction in chat.

This is a **dev-only** operator skill (`.claude/skills/`). It is NOT shipped with the plugin.

## Usage

```
/larch:audit-runs --skill  [] [--repo owner/name] [--allow-concurrent]
```

(Plugin slash-alias: `/audit-runs …` may also resolve to this skill depending on marketplace wiring; prefer `/larch:audit-runs` when unsure.)

### Args

- `--skill ` (**required**): which skill’s run logs to audit. Closed enum: `design` or `implement`. Reject missing or out-of-enum values before any side effect (no `gh` call, no tmpdir). Parse from argv before preflight.
- `` (optional positional): when present, describes which PRs to audit. When omitted or empty, treat as `since last audit` (same error paths as that form). Supported forms:
  - `last N PRs` — N most-recently-merged PRs targeting `main`
  - `since last audit` — PRs merged after the prior audit report's `audited_pr_range.last`; error if no prior report exists or no new PRs have merged (do NOT file an empty report)
  - `since ` — PRs merged after that instant (same interpretable forms as GitHub `mergedAt`: `…Z` or explicit `±HH:MM` offset). This filter is **not** tied to the Pacific wall-clock convention used for audit report titles and `audit_timestamp`.
  - `#N` or `PR #N` — exactly one PR
  - Default when empty: treat as `since last audit` (requires a prior audit-report issue; continues to error if no prior report or malformed frontmatter — same as the explicit `since last audit` form)
- `--repo `: target repo. Default: `character-ai/larch`
- `--allow-concurrent`: override the shared 5-minute `audit-report` concurrency guard (not skill-scoped)
- **Removed flag**: `--no-fix-issues` is not supported. If any token in the skill argv is exactly `--no-fix-issues`, refuse immediately with a clear usage error (flag removed); do not proceed or silently ignore it.

## Pre-flight

After parsing and validating `--skill` into `$SKILL`:

```bash
PREFLIGHT_OUT=$(python3 "$PWD/python/cli.py" audit-runs preflight \
  --skill "$SKILL" --repo "" [--allow-concurrent])
```

Read `PREFLIGHT_OK` and `REASON` from stdout. Fail-fast when `PREFLIGHT_OK=false`; print `REASON` to the user. Contract: `python/cli.py audit-runs preflight`.

## Verbal-Description Resolution

```bash
RESOLVE_OUT=$(python3 "$PWD/python/cli.py" audit-runs resolve-prs \
  --skill "$SKILL" --repo "" [--verbal-description ""])
```

Read `PR_LIST`, `PR_COUNT`, `IMPLICIT_SINCE_LAST_AUDIT`, `PRIOR_REPORT_NUMBER`, `RESOLVED_ECHO`, and `ERROR` from stdout. Fail-fast when `ERROR` is non-empty; print it to the user. Print `RESOLVED_ECHO` before scanning. Contract: `python/cli.py audit-runs resolve-prs`.

## Scan Registry

The scan list is externalized in `.claude/skills/audit-runs/scans-$SKILL.tsv` (one row per scan: `name`, `type`, `pattern`, `expected_outcome`, `severity`). JSON emission, category filtering, and scan implementations live in `python/audit_runs.py` behind `python/cli.py audit-runs scan-run`; offline coverage lives in `python/test_audit_runs.py`. **Adding a scan** requires coordinated updates: (1) a new `scans-$SKILL.tsv` row in the relevant per-skill registry, (2) matching `python/audit_runs.py` scan-run logic, (3) any counter wiring in `python/audit_runs.py` `compute_counters_main` when the scan feeds cumulative totals, (4) this `SKILL.md` scan table if the operator-facing baseline changes, and (5) hermetic `python/test_audit_runs.py` coverage for the new NDJSON shape and counter path. **Plan fidelity**: substantive changes to that surface (new counters, new cumulative YAML keys, or registry-wide behavior) should be tracked in their own issue/PR when they go beyond a routine scan-row + test update. Routine `changelog-rebase-conflicts` / `changelog_rebase_conflicts` / `ns_retries_cursor_specialist` wiring is part of this skill’s maintained baseline, not an ad-hoc add-on. **Operator parity with run-log audit-title hygiene on `main`**: audit-title and search-exclusion work assumes run logs and issue titles stay aligned with the same `^\[(Run Logs Audit |Implement Run Logs Audit |Design Run Logs Audit ).* Report\]` title regex used by the audit-report writer; the pre-lock `python3 "$PWD/python/cli.py" git check-main-sync` probe uses the locally cached `origin/main` ref (no fetch). If the probe fails with `SYNC_STATUS=probe-error`, operators must `git fetch origin main` before locking. This is the same freshness requirement as the audit-report title migration. Treat main-sync as a first-class preflight gate next to audit-title hygiene, not an undocumented side effect.

Read the registry at runtime:
```bash
SCANS_TSV="$PWD/.claude/skills/audit-runs/scans-$SKILL.tsv"
```

### Scans (baseline — see `scans-implement.tsv` / `scans-design.tsv` for the machine-readable registry)

Implement currently uses the full table below. Design currently uses `cache-freshness` and `guideline-assessment` unless/until `scans-design.tsv` grows additional rows.

| Scan | What | Where |
|---|---|---|
| Required-file presence | Compare against `docs/run-logs-required-files.tsv` (NDJSON `result` is `pass` / `fail` / `skip` / `error`) | run-log root |
| EXON misclassification | `\| FINDING_.* \| 0 \| 0 \| [1-9]+ \|.*\| rejected \|` | `round-*/voting-tally.md` |
| OOS category mangle | plan-review **accepted** rows only: non-empty `category` not in `{code-quality, risk-integration, correctness, architecture, security}` (code-review accepted prose categories are ignored by design) | `review-findings-full.jsonl` |
| NS-retry sidecars | `reviewer_signals[].ns_retry_reason` when present; legacy `*-ns-retry*` file fallback when the concise carrier is missing (`result:"skip"` only when both are absent) | `round-*/round-meta.json` (+ legacy `round-*/`) |
| Cursor CI stall causes | `cursor-ci-stall-*.json` sidecars: informational histogram of `.channel` values (pass when none) | `round-*/` |
| Codex round-1 adherence | round 2+ panel-manifest should not contain `tool=codex` | `round-N/panel-manifest.ndjson` |
| Codex generalist waste | `reviewer_signals` entry for `codex-generalist-output.txt` is `NO_ISSUES_FOUND` only AND timing > 120s (`result:"skip"` when carrier missing) | `round-1/round-meta.json` + `timing-report.json` |
| Execution-issues categories | non-Warnings entries in `execution-issues.ndjson` | `execution-issues.ndjson` |
| Cache freshness | `manifest.json::larch_version` vs latest plugin version (`result: informational` when the run lags current; empty `larch_version` remains `fail`; other rows may emit `skip`/`error`) | `manifest.json` |
| Guideline assessment | committed Gate C guideline assessment is present and non-empty when the artifact exists (`result:"informational"` when absent) | `architectural-guideline-assessment.md` |
| Changelog rebase/conflicts (heuristic) | `execution-issues.ndjson` bodies mentioning changelog + rebase/conflict | `execution-issues.ndjson` |
| Coder tool | `CODER_TOOL` field | `round-*/coder.env` |
| Trailing-content NO_ISSUES_FOUND | `reviewer_signals[].first_pass_trailing_content == true` (`result:"skip"` when carrier missing; legacy `*-first-pass.txt` no longer primary) | `round-*/round-meta.json` |
| OOS silent drop | accepted non-security `### OOS_` blocks vs filed GitHub URLs, Inline-triage commit lines, and rejected-OOS markers in `oos-issues.ndjson` | `oos-accepted-*.md`, `oos-issues*.ndjson`, `oos-issues-created.md`, git log on run-log repo root |

## Scanning

```bash
# lint-consecutive-bash: ok map output feeds per-run scan loop
# Map each PR to its run-log directory
RUN_MAP_TSV=$(python3 "$PWD/python/cli.py" audit-runs map-runs \
  --skill "$SKILL" --pr-list "$PR_LIST" --repo "")
# TSV: pr_numberrun_idstarted_atlarch_versioncloses_issue
```

Then for each PR row in the TSV:

```bash
python3 "$PWD/python/cli.py" audit-runs scan-run \
  --skill "$SKILL" \
  --run-dir "larch-logs/$SKILL/" \
  --pr  \
  --scans-tsv "$SCANS_TSV" \
  --required-files-tsv "$PWD/docs/run-logs-required-files.tsv" \
  --current-version "" \
  > "$TMPDIR/scan-results-.ndjson"
```

Read `scan-results-*.ndjson` files as NDJSON (one JSON object per scan per line). Each line’s `result` is not limited to pass/fail: treat **`informational`**, **`skip`**, and **`error`** as first-class outcomes when writing the report (for example `cache-freshness` behind current vs missing inputs vs manifest/registry drift). Contract: `python/cli.py audit-runs scan-run`.

**Cross-cutting checks (NDJSON + operator judgment):** the synthetic `cross-cutting` object (and `cache-freshness` / manifest fields) flags **manifest integrity** — empty `ended_at` / `pr_number`, and `manifest_pr_number_mismatch_with_audited_pr` / legacy `self_deploying_gap` when `manifest.json`’s `pr_number` disagrees with the audited PR (run-log vs merge skew / self-deploying version gaps). When `run_version  --search "" --json number,title,state,closedAt
   ```
2. **Open matches** (including `[IMPLEMENTING] …` titles): **`proposed_augmentations`**. In **`## Open issues snapshot`**, when an augmented issue’s title starts with `[IMPLEMENTING]`, note that the finding **recurred in this batch (pre-fix)** for that issue number.
3. **Closed matches only** (no open match for this finding): apply the **version-window** check before proposing `proposed_new_issues`:
   - Resolve fix merge time: prefer `gh pr list --state merged --search "closes #" --repo  --json number,mergedAt,title,body` and pick **one** merged PR you attribute to the fix (see **PR disambiguation** below). If that query returns **no** candidates, fall back to `gh issue view  --json closedAt,createdAt` for timing only and set `matched_pr` / merge metadata to unknown in your notes / `version_window_checks` rationale.
   - **PR disambiguation (normative):** When multiple merged PRs match the search, prefer the PR whose `body`/`title` contains an explicit closing reference for `#` (`closes`, `fixes`, `resolved`, case-insensitive). If still tied, prefer the PR with `mergedAt` closest **after** the issue `createdAt` (smallest positive delta). **If no candidate has `mergedAt` strictly after `createdAt`**, use the candidate with the latest `mergedAt` (ISO-8601 timestamps sort lexicographically). If still ambiguous, **do not** silently suppress: treat as **in-scope** (`decision: propose`, `in_scope: true`) and record both PR numbers plus a one-line operator rationale in the audit-report prose (or in the `finding` slug row’s implied narrative). When the search returns **zero** PRs, use issue `closedAt` only; keep `fix_shipped_in: unknown` unless you can attribute a merge elsewhere.
   - Find the next plugin version shipped after that instant (file-shaped contract, not subject-line guessing alone):
     ```bash
     git log --oneline --grep="Bump version" --after="" --reverse -- .claude-plugin/plugin.json | head -1
     ```
     Let `BUMP_SHA` be that commit hash (first field of the line). Read the shipped plugin version from the tree at that commit:
     ```bash
     git show "$BUMP_SHA:.claude-plugin/plugin.json" | jq -r '.version // empty'
     ```
     **Normalize** the returned dotted version for comparisons: strip a single leading `v`, trim ASCII whitespace, require three **integer** components `MAJOR.MINOR.PATCH`, then compare **numerically per component** (so `1.10.0` is greater than `1.9.999`; do **not** use naive string sort on the dotted token). If **no** bump commit exists after that instant **or** `.version` is empty, set `fix_shipped_in: unknown` — **do not** skip the proposal solely for missing bump metadata (treat as in-scope for recurrence unless other reasoning applies).
   - If either `fix_shipped_version` or an audited `larch_version` fails that three-integer parse (for example `34.0.0-rc1`, extra dotted segments, or odd strings), treat that side as `unknown` for the inequality: you cannot prove the fix is strictly newer than **every** audited run, so **do not** apply closed-only suppression on that basis alone — **propose** and record the parse gap in `version_window_checks`.
   - Compare `fix_shipped_version` (parsed semantic version, or `unknown`) against each audited run’s `manifest.json::larch_version` from this batch’s run-map / scan inputs (normalize each batch version the same way).
   - If `fix_shipped_version` is known and **strictly greater than** every audited `larch_version`, the fix post-dates all audited runs → **do not** propose a new issue for this closed-only match.
   - If `fix_shipped_version` is `unknown`, **or** `fix_shipped_version ≤` any audited `larch_version`, the closed fix was in scope for at least one run → propose **`proposed_new_issues`** (recurrence).
4. **Record** each closed-issue evaluation in **`version_window_checks`** (see **Frontmatter**). Use `version_window_checks: []` when no closed issue was evaluated for any finding.

**Precedence:** any **open** match for the finding → `proposed_augmentations` only (even if older closed duplicates exist).

### Post-report user prompt

After the audit report issue is filed and prior reports are handled per **Close Prior Reports**:

1. Print the **full audit-report body** verbatim to chat (the same markdown submitted as the issue body), then print the **audit-report URL**.
2. **Zero-findings short-circuit**: if `proposed_new_issues` and `proposed_augmentations` are both empty, state `No findings — no bug issues to file.` and exit — do **not** ask the 3-way question.
3. **Otherwise**, ask the operator a 3-way question: (1) file/augment all, (2) discuss specific findings first, or (3) skip filing. Act on the response:
   - **File/augment all**: file new issues via `/larch:issue` (dedup ON); post augmentation comments with `gh issue comment  --repo "" --body-file "$TMPDIR/audit-augment-.md"` (write the **Augmentation comment shape** markdown to that file first — same `--body-file` pattern as `issue create-one`; do not pass multi-line tables through an inline `--body` string).
   - **Discuss first**: wait for operator direction; file or augment per finding only as approved.
   - **Skip filing**: exit cleanly; the audit report already captures proposed findings for the historical record.

4. **Post-report session summary (audit-report issue only):** **only when** an audit-report issue was actually filed (`issue create-one` returned a non-empty issue number / `AUDIT_REPORT_NUMBER`) **and** you did **not** end the run at step 2’s **zero-findings short-circuit** (that path exits immediately after the chat message — no 3-way walkthrough, **no** session-summary). After step 3’s per-finding walkthrough completes (filed, augmented, skipped, or mixed), compose `$TMPDIR/session-summary.md` and post it as a single comment on that audit-report issue (supplementary history). **Skip** this entire step whenever **no** audit-report issue exists — for example the zero-PR `since last audit` short-circuit (no `issue create-one` call), preflight/resolve failures before filing, or any other path that never yields an audit-report issue number (there is nothing to comment on).

   ```markdown
   ## Post-report session summary

   **3-way decision**: 

   **Per-finding actions**:

   | Finding | Decision | Filed as | URL |
   |---|---|---|---|
   | ... | filed-as-drafted \| modified \| skipped | #N or — | url or — |

   **Augmentations**:

   | Target issue | Action | Comment URL |
   |---|---|---|
   | #N | posted \| skipped | url or — |

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [character-ai](https://github.com/character-ai)
- **Source:** [character-ai/larch](https://github.com/character-ai/larch)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-character-ai-larch-audit-runs
- Seller: https://agentstack.voostack.com/s/character-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.