Audit Review Decisions

Audit Review Decisions Skill

Mine merged PR review threads for agreed-but-deferred suggestions that were never implemented. Identify review debt before it compounds.

When to Use

• User says "audit review decisions", "find deferred review items", "surface review

debt", "what did reviewers flag for later"

Arguments

• $1 — Time period (e.g. 14d, 30d, 7d). Default: 14d.
• $2 — Output path. Default:

${AUTOSKILLIT_TEMP}/audit-review-decisions/review_decisions_audit_$(date +%Y-%m-%d_%H%M%S).md

Critical Constraints

NEVER:

• Create files outside ${AUTOSKILLIT_TEMP}/audit-review-decisions/
• Have triage or validation subagents make GitHub API calls (local data only for Step 2)
• Post duplicate [AUDIT] markers — check for existing marker before posting
• Run subagents in the background (run_in_background: true is prohibited)
• Use gh pr list without --limit to avoid pagination truncation
• Use \| in Grep patterns — use | for alternation (ERE, not BRE)

ALWAYS:

• Save raw PR JSON to temp before any analysis (Step 1)
• Use GraphQL alias batching (~20 PRs per query) for data collection
• Include rateLimit { cost remaining resetAt } in every GraphQL query
• Sleep 1s between consecutive mutating GitHub API calls (Step 5 watermark posts)
• Step 2 triage subagents read local JSON files only — zero API calls
• Step 3 validation subagents grep the actual current codebase
• Skip threads that already contain an [AUDIT] comment
• Resolve owner/repo from git remote get-url origin — never hardcode
• Use /autoskillit: prefix when invoking any other skill

---

Workflow

Step 0: Watermark Resolution

1. Parse $1 for time period. Default 14d. Compute PERIOD_DAYS.

1. Resolve OWNER and REPO from git remote get-url origin.

1. Query the most recent [AUDIT] sentinel comment across recently merged PRs:

``bash gh api graphql -f query=' query($owner:String!, $name:String!) { rateLimit { cost remaining resetAt } repository(owner:$owner, name:$name) { pullRequests(first:500, states:MERGED, orderBy:{field:UPDATED_AT,direction:DESC}) { nodes { number reviewThreads(first:50) { nodes { comments(first:10) { nodes { body createdAt } } } } } } } }' -f owner="${OWNER}" -f name="${REPO}" ` Extract the most recent createdAt from any comment whose body starts with [AUDIT]. Store as LASTAUDITTS` (empty string if none — first run).

1. Compute SCAN_SINCE:

• If LAST_AUDIT_TS is set: max(LAST_AUDIT_TS, date -d "now - PERIOD_DAYS days")
• Else: date -d "now - PERIOD_DAYS days" --iso-8601=seconds

1. Log: Scan window: ${SCAN_SINCE} to now (${PERIOD_DAYS}d configured, last audit: ${LAST_AUDIT_TS:-none})

---

Step 1: Data Collection (GraphQL Batch)

1. List merged PRs in the scan window:

``bash SCAN_DATE=$(echo "${SCAN_SINCE}" | cut -c1-10) PR_NUMS=$(gh pr list --state merged \ --search "merged:>=${SCAN_DATE}" \ --json number --limit 500 | jq -r '.[].number') ``

1. Create temp directory:

``bash mkdir -p "${AUTOSKILLIT_TEMP}/audit-review-decisions/raw" ``

1. Batch fetch in groups of 20 using GraphQL aliases. For each batch, build a query

with aliased pr${i}: pullRequest(number: ${NUM}) nodes. Each node fetches: ``graphql number title mergedAt reviews(first: 100) { nodes { author { login } body state submittedAt } } reviewThreads(first: 100) { pageInfo { hasNextPage endCursor } nodes { isResolved comments(first: 100) { nodes { databaseId author { login } body path line createdAt } } } } ` Include rateLimit { cost remaining resetAt } at query root. After the initial fetch, for each PR where reviewThreads.pageInfo.hasNextPage is true, issue additional aliased queries with reviewThreads(first:100, after:$endCursor) until hasNextPage is false. Merge the nodes` arrays across pages before filtering.

1. For each PR in the batch response:

• Filter out threads whose comments list contains any comment with body

starting with [AUDIT] (already watermarked — skip entirely).

• If the PR has zero remaining threads: skip saving.
• Otherwise: save filtered data to

${AUTOSKILLIT_TEMP}/audit-review-decisions/raw/pr_${number}.json

---

Step 2: Triage (Haiku — Broad Pass)

1. List all JSON files in raw/. Split into batches of ~5 files per agent.

1. Launch parallel Haiku subagents (one per batch, model: "haiku"). Each agent:

• Reads its assigned JSON files only (no API calls).
• Flags a thread if it matches any signal:
• ` {reviewer_quote}

Current relevance: VALID — {impact}

Suggested issue title: {suggested_title} Affected files: {path}

### MEDIUM Priority
{Same structure}

### LOW Priority
{Same structure}

## RESOLVED Findings

{List: PR, file, one-line description of what was fixed}

## STALE Findings

{List: PR, file, one-line description of why no longer applicable}

## Open PR Findings

{Findings from PRs that were open (not merged) at scan time — may still be addressed.
Same per-finding structure but labeled as pending.}

## Pattern Analysis

**Most common deferral phrases (by frequency):**
{Table: phrase | count | % of all candidates}

**Dimensions with highest VALID rate:**
{Table: dimension | valid | resolved | stale | valid_rate}

**Systemic escape hatches detected:**
{Narrative: which phrases act as systematic blockers to tracking, with counts}

**Recommendations:**
{2–4 concrete process recommendations based on the pattern data}

---
6. After writing the file, print a terminal summary:
   ```
   audit-review-decisions complete
   Output: {OUTPUT_PATH}
   VALID: {N} | RESOLVED: {N} | STALE: {N}
   Top finding: {first HIGH priority suggested_title, or "none"}
   ```

---

### Step 5: Watermark (Thread Annotation)

For every finding processed in Steps 2–3 (all classifications — VALID, RESOLVED, STALE):

1. **Re-check for existing audit marker (live)**: fetch the thread's current comments
   directly from the GitHub API — do not use the Step 1 JSON cache, which already
   filtered out `[AUDIT]`-marked threads and cannot detect markers posted after Step 1:
   ```bash
   gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments" \
     --jq "[.[] | select(.id == ${COMMENT_ID} or .in_reply_to_id == ${COMMENT_ID}) | .body | startswith(\"[AUDIT]\")] | any"
   ```
   If the result is `true`: skip this thread (idempotent — no duplicate post).

2. **Determine marker body** based on classification and ticket status:

   | Classification | Ticket created? | Marker body |
   |---|---|---|
   | VALID | Yes | `[AUDIT] — tracked in #{issue_number}` |
   | VALID | No | `[AUDIT] — acknowledged, no action taken` |
   | RESOLVED | — | `[AUDIT] — verified resolved in current codebase` |
   | STALE | — | `[AUDIT] — no longer applicable` |

3. **Post reply comment**:
   ```bash
   gh api "repos/${OWNER}/${REPO}/pulls/${PR_NUMBER}/comments/${COMMENT_ID}/replies" \
     --method POST \
     --field body="${MARKER_BODY}"
   sleep 1
   ```
   `COMMENT_ID` is the `databaseId` of the first comment in the thread (from Step 1 JSON).

4. **Thread reply constraint**: These calls cannot be batched via the reviews API — each
   requires an individual POST. The 1s delay between calls is mandatory per GitHub API
   discipline.

5. Log progress per finding: `[AUDIT] Posted marker on PR #{number} thread {comment_id}: {marker_body}`

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [TalonT-Org](https://github.com/TalonT-Org)
- **Source:** [TalonT-Org/AutoSkillit](https://github.com/TalonT-Org/AutoSkillit)
- **License:** MIT

Install and usage instructions live in the source repository linked above.