Audit Engine

> Orchestration Log: When this skill is activated, append a log entry to outputs/orchestration_log.md: > `` > ### Skill Activation: Audit Engine > **Timestamp:** [current date/time] > **Actor:** AI Agent (audit-engine) > **Input:** [paper + repo being audited] > **Output:** [brief summary — e.g., "Audited 18 claims: 12 CONFIRMED, 3 PARTIAL, 2 MISSING, 1 MISMATCH"] > ``

Audit Engine

Core Principle

Papers make claims. Code embodies what was actually done. This engine systematically checks whether the two agree. For every empirical or technical claim in the paper — datasets used, models trained, metrics reported, hyperparameters set, ablations run — the engine locates supporting evidence in the linked repository and classifies the match.

This is the complement to verification-engine, which checks citations against external sources. Audit-engine checks the paper's own claims against the paper's own code. Together they cover both failure modes of LLM-assisted writing: mis-cited prior work and unsupported own-work claims.

Inspired by the /audit command in the Feynman research agent (Companion AI, 2026), adapted to the IS/CS methodological style of this plugin.

When to Activate

• User says "audit the paper", "check paper vs. code", "verify my experiments",

"reproducibility audit", "does the code match what I wrote"

• Before submitting a paper with an accompanying code release
• Before open-sourcing the repo of a published paper
• When reviewing someone else's paper + artifact
• As optional Phase 7.5 of the paper machine pipeline (after verify-citations,

before prepare-submission)

When NOT to Activate

• The paper has no code artifact (pure theory, position paper, qualitative study

without computational analysis) → say so and exit

• The user wants to verify citations → activate verification-engine instead
• The user wants to check writing quality → activate the writing-engine

/analyze-writing command

---

Inputs

Required:

1. Paper source — paper.tex, draft.md, or explicit $ARGUMENTS path
2. Code repository — one of:

• Local path (./experiments/, ~/repos/myproject)
• GitHub URL (clone or use gh repo view / WebFetch on raw files)
• Archive link (Zenodo, OSF) — ask user to download locally first

If the repo location is not supplied, scan the paper for common signals:

• "Code available at [URL]" / "Our implementation is at [URL]"
• GitHub URLs in footnotes or acknowledgements
• A code_availability section
• A REPRODUCIBILITY.md, ARTIFACT.md, or similar file sibling to the paper

If still not found: ask the user once, then exit.

---

Step 1: Extract Auditable Claims

Scan the paper for claims that can be checked against code. Ignore claims that are purely conceptual, historical, or theoretical.

Claim Categories (check in order)

| Category | What to look for | Priority | |----------|------------------|----------| | Dataset | Named datasets, split sizes, sample counts, data sources | HIGH | | Model | Model names, architectures, parameter counts, checkpoints | HIGH | | Training | Epochs, batch size, learning rate, optimizer, hardware | HIGH | | Metrics | Reported numbers (accuracy, F1, BLEU, loss values, percentages) | HIGH | | Experiments | Named experimental conditions, ablations, baselines | HIGH | | Hyperparameters | Specific values in tables or "Training Details" | MEDIUM | | Preprocessing | Tokenization, normalization, filtering steps | MEDIUM | | Evaluation | Test protocol, prompt templates, judge models, seeds | MEDIUM | | Infrastructure | GPUs, training time, framework versions | LOW | | Figures | Plots claimed to come from "our experiments" | MEDIUM |

Extraction Pattern

For each claim, record:

{
  id: "C01",
  category: "Model",
  section: "4.2 Model Training",
  claim_text: "We fine-tune LLaMA-3-8B for 3 epochs with a learning rate of 2e-5.",
  testable_facts: [
    "model == LLaMA-3-8B",
    "epochs == 3",
    "learning_rate == 2e-5"
  ],
  priority: "HIGH"
}

Claims with concrete numbers, names, or identifiers are testable. Vague claims ("we use a standard transformer") are not auditable — mark them as NOT_AUDITABLE and skip.

Output: outputs/audit_claims.md — numbered list of all testable claims.

---

Step 2: Map the Repository

Before searching, build a lightweight mental map of the repo. Do not read every file.

1. Top-level listing — Glob on **/*.{py,ipynb,yaml,yml,json,toml,sh,md}

at depth 2-3

1. Identify key files by name convention:

• train.py, main.py, run_experiments.py, eval.py → entry points
• config.yaml, hparams.json, sweep.yaml, *.toml → configuration
• requirements.txt, pyproject.toml, environment.yml → dependencies
• README.md, REPRODUCE.md, docs/ → documentation
• results/, outputs/, logs/, wandb/ → experiment artifacts
• datasets/, data/, load_data.py → data loaders

1. Detect framework — PyTorch, JAX, TensorFlow, HuggingFace, scikit-learn —

this guides search patterns

1. Detect experiment tracking — wandb, mlflow, tensorboard, plain CSV logs

Record this as an internal map; do not output it unless the user asks.

---

Step 3: Search for Evidence (per claim)

For each testable claim, systematically search for supporting code evidence.

Search Strategy

Use Grep and Read — NOT an agent — for transparency. Each lookup should produce a file path and line number that can be cited in the report.

Example — Model claim "LLaMA-3-8B, 3 epochs, lr=2e-5":

1. Search for the model name: Grep "llama-?3-?8b|Llama-3-8B" --type py
2. Search for learning rate: Grep "2e-?5|0.00002|learning_rate.*2e-5"
3. Search for epochs: Grep "epochs\s*[:=]\s*3|num_epochs.*3"
4. Check config files: Read config/*.yaml for matching values
5. If wandb/mlflow logs exist, grep those too

Example — Metric claim "we report an F1 of 0.87":

1. Search results files: Grep "0\.87" --type json --type csv --type md
2. Search eval scripts: Grep -l "f1_score|F1" eval*.py
3. Check if the number appears in a logged output

Example — Dataset claim "trained on 12,000 examples from OpenReview":

1. Search for dataset loader: Grep "openreview" -i
2. Check size assertions: Grep "12000|12_000|len\(.*\).*12"
3. Read the data loading function to confirm source

Record Evidence

For each claim, record:

{
  id: "C01",
  searches: ["llama-3-8b", "lr=2e-5", "epochs=3"],
  hits: [
    {file: "train.py", line: 42, snippet: "model_name = 'meta-llama/Llama-3-8B'"},
    {file: "config/train.yaml", line: 7, snippet: "learning_rate: 2e-5"},
    {file: "config/train.yaml", line: 8, snippet: "epochs: 5"}  // NOTE mismatch
  ]
}

Do not hallucinate hits. If Grep returns nothing, record an empty hits list.

---

Step 4: Classify Each Claim

Classification Rubric

| Status | Criteria | Evidence | |--------|----------|----------| | CONFIRMED | Every testable fact in the claim has matching code evidence | File + line for each fact | | PARTIAL | Some facts confirmed, others missing or unchecked | Confirmed facts listed; gaps called out | | MISSING | No code evidence found for any fact in the claim | Which searches returned empty | | MISMATCH | Code evidence exists but contradicts the claim | Side-by-side: paper says X, code says Y | | NOT_AUDITABLE | Claim is too vague to check, or code is not available | Brief reason |

Rules of Engagement

• Be conservative. If you're not sure a search hit actually supports the claim,

mark PARTIAL and explain what's missing.

• Never mark CONFIRMED without a file:line reference. "I think it's probably

in the training script" is not evidence.

• Treat MISMATCH as load-bearing. Even a single MISMATCH is worth flagging

prominently — these are the findings the user most needs to know.

• Distinguish MISSING from NOT_AUDITABLE. MISSING means the claim is

checkable but no evidence exists (red flag). NOT_AUDITABLE means the claim itself is too vague (usually fine, but worth rewriting).

• Do not run code. This engine is a static audit. Running experiments is the

job of a separate replication step (future extension point).

---

Step 5: Generate Audit Report

Save to outputs/audit_report.md.

Report Template

# Paper-vs-Code Audit Report

**Paper:** [paper title]
**Paper source:** [paper.tex | draft.md | path]
**Code repository:** [local path or URL]
**Commit / version audited:** [git SHA if available, else "working tree"]
**Date:** [YYYY-MM-DD]
**Auditor:** audit-engine (Open Academic Paper Machine v6.4)

## Summary

| Status | Count | % |
|--------|-------|---|
| CONFIRMED | [n] | [%] |
| PARTIAL | [n] | [%] |
| MISSING | [n] | [%] |
| MISMATCH | [n] | [%] |
| NOT_AUDITABLE | [n] | [%] |

**Overall signal:** [one sentence — e.g., "Core experiments check out; 2 mismatches
in reported hyperparameters need attention before submission."]

## Critical Findings

### MISMATCH — Paper and Code Disagree

#### [C03] [Section 4.2] "We train for 3 epochs with learning rate 2e-5"
- **Paper says:** epochs = 3, lr = 2e-5
- **Code says:**
  - `config/train.yaml:8` → `epochs: 5`
  - `config/train.yaml:7` → `learning_rate: 2e-5`  ✓
- **Recommendation:** Update the paper to say 5 epochs, or re-run with 3 and
  re-check the reported numbers.

[repeat for each mismatch]

### MISSING — Claim Not Supported by Code

#### [C07] [Section 5.1] "We evaluate on the held-out 10% split (1,200 examples)"
- **Paper says:** held-out split of 1,200 examples
- **Searched:** `held.out`, `test_split`, `1200`, `0\.1`
- **Result:** No split logic found in `data/loader.py`. `eval.py` loads the entire
  dataset without stratification.
- **Recommendation:** Either add split logic to the repo or remove the claim from
  the paper.

[repeat for each missing]

## Detailed Findings by Section

### Section 4 — Method

| # | Claim | Category | Status | Evidence |
|---|-------|----------|--------|----------|
| C01 | LLaMA-3-8B model | Model | CONFIRMED | `train.py:42` |
| C02 | 2e-5 learning rate | Training | CONFIRMED | `config/train.yaml:7` |
| C03 | 3 epochs | Training | MISMATCH | see Critical Findings |
| C04 | AdamW optimizer | Training | CONFIRMED | `train.py:88` |

### Section 5 — Experiments

| # | Claim | Category | Status | Evidence |
|---|-------|----------|--------|----------|
| C05 | F1 = 0.87 on test set | Metric | CONFIRMED | `results/test_metrics.json:12` |
| C06 | 3 random seeds | Training | PARTIAL | seeds {42,43} found in `run.sh`, third seed unclear |
| C07 | Held-out 10% split | Dataset | MISSING | see Critical Findings |

[repeat for each section]

## Not Auditable

Claims that are too vague to check or depend on external state:

| # | Claim | Reason |
|---|-------|--------|
| C12 | "A standard transformer architecture" | Vague — no specific config to check |
| C15 | "Comparable to human performance" | Depends on external human benchmark |

## Repository Map (for context)

- **Entry points:** `train.py`, `eval.py`, `run_sweep.sh`
- **Configs:** `config/train.yaml`, `config/eval.yaml`
- **Data loading:** `data/loader.py`
- **Dependencies:** `requirements.txt` (53 packages)
- **Experiment tracking:** wandb (runs in `wandb/`)
- **Results:** `results/` (JSON + CSV logs)

## Methodology Note

This audit was a **static audit**: code was read and grepped, but no experiments
were re-run. Matches indicate that the paper's claims are consistent with the code
*as written*, not that the code was verified to produce the reported numbers.

A full reproducibility check would additionally:
1. Install dependencies in a clean environment
2. Re-run training with the exact config
3. Re-run evaluation and compare numbers to the paper

See Extension Points below.

## Extension Points

- **Dynamic replication:** re-run the training/eval pipeline in a Docker container
  and compare metrics to the paper (Feynman-style `/replicate`, not yet implemented
  in this plugin)
- **Diff-based re-audit:** after fixing mismatches, re-run on changed claims only
- **Multi-repo audits:** audit a paper that uses code from multiple repos (e.g.,
  baseline models from external sources)

---

Prioritization Strategy

If the paper has many claims, process in this order and save intermediate results after each tier so the user can act early:

Tier 1 — High-stakes claims (verify first)

• Reported numbers in tables and abstract
• Model identifier and size
• Dataset identifier and size
• Any claim the paper's contribution depends on

Tier 2 — Method details

• Hyperparameters, optimizers, schedulers
• Preprocessing, tokenization
• Evaluation protocol

Tier 3 — Context

• Infrastructure (GPUs, training time)
• Framework versions
• Figure provenance

After Tier 1, present a progress snapshot:

AUDIT PROGRESS
Tier 1 (High-stakes):  [12/12] Complete
  CONFIRMED: 9 | PARTIAL: 1 | MISSING: 1 | MISMATCH: 1
  >> 1 mismatch found — see audit_report.md

Tier 2 (Method):       [0/8]  Queued
Tier 3 (Context):      [0/4]  Queued

Continue to Tier 2? [proceeding unless you redirect]

---

Limitations

What This Engine Can Do

• Check named entities (datasets, models) against code references
• Verify hyperparameter values in configs match the paper
• Confirm metrics appear in logged results
• Catch contradictions between paper and code (MISMATCH)
• Catch unsupported claims (MISSING)
• Flag vague claims that should be rewritten (NOT_AUDITABLE)

What This Engine Cannot Do

• Run the code and reproduce the numbers (static audit only)
• Detect whether the code would actually produce the claimed behaviour if run
• Audit closed-source or paywalled dependencies
• Audit manual/qualitative steps ("we prompted GPT-4 iteratively until...")
• Replace a human artifact reviewer

Failure Modes to Watch For

• Config vs. code divergence — the YAML says one thing, train.py hard-codes

another. Always check both.

• Scripts that override configs — run.sh may pass --epochs 10 that overrides

the YAML. Grep shell scripts too.

• Multiple configs — papers often report one experiment; the repo has ten.

Match the config to the paper, not the other way around.

• Notebooks — .ipynb files are JSON; Grep works but read carefully.

---

Relationship to Other Skills

• verification-engine — checks external citations against sources.

audit-engine checks own-work claims against own code. Run both before submission for full coverage.

• review-engine — simulated peer review. An audit report is a natural

input to a simulated reviewer.

• prepare-submission — a clean audit report is a strong artifact to include

with submissions to venues that accept reproducibility statements (e.g. NeurIPS Reproducibility Checklist, ML Reproducibility Challenge).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: TobiasBlask
• Source: TobiasBlask/open-paper-machine
• License: MIT

Install and usage instructions live in the source repository linked above.