Integrity Auditor

Integrity Auditor

Overview

Paper-integrity audit package. Single stage, full quality from the start. The agent reads each reference, then carries out three evidence tracks (image / numerical / logical) and produces a structured audit_report.md with Level 1–4 graded findings.

This skill ships no LLM SDK — it is the skill instructions, references, templates, and single-purpose forensics_tools/ only.

The substantive work is decomposed into reference playbooks under references/:

| Reference | Topic | |---|---| | references/00-incremental-execution.md | how to do this without losing work: batches, persistence, resume — read first | | references/01-image-evidence.md | image evidence: panel split, dup detection, rotate/flip alignment, Western-blot continuity | | references/02-numerical-evidence.md | numerical evidence: n-consistency, mean/SD/SEM recompute, P-value sanity, decimal trail, Benford with caveats, deterministic-column-pair and last-digit chi-square sweepers, variance-reporting consistency | | references/02a-supplement-acquisition.md | publisher CDN routes: how to get hi-res figures and source-data XLSXs even when the article PDF is paywalled | | references/02b-ml-paper-arithmetic.md | ML / non-biology papers: arithmetic re-derivation of every quoted improvement against tabulated benchmark cells; leaderboard archive routes | | references/03-logical-evidence.md | logical evidence: conclusion-chain compression, missing controls, replication gap | | references/04-evidence-grading.md | 4-level finding grading + reviewable-evidence format (DOI / figure-id / pointer / transformation / requested raw data) | | references/05-quality-gate.md | self-check before delivery |

Also:

• templates/audit_report.md — report skeleton the agent fills.
• forensics_tools/image_dup.py — perceptual-hash (dHash + aHash) duplicate detector for figure / panel PNGs. Single-purpose pure-Python utility (Pillow only). Catches untransformed dups.
• forensics_tools/image_dup_orb.py — ORB feature-matching duplicate detector with horizontal-flip augmentation. Catches transformed dups (rotation / flip / crop / brightness change) that perceptual hashing misses. Pair with image_dup.py: use phash first, escalate to ORB when phash distance is suspicious-but-inconclusive (16–60 range). Deps: OpenCV + NumPy.
• forensics_tools/panel_split.py — whitespace-gutter panel splitter. Pair with image_dup.py / image_dup_orb.py for cross-panel duplicate detection; whole-figure phash without panel splitting almost never finds anything.
• forensics_tools/channel_check.py — RGB channel-content classifier (DAPI / Flag / Merge / other) for fluorescence sub-images. Catches within-panel label swaps (e.g., a "DAPI" sub-image that is actually a Merge); cross-panel phash cannot catch this class.
• forensics_tools/decimal_match.py — cross-cell last-N-decimal matching sweeper for source-data XLSX. Detects fabrication where many distinct values share trailing decimal patterns (Kang Tiebang whistleblower class). Single-purpose pure-Python utility (openpyxl only). See references/02-numerical-evidence.md Check 1.5.
• forensics_tools/magnitude_consistency.py — supplement-text vs source-data XLSX unit/scale consistency. Catches unit-confusion (TWh vs GWh, mM vs µM, MHz vs Hz, etc.) and order-of-magnitude transcription errors via entity-overlap + literal-value matching across a generic SI-prefix-aware unit taxonomy covering energy / power / mass / length / area / volume / time / voltage / current / frequency / pressure / concentration / amount / force / dose / genomics-bp / CO2 / currency. Cross-family pairs (e.g., kV vs TWh) are automatically rejected. Pair with bilingual_cn_geography.json (or your own JSON map) for cross-language entity matching. Empirical baseline: Hu et al. 2026 Nature Tongyu county 1000× unit error. See references/02-numerical-evidence.md Check 1.6.
• forensics_tools/xlsx_aggregate_consistency.py — cross-XLSX same-quantity sum/row consistency. Detects when two source-data tables in the same paper purport to carry the same aggregate quantity but disagree by a small systematic margin (e.g., Hu et al. 2026 Nature MOESM3 vs MOESM6 1110.78 vs 1103.89 TWh, 0.62 percent diff, all 31 provinces same sign). Reuses the unit taxonomy from magnitude_consistency.py. Empirical baseline same paper Level 1 finding. See references/02-numerical-evidence.md Check 1.7.
• tests/smoketest.sh — /latest/ | use slug directly; also read output/experiment-suite//latest/` if present |

Compute slug:

TITLE_OR_ID=""
SLUG=$(python3 -c "import re,hashlib,sys; t=sys.argv[1]; n=re.sub(r'[\\s_]+','-',re.sub(r'[^\\w\\s-]','',t.lower().strip())).strip('-')[:40].rstrip('-'); h=hashlib.sha1(t.encode()).hexdigest()[:8]; print(f'{n}-{h}')" "$TITLE_OR_ID")
# For local-slug mode, set SLUG to the existing paper-writer slug directly.
TS=$(date +%Y-%m-%d_%H%M%S)
RUN=output/integrity-auditor/$SLUG/$TS

mkdir -p "$RUN/findings/image" "$RUN/findings/numerical" "$RUN/findings/logical"
ln -sfn "$TS" "output/integrity-auditor/$SLUG/latest"

In commands below $RUN = output/integrity-auditor//latest.

Step 2 — Gather materials into the run

Open references/02a-supplement-acquisition.md first if the input is a DOI / arXiv ID / external PDF reference. Many publishers (Nature / Springer being the canonical example) gate the article body behind a paywall while serving the supplementary tables, source data, and hi-res figures on open CDN endpoints. Never conclude "paywalled, audit blocked" before trying the supplement / figure CDN routes.

Acquisition order (try every step; do not stop early):

1. Article HTML landing page — usually open even when PDF is gated. curl -sL -A "Mozilla/5.0" "" -o $RUN/paper.html. This page typically embeds the abstract, all main-figure captions, and direct CDN URLs for every supplementary file and high-resolution figure.
2. Harvest CDN URLs from the HTML: grep -oE 'https?://[^"]*(MOESM|Fig[0-9]_HTML|mmc|MEDIA)\.(?:pdf|xlsx|docx|png)' $RUN/paper.html | sort -u. Download every distinct URL into $RUN/supplementary/ and $RUN/figures_hires/.
3. Author Correction PDF — curl -sL -A "Mozilla/5.0" ".pdf" -o $RUN/correction.pdf then file $RUN/correction.pdf to confirm it is a real PDF (not HTML auth wall).

3a. CRITICAL — the correction's own supplementary — fetch the correction's landing HTML, grep for its MOESM supplementary URLs (separate DOI namespace from the parent article), download every one. These often contain the pre-correction originals of replaced figures. See references/02a-supplement-acquisition.md for the exact recipe. Empirical baseline: the Wang Ping 2025 Nature audit only surfaced its image-track Level 4 finding because the correction's MOESM1_ESM.pdf carried the original (pre-correction) Fig 2 and Extended Data Figs 7, 10.

1. Article PDF — curl -sL -A "Mozilla/5.0" ".pdf" -o $RUN/paper.pdf then file $RUN/paper.pdf. If gated, record that fact in the manifest and continue with whatever the earlier steps acquired.
2. For a local paper-writer slug only: read the matching results.json, data_contract.md, figures/manifest.json, bibliography.bib.

Write $RUN/input_manifest.md listing every artefact actually acquired. At minimum it must enumerate what was downloaded and what was attempted-but-blocked.

Extract structured material:

# Text (every numeric claim, caption, figure reference will be greppable later)
pdftotext -layout "$PDF" "$RUN/paper.txt"

# Panels (one .png / .ppm per embedded raster image)
mkdir -p "$RUN/panels"
pdfimages -all "$PDF" "$RUN/panels/page"
N_RASTER=$(ls "$RUN/panels" 2>/dev/null | wc -l)

# Fallback for vector-figure papers (e.g., paper-writer outputs):
# pdfimages only extracts embedded raster images; a paper built from matplotlib
# vector PDFs through \includegraphics will yield zero panels here. In that case,
# render each page to a PNG so visual inspection is still possible.
if [ "$N_RASTER" -eq 0 ]; then
  pdftoppm -r 150 "$PDF" "$RUN/panels/page" -png
fi

# For a local paper-writer slug audit, also pull the production-side figure PDFs directly
# (these are the originals, before LaTeX embedding):
if [ "$INPUT_MODE" = "slug" ]; then
  mkdir -p "$RUN/figures_from_suite"
  cp output/experiment-suite/$SLUG/latest/figures/*.pdf "$RUN/figures_from_suite/" 2>/dev/null
  cp output/experiment-suite/$SLUG/latest/figures/manifest.json "$RUN/figures_from_suite/" 2>/dev/null
fi

ls "$RUN/panels" | wc -l   # record in input_manifest.md

If pdfimages / pdftotext / pdftoppm from poppler-utils is not installed, install via system package manager and retry. Do not proceed without either raster panels or page renderings — image evidence track depends on them.

If the article PDF is paywalled and the article HTML page yields no supplementary or figure CDN URLs and the Author Correction PDF is also gated, then and only then write _paywall_blocked.md per track. The Wang Ping 2025 Nature audit (output/integrity-auditor/wang-ping-hdac6-valine-10-1038-s41586-024-08248-5/latest/) demonstrates that the article body being gated says nothing about whether the supplementary source data is gated; the latter is almost always open and is where the substantive audit lives.

Step 3 — Run the three audit tracks (REQUIRED — this is the whole job)

Open references/00-incremental-execution.md first. Then carry out the three tracks below across many turns, persisting state to $RUN/ after every batch.

3.0 Paper-type triage (do this first; it picks which sweepers run)

Before invoking any sweeper or forensics tool, classify the paper. The substantive audit method differs by class — running biology sweepers on an ML paper wastes the run and pollutes the report with "swept clean" lines that are misleading (nothing was actually swept).

| Class | Signals | Image-track method | Numerical-track method | |---|---|---|---| | bio | microscopy / Western blot / fluorescence / source-data XLSX / supplementary MOESM*.xlsx | forensics_tools/image_dup.py, panel_split.py, channel_check.py per references/01-image-evidence.md | sweepers from references/02-numerical-evidence.md Checks 1–4 | | ml | schematic architecture diagrams, training-curve line plots, benchmark-score tables, no source-data XLSX, leaderboard URLs in body | figure-vs-text consistency only (forensics tools inapplicable — schematic figures lack pixel-level forensic signal) | arithmetic re-derivation per references/02b-ml-paper-arithmetic.md; sweepers do not apply (per-cell n too small for chi-square; cells heterogeneous in scale) | | other (theory / review / clinical) | no source data, no original figures | document scope-limit; usually only logical-track is informative | logical-track only; record "numerical track inapplicable" |

Quick triage heuristic (works in one line):

# If ≥ 1 XLSX in supplementary/, treat as bio; if arXiv ID + no XLSX + ≤ 5 figures, treat as ml.
N_XLSX=$(ls $RUN/supplementary/*.xlsx 2>/dev/null | wc -l)
[ "$N_XLSX" -ge 1 ] && PAPER_CLASS=bio || PAPER_CLASS=ml
echo "$PAPER_CLASS" > $RUN/paper_class.txt

Record the chosen class in $RUN/input_manifest.md so reviewers can see the methodological branch.

When a sweeper or forensics tool is inapplicable for the paper class, write findings//_inapplicable.md (NOT _clean.md) — see "Important rules" below for the semantic distinction. A _clean.md claims "I swept and found nothing"; an _inapplicable.md claims "the tool does not apply to this paper class; here is what was checked instead". Conflating the two understates audit honesty.

3.1 Image evidence

Open: references/01-image-evidence.md (for bio class). For each figure panel, check within-figure duplicates, cross-figure duplicates, transformations (rotate / flip / crop / brightness), and Western-blot background continuity. Each anomaly becomes one $RUN/findings/image/.md written in the per-finding format described in references/04-evidence-grading.md. A "no issue" outcome is recorded as _clean.md.

For ml class: forensics tools do not apply (schematic figures lack pixel-level signal). Substitute check is figure-vs-text consistency. Record as findings/image/_inapplicable.md listing which tools were considered and what consistency check was substituted.

3.2 Numerical evidence

Open: references/02-numerical-evidence.md (for bio class) or references/02b-ml-paper-arithmetic.md (for ml class). Bio class: extract every numeric claim from paper.txt with its location (section / figure / table); recompute means / SDs / SEMs against source data when available; otherwise check internal consistency (does n=6 in the body match the figure caption?); run the four sweepers (Checks 1–4). ML class: arithmetic re-derive every quoted delta / average / improvement against the table cells (Checks 1–4 are inapplicable; record as _inapplicable.md and the arithmetic re-derivation as _clean.md or as one finding per mismatch).

Each mismatch (either class) becomes $RUN/findings/numerical/.md. For a local paper-writer slug, also reconcile every number in the paper against output/experiment-suite//latest/results.json — every reported metric must trace back to a summary or runs entry.

In addition (both classes), run Check 5 — variance-reporting consistency from references/02-numerical-evidence.md: scan every table caption for restart / seed / std / error-bar mentions; flag the case where some tables in the same paper report restart-averaged numbers and others do not, when the un-averaged tables carry sub-1-pp claims. This was the BERT NSP-overclaim finding mechanism.

3.3 Logical evidence

Open: references/03-logical-evidence.md. Compress each headline claim into "A through B causes C" form. Check whether the experiments actually exercise A, B, and the A→B→C link with controls (positive / negative / rescue / dose / time). Each gap becomes $RUN/findings/logical/.md.

3.4 Grading and report assembly

Open: references/04-evidence-grading.md and templates/audit_report.md. For each finding, assign Level 1 (suspicious / could be benign), Level 2 (obvious anomaly), Level 3 (cannot resolve without raw data), Level 4 (high suspicion of misconduct). Copy the template to $RUN/audit_report.md and fill it section by section, citing each finding file by relative path. The report must include a "Requested raw data" section listing exactly what the author needs to supply for any Level 3 finding to be resolved.

3.5 Quality gate

Open: references/05-quality-gate.md. Verify: no editorial verdicts in the report, every finding has a reviewable artefact pointer, every Level ≥ 2 finding names the raw data needed to resolve it, the manifest matches what was actually examined.

Step 4 — Deliver

Report:

1. output/integrity-auditor//latest/input_manifest.md
2. output/integrity-auditor//latest/findings/{image,numerical,logical}/*.md
3. output/integrity-auditor//latest/audit_report.md
4. Stats per the report format in references/05-quality-gate.md.

Cross-skill data flow (path convention)

When the input is a local paper-writer slug, the auditor is read-only against:

• output/paper-writer//latest/paper/main.pdf — the paper under audit
• output/paper-writer//latest/paper/bibliography.bib — citation provenance
• output/experiment-suite//latest/results.json — numerical ground truth + provenance.mode (every "measured" claim in the paper must be backed here)
• output/experiment-suite//latest/data_contract.md — dataset binding (does the data actually exist; checksum recoverable)
• output/experiment-suite//latest/figures/manifest.json — figures the paper is allowed to reference (basenames only)

Never modify another skill's outputs. The audit is a third-party read.

Importa

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ai4s-research
• Source: ai4s-research/ai4s-skills
• License: MIT

Install and usage instructions live in the source repository linked above.