Reviewing Paper Sources

Reviewing Paper Sources

Overview

A paper review is an evidence-gathering exercise, not a reading exercise. The findings that change a paper come from things you compute or verify, not from things you notice while reading: recomputing the headline statistic, checking every reference against its publisher record, compiling the sources yourself, rendering the figures at print size.

Core principle: never trust an artifact you did not produce. The committed PDF is stale until proven current. A reference is wrong until checked against a primary source. A reported rate means nothing until you have its sample size.

Mode: establish this first

| | Mode A — internal | Mode B — external | |---|---|---| | Situation | Advisor/self-review of your own group's draft | Referee report for a venue | | Register | Blunt, shared stake in the outcome | Neutral, addressed to authors + chair | | Proposes fixes | Yes, including edits to the sources | No — diagnose only, never rewrite the authors' text | | Artifacts | Review .md + .pdf, *-annotated.tex, refs-corrected.bib | Review .md + .pdf only | | Scores/verdict | Yes (both modes) | Yes |

If it is ambiguous, ask. "My student's paper" ⇒ Mode A. "I'm reviewing for X" ⇒ Mode B.

Also establish where in its lifecycle the paper is, because it changes what the review is for: pre-submission (triage by what fits the deadline), already submitted (target what reviewers will attack, and prepare answers), or post-rejection (framing and structural weaknesses).

Identify the real artifact before reviewing

The paper is what was submitted, and the repository may not contain it. Before writing anything, reconcile:

• A commented-out \input does not mean the content is absent. Sections get consolidated: check where each heading actually lives (grep -n '^\\section' sections/*.tex) before concluding anything is missing. One file routinely holds two top-level sections. A file named eval_*.tex that nothing includes is usually a superseded draft, not a lost result.
• Map the built structure against the submitted PDF heading by heading, then compare word multisets per page. Differences that are only ligatures (di+erent for different) or missing figure-embedded numerals mean the submitted PDF was re-saved through a viewer (check pdfinfo Producer for Quartz, etc.) and its text layer, not its content, differs.
• Diff the reference lists. A name present in the submitted bibliography but absent from yours is either a citation removed post-submission or a phantom entry from a stale .bbl. Both are findings.
• If the repository still cannot produce the submitted paper, ask for the submitted PDF rather than reviewing a reconstruction. Say what you would otherwise be guessing at.

Workflow

Create a todo per phase. Do not skip phases 1–4 to get to the writing.

0. Scope the review. Before any work, present the optional modules with their costs and ask which to run. Record the answer in the review header so the artifact states its own coverage. Ask once; do not re-ask mid-review.

| Module | Skill | Cost | |---|---|---| | Core review | (this skill) | always | | Verify bibliography | verifying-bibliography | minutes for a 57-entry file, seconds once cached | | Find missed related work | surveying-literature (gap sweep) | ~1.5 min for an 18-entry bibliography, ~90 API calls | | Map the field | surveying-literature (--field-map TOPIC) | similar | | Deep paper comparison | comparing-papers | ~1 min per paper, fetches LaTeX source | | Check the mathematics | verifying-proofs | seconds for a 46-proof corpus; no dependencies in its default mode | | Expand a derivation | explaining-derivations | one subagent and one PDF per theorem; minutes each |

Before running a gap sweep, run it with --seeds-only first: it makes no requests and shows exactly which queries will be used. Wrong angles mean wrong results, and OpenAlex bills each search against a small daily budget.

Offer bibliography verification pre-selected: measured against two bibliographies with known ground truth, it has the highest finding-rate per minute of any phase, and it found a defect a careful manual pass had missed.

1. Ground truth. Compile the sources yourself into a scratch dir: latexmk -pdf -interaction=nonstopmode -outdir=/tmp/pbuild main.tex Then compare against any committed PDF. A stale committed PDF is a finding in itself — it can carry the wrong title, or real author names in a double-blind submission. Record page count, LaTeX errors, undefined citations/references, and overfull boxes. Also list the directory: unused files whose names identify authors (figure_JH.pdf) are an anonymity risk worth reporting.

If it does not compile, fix it before reviewing — a review of a document you could not build is worth little. Then ask which artifact is authoritative before writing anything (see Identify the real artifact).

Delete stale .bbl before testing a bibliography. \bibliography{...} inputs ./main.bbl, so a leftover .bbl in the source directory silently shadows your test even with -outdir. If citations resolve suspiciously well, or no .blg appears, that is what happened. This also explains a class of real defect: a submission built past a fatal error against an old .bbl ships a bibliography that no longer matches its own citations.

2. Venue rules. Find the CFP (WebSearch/WebFetch) and check: page limit and whether references count, anonymity requirements, format/template, deadline. The deadline determines how you triage. Report compliance explicitly.

Read the template's own comments — they list the commands that block publication. grep '^%' .sty main.tex typically yields lines like "\nocopyright -- Your paper will not be published if you use this command". Collect them, then grep the comment-stripped source for live usage. Check \vspace too: negative vertical space to win room is the most common silent violation. Verify the page split by extracting text per page rather than trusting the page count — "7 pages of content, 9 total" means the References heading must fall at the top of page 8, and that is a one-command check.

3. Read the rendered pages, not just the source. pdftoppm -r 110 -png main.pdf /tmp/pg, then read the images. This is the only way to catch illegible figures, encoding inconsistencies, and space allocation. Estimate what fraction of the page budget the figures consume.

Extract the text layer of every figure — pdftotext Figures/*.pdf -. Plots are generated by scripts, so their legends contain series the prose may never mention. On one submission this surfaced a fourth series in the headline scalability figure — a CPU build of the authors' own engine — whose name appeared zero times in the .tex. It was the ablation separating the paper's GPU claim from the effect of reimplementation alone, and no reader could have known it existed. Compare the series names against the text and ask about any that are unexplained.

4. Audit the claims. For every headline number: find its sample size, recompute the statistic, and check whether the comparison is licensed. Read the primary sources for numbers the paper quotes from others — protocols rarely match. See reference/claim-audit.md.

For the baseline-scale and seed-count checks specifically, invoke comparing-papers: python3 ~/.claude/skills/comparing-papers/assets/run-compare.py . --against "" --out review-assets/ It fetches the baseline's LaTeX source, extracts updates x batch against the baseline's own published scale, and computes the ratio with both quotes. On one real paper this reproduced a critical finding (6.4M vs 60M examples, ~11%) that had taken a manual pass through two appendices.

4b. Check the mathematics. For any paper with theorem or proof environments, invoke verifying-proofs: python3 ~/.claude/skills/verifying-proofs/assets/run-proofcheck.py main.tex --out review-assets/ Its default mode needs no external tooling and reports what the structure of the argument gives up: an induction with no base case, a claim dependency cycle, a restatement that drops a hypothesis, a division by something nobody proved non-zero. Read the coverage table before the findings — "54 of 138 inference steps were mechanically checkable" is usually the more important number, and a dense cluster of UNVERIFIED inside one proof is a finding in itself.

Then do what the tool cannot: reference/structural-audit.md in that skill is the non-mechanical half — whether the hypothesis is used, whether quantifier order survives, whether the induction covers its claim.

When a specific derivation is load-bearing and you cannot follow it, invoke explaining-derivations. A step nobody can make explicit is evidence against the derivation, and its gap ledger converts that into review findings with severities.

5. Audit the bibliography — every entry, no exceptions. Invoke verifying-bibliography: python3 ~/.claude/skills/verifying-bibliography/assets/run-bibcheck.py refs.bib --bbl main.bbl --out review-assets/ Consume bibcheck-report.md, then do what the tool cannot: read each load-bearing citation's abstract and check the submission's characterisation against it (selective citation), and write refs-corrected.bib. See reference/bibliography-audit.md for the audit-table format and the corrected-bib rules.

WEAK and UNVERIFIED rows are not passes — they are the entries a human must check, and a fabricated reference is exactly the one no database can find. Run the checker over the whole file, not only the cited subset: uncited entries are also unaudited.

This is the phase most likely to be skipped and most likely to produce a finding.

6. Write the review into review--.md using reference/review-template.md, then render it and every audit report:

MD2PDF=~/.claude/skills/_shared/md2pdf/md2pdf
"$MD2PDF" --review review--.md
"$MD2PDF" --review review-assets/*.md        # bibcheck, proofcheck, gaps, comparison

--review breaks the metadata block per label, turns the repository-state blockquote into a callout box, and tightens the audit tables. It is one self-contained script, so copying just that file into review-assets/ keeps the build reproducible after the skill is gone.

Read what it prints. A character no font can render is dropped silently by the engine — the sentence then reads as though you never wrote it. md2pdf reports every dropped character; treat that warning as a defect in the review, not noise. It also says when it had to degrade something (a quoted macro typeset literally, math typeset as source text).

Do not hand-tune the LaTeX for wide tables. Wide tables already step down a font size and long DOIs already get break opportunities inserted; a table still running off the page means the content needs splitting, not the preamble.

7. Annotate the sources (Mode A only). Produce main-annotated.tex with changes.sty markup and a corrected .bib. See reference/annotating-with-changes.md — check the document class for a forbidden-package list first, and expect the [final] accept-all build to be less reliable than the markup build. Render the marked-up pages and look at them; markup that compiles can still be garbage.

8. Verify, then offer both formats. Every artifact compiles; every claim in the review traces to something you ran or read.

Refute your own findings before delivering. Dispatch one subagent with the finished review and the sources, instructed to refute each critical and major finding and to default to "refuted" under uncertainty. Every surviving objection must be visibly resolved — the finding is amended, or it is kept with the reason the objection fails. An objection neither answered nor recorded is the one that comes back in the authors' response.

Two things this needs to be told, both learned from running it:

• Adjudicate against the artifact the review pins, not the working tree. On the

one review this was tried against, the tree had moved one commit past the reviewed artifact and already conceded five of seven findings. Judging a review by revisions made in response to it is circular, and the refuter has to be told so or it will report those concessions as refutations.

• **Have it check the review's own arithmetic and labels, not only its

conclusions.** The value showed up inside sound findings rather than against them: 0 of 7 findings were overturned, and 3 carried a wrong sub-claim — a confidence interval attributed to the wrong method, a "the metric cannot fail by construction" assertion contradicted by the paper's own figure, and a parenthetical whose numbers could not average to the figure they were said to average to. Each would have been corrected by an author, and a review's authority rests on getting exactly that kind of thing right.

Nothing else in the workflow re-checks the review's own statistics — phase 4 audits the paper's arithmetic, not yours.

Close by listing what exists, .md and .pdf side by side, so the user can read it either way without asking — the .md is what they will edit and diff, the .pdf is what they will circulate or annotate:

review-claude-20260815.md   review-claude-20260815.pdf   (8 pp)
review-assets/bibcheck-report.md   .../bibcheck-report.pdf   (2 pp)

Offer to open the PDF. Report page counts: a review that ran to 30 pages is a finding about the review, not about the paper.

Hard rules

• Never modify the originals. main.tex and refs.bib are inputs. Write main-annotated.tex, refs-corrected.bib. If you must test a corrected bib against the real build, back up, test, restore, and say so.
• Never install tooling without asking. Missing pandoc/LaTeX is a question for the user, not a brew install.
• Never commit anything. Leave the artifacts untracked; integration is the author's call.
• Every number in the review must be one you computed or read from a source. No "approximately" derived from memory.
• Report what you could not verify. An unverifiable reference or an anonymous artifact is a stated limitation, not a silent pass.
• A disqualifying finding sets a floor the scores cannot lift. Never sum or average the §2 criteria. If soundness of evaluation fails, the recommendation follows soundness, whatever novelty scored — and the §2 disqualifying line must name every finding that gates the outcome. Across five reviews on file the verdict line once named M1–M3 while its own prose named a fourth item carrying desk-reject risk; a reader acting on the bold line would have submitted with it outstanding.

Findings that recur

Run this list against every quantitative paper; each is expanded in reference/claim-audit.md: sample size behind a 0%/100% rate (rule of three) · protocol match to each baseline · baseline training scale vs its published scale · seed count · self-built baselines · ablations for the claimed mechanism · metric saturation by construction · selection on the reported construct · per-item denominators hidden inside an aggregate · unevaluated contributions · cost of the method · selective citation.

Read the commented-out text (grep -n '^\s*%' sections/*.tex). Authors delete their own caveats under page pressure, and a caveat they wrote and cut is the strongest recommendation you can make — you are asking them to restore their own sentence, not accept yours.

Common mistakes

• Reviewing the committed PDF. It may not correspond to the sources. Compile first.
• Spot-checking the bibliography. In practice errors cluster in the references the authors were least likely to re-read — the classic ones. Sampling misses them.
• Clearing an entry after checking authors only. Check authors, title, venue, volume/issue, pages, and DOI. A correct DOI beside a wron

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: chgagne
• Source: chgagne/claude-skills-research
• License: MIT

Install and usage instructions live in the source repository linked above.