Paper Diff

paper-diff

1. Purpose and scope

This skill produces the user-facing comparison artifact between the figures of the source paper and the rendered output of a Mobius simulation. It is the visible evidence that the simulation reproduces the physics described in the paper. Its product backs the green "reproduces Figure 3" badge surfaced near the simulator and the side-by-side panel that opens when a user clicks the badge.

The skill runs after primitive-generator has emitted Python code, after the simulation has executed in the Claude Managed Agents sandbox, and after at least one frame per relevant figure has been exported. It runs before the orchestrator pitches the final assembled simulator. It does not run during the live slider loop; the diff is computed once per stable simulation snapshot and cached.

What this skill does NOT do. It does not extract figures from the paper PDF (that is paper-parser), does not decide whether to block the pipeline (that is science-integrity), does not render the side-by-side panels in the browser (that is viz-mapper), and does not invent ground-truth values.

2. Distinction from science-integrity.figure_diff

Load-bearing. Both skills compare images but serve orthogonal purposes.

science-integrity.figure_diff is one of the five deterministic integrity checks (units, CFL, conservation, figure-diff, claim-match). It returns pass | warn | fail, writes into simspec.integrity.checks[], and on fail blocks the pipeline so the orchestrator re-dispatches primitive-generator. It is silent on the user surface.

paper-diff is the user-facing report. It returns a rich structured object enumerating matched features, missing features, spurious features, color and axis agreement, and surviving numerical claims. It is always rendered in the UI and never blocks. Even on catastrophic disagreement, the simulator renders with a red "divergent" badge and a panel listing every divergence verbatim. The bet: an honest "this does not match Figure 3 because the simulation produced 2 peaks where the paper shows 4" is more valuable to a user and a judge than a silently downgraded match.

The two skills share the underlying VLM call shape and the deterministic-metric primitives, but their output schemas, blocking semantics, and rendering pathways are different. science-integrity.figure_diff is a gate; paper-diff is a report.

3. Citation of the SINTEF gap

JutulGPT (arXiv 2603.00214, SINTEF Digital + NTNU, MIT licence) is one intellectual ancestor of Mobius. In Section 6.1 the authors enumerate explicit limitations of their reference implementation. Two are directly relevant:

> "the agent does not yet perform structured comparison of simulation outputs"

> "systematic multimodal output comparison... remains an important development direction"

These are verbatim from page 11 / 19 of the v1 manuscript. paper-diff exists exactly to occupy this gap. JutulGPT relies on the simulator's internal conservation tolerances as its sole validity certificate - "if it ran, it is physical". Mobius adds a second certificate: "and what came out matches the paper's published figures, structurally and numerically, at the feature level". The pairing is the moat; neither half alone is novel.

This is also the primary citation hook for the hackathon pitch: SINTEF flags multimodal output comparison as future work in February 2026; Mobius ships it in April 2026 with Opus 4.7 vision.

4. Inputs

For each figure id in simspec.paper.figures[] the skill expects three artifacts.

ref_path. The reference PNG extracted from the paper PDF by paper-parser, under artifacts/figures/.png. Resolution is whatever the PDF provides; the skill rescales.

sim_path. The simulation frame for the same figure, exported by the execution stage. The orchestrator picks the frame closest to the figure caption's time anchor ("steady state", "t = 5 ns", "first 100 cycles average"). Multi-panel figures pair frames one-to-one.

claims. The subset of paper.numerical_claims[] whose section anchors to this figure (Fig. 3, Fig. 3a). Each carries value, UCUM unit, confidence, and verbatim source quote. The skill uses them to ground the VLM: rather than "do these look similar", we ask "does the simulation reproduce the quantities this figure claims".

Optional mode flag: "fast" skips the VLM stage (used during the slider loop); "full" is the default.

5. Output

A paper_diff_report object per figure, one entry per panel. The shape:

{
  "figure_id": "fig-3",
  "similarity_score": 0.82,
  "panels": [
    {
      "label": "panel-a",
      "ref_path": "artifacts/figures/fig-3.png",
      "sim_path": "artifacts/sim_frames/fig-3_panel-a.png",
      "similarity_score": 0.84,
      "features": {
        "present":  ["five interference peaks", "central maximum brightest", "envelope falls off symmetrically"],
        "missing":  ["axis labels in micrometers"],
        "spurious": ["faint vertical banding from FFT aliasing"]
      },
      "dominant_color_match": true,
      "axis_scale_match": true,
      "peak_count_match": true,
      "claim_overlay": [
        { "claim": "fringe spacing 0.63 mm at 1 m screen distance", "present_in_sim": true,  "value_diff_percent":  1.8 },
        { "claim": "first minimum at 0.31 mm from center",          "present_in_sim": true,  "value_diff_percent": 12.4 }
      ]
    }
  ],
  "summary_md": "Five visible peaks, central maximum brightest, fringe spacing within 2% of paper. First-minimum location off by 12%, likely due to coarser sampling at the aperture edge.",
  "badge": "match"
}

badge thresholds: "match" (similarity >= 0.80 and no missing critical feature); "approx" (0.60 extractor — keep them equal). details[]` is recommended and carries the per-figure rich report shape from §5; the UI renders it verbatim.

If ref_path or sim_path is missing, short-circuit Stage 1+2 and emit a fenced JSON with vlm_status + score: 0.0:

{
  "ssim": 0.0, "vlm_judgment": "uncertain", "diff_score": 0.0, "score": 0.0,
  "details": [], "vlm_status": "missing-input",
  "error": "sim_path /runs//scene.png does not exist"
}

DO NOT emit narration, summaries, or follow-up questions after the closing fence — they break the orchestrator's downstream consumption and get silently dropped.

6. Two-stage diff

Deterministic first, VLM second. Cheap before expensive, in the order that maximises explainability.

Stage 1: deterministic image metrics. Over the rescaled image pair (512 x 512):

• SSIM via skimage.metrics.structural_similarity on the luminance channels, mapped to [0, 1].
• Histogram overlap on RGB via numpy.histogram with 32 bins per channel, summed per-bin min. Catches dominant-color agreement independently of spatial layout.
• Edge IoU via skimage.feature.canny, intersection over union of edge pixels. Catches axes and dominant outlines.

Stage 1 score = 0.5 * ssim + 0.25 * hist + 0.25 * edges. Bounded, sub-second, never hallucinates.

Stage 2: Opus 4.7 vision. One multimodal call to client.messages.create with two image content blocks (ref and sim, base64-encoded PNG, media_type: "image/png") and a text block carrying the schema and claim list. Model: claude-opus-4-7. Output is strict JSON matching the panel schema.

If the VLM call fails (network, rate limit, parse error), the report falls back to Stage 1 only with "vlm_status": "unavailable" plus the error string. The pipeline never crashes on VLM failure.

7. VLM prompt (verbatim)

The exact system prompt used in Stage 2. ~200 words, no slack:

> You are a structural-fidelity comparator for physics simulation outputs. You receive two images: a reference figure from a published paper, and a frame produced by a simulation that claims to reproduce it. Your job is to enumerate, in strict JSON, every structural feature the two images share, every feature present in the reference but missing in the simulation, every feature present in the simulation but spurious. You also report whether dominant colors match, whether axis scales match, and whether the count of dominant peaks or features matches. You compare to a list of numerical claims; for each, mark whether it is presentinsim and the valuediffpercent if a numerical value can be read from the simulation. > > Forbidden: artistic judgment. Do not say "the simulation looks ugly" or "the colors are not as nice". You are not an art critic. You are reporting structural physics fidelity. Forbidden: speculation beyond what the images show. Do not say "this is probably caused by". Forbidden: prose outside the JSON. The response MUST be a single JSON object matching the panel schema in the user message; no preamble, no postscript, no markdown fences. > > If you cannot determine a field from the images, set it to null and note it in features.missing with the prefix "uncertain:".

The user message carries the schema fragment, the claim list, and a 30-word context blurb pulled from the figure caption. Temperature is 0; we do not want creativity in a structural diff.

8. Caching

Cache key: sha256(ref_bytes || sim_bytes || claims_canonical_json || prompt_version). Cached entries at /tmp/mobius_paper_diff_cache/.json. A hit short-circuits both stages. prompt_version is bumped manually whenever the system prompt changes, invalidating old entries cleanly.

The cache is content-addressed: sliders changing parameters produce new sim bytes and new cache misses, exactly the desired behavior. Reference figures rarely change within a session.

Eviction: 200 MB or 500 entries, LRU. The directory is gitignored.

9. Visual integration

The report renders as a Ciechanowski-style side-by-side panel:

• Left: paper reference on --c4 rose-tinted matte, 1.5 px stroke, IBM Plex Sans caption.
• Right: simulation frame on --c3 blue-tinted matte, identical aspect ratio, caption "Mobius - reproduced".
• Center column of annotations, one row per features.present (green tick --c2), features.missing (orange dash --c1), features.spurious (red cross --c5). Hover highlights the corresponding region in both images.
• Bottom: summary_md as body-type prose, plus an accordion expanding the claim_overlay table.

The badge is a pill inline with the simulator title bar: --c2 match, --c1 approx, --c5 divergent. Click opens the panel as a modal. All easing follows the project's quintic smoothstep x*x*x*(x*(x*6-15)+10).

10. Examples - three worked diff scenarios

Match. Young's double-slit. Paper shows five fringes with brightest central maximum, fringe spacing 0.63 mm at 1 m. Sim reproduces five fringes, fringe spacing 0.642 mm. SSIM 0.91, hist 0.88, edges 0.86. VLM: "five interference peaks, central maximum brightest, envelope symmetric". claim_overlay fringe spacing differs by 1.8%. Badge: match.

Approx. Damped harmonic oscillator. Paper omega_0 = 2.0, gamma = 0.1. Sim 1.94 / 0.097, 3% error from coarser dt. SSIM 0.74. VLM: "correct exponential envelope, period off by 3%". Badge: approx. The user sees the honest "off by 3%" pill.

Divergent. Four-slit grating. Paper shows four primary maxima per order; sim, due to a primitive defaulting slit_count = 2, shows two. SSIM 0.41. VLM: "two primary maxima where four were expected; missing slits 3 and 4". Badge: divergent. The orchestrator may re-dispatch primitive-generator with the diagnostic; the user sees an unedited red pill explaining the gap.

11. Red flags

Never lie about a match. If similarity_score .png`; paper-diff reads, never modifies.

primitive-generator indirectly produces sim frames via executed code; paper-diff sees only the rendered output.

viz-mapper reads the report and renders the side-by-side panels and badge pill. The report shape is the contract; viz-mapper does not call the VLM.

science-integrity runs in parallel and writes one perceptual-hash check into simspec.integrity.checks[]. The two skills are independent measurements feeding different surfaces.

narration may quote summary_md in an observation cue. The orchestrator wires the dependency; paper-diff does not call narration directly.

The Stage 1 + Stage 2 script lives at scripts/run_diff.py. Invoke as python scripts/run_diff.py [--claims claims.json]; it also exposes run_diff(ref_path, sim_path, claims) for inline orchestrator use.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: leventilo
• Source: leventilo/mobius
• License: MIT

Install and usage instructions live in the source repository linked above.