# Paper Diff > Compare simulation output to the original paper's reference figures using Opus 4.7 vision plus deterministic image metrics. Produces structured diff with feature-level present/missing/spurious lists and a similarity score. Run after primitive-generator + execution, before pitch the result to the user. Distinct from science-integrity's figure_diff (which is one of five integrity checks); paper-dif… - **Type:** Skill - **Install:** `agentstack add skill-leventilo-mobius-paper-diff` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [leventilo](https://agentstack.voostack.com/s/leventilo) - **Installs:** 0 - **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [leventilo](https://github.com/leventilo) - **Source:** https://github.com/leventilo/mobius/tree/main/skills/paper-diff ## Install ```sh agentstack add skill-leventilo-mobius-paper-diff ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # paper-diff ## 1. Purpose and scope This skill produces the user-facing comparison artifact between the figures of the source paper and the rendered output of a Mobius simulation. It is the visible evidence that the simulation reproduces the physics described in the paper. Its product backs the green "reproduces Figure 3" badge surfaced near the simulator and the side-by-side panel that opens when a user clicks the badge. The skill runs after `primitive-generator` has emitted Python code, after the simulation has executed in the Claude Managed Agents sandbox, and after at least one frame per relevant figure has been exported. It runs before the orchestrator pitches the final assembled simulator. It does not run during the live slider loop; the diff is computed once per stable simulation snapshot and cached. What this skill does NOT do. It does not extract figures from the paper PDF (that is `paper-parser`), does not decide whether to block the pipeline (that is `science-integrity`), does not render the side-by-side panels in the browser (that is `viz-mapper`), and does not invent ground-truth values. ## 2. Distinction from science-integrity.figure_diff Load-bearing. Both skills compare images but serve orthogonal purposes. `science-integrity.figure_diff` is one of the five deterministic integrity checks (units, CFL, conservation, figure-diff, claim-match). It returns `pass | warn | fail`, writes into `simspec.integrity.checks[]`, and on `fail` blocks the pipeline so the orchestrator re-dispatches `primitive-generator`. It is silent on the user surface. `paper-diff` is the user-facing report. It returns a rich structured object enumerating matched features, missing features, spurious features, color and axis agreement, and surviving numerical claims. It is always rendered in the UI and never blocks. Even on catastrophic disagreement, the simulator renders with a red "divergent" badge and a panel listing every divergence verbatim. The bet: an honest "this does not match Figure 3 because the simulation produced 2 peaks where the paper shows 4" is more valuable to a user and a judge than a silently downgraded match. The two skills share the underlying VLM call shape and the deterministic-metric primitives, but their output schemas, blocking semantics, and rendering pathways are different. `science-integrity.figure_diff` is a gate; `paper-diff` is a report. ## 3. Citation of the SINTEF gap JutulGPT (arXiv 2603.00214, SINTEF Digital + NTNU, MIT licence) is one intellectual ancestor of Mobius. In Section 6.1 the authors enumerate explicit limitations of their reference implementation. Two are directly relevant: > "the agent does not yet perform structured comparison of simulation outputs" > "systematic multimodal output comparison... remains an important development direction" These are verbatim from page 11 / 19 of the v1 manuscript. paper-diff exists exactly to occupy this gap. JutulGPT relies on the simulator's internal conservation tolerances as its sole validity certificate - "if it ran, it is physical". Mobius adds a second certificate: "and what came out matches the paper's published figures, structurally and numerically, at the feature level". The pairing is the moat; neither half alone is novel. This is also the primary citation hook for the hackathon pitch: SINTEF flags multimodal output comparison as future work in February 2026; Mobius ships it in April 2026 with Opus 4.7 vision. ## 4. Inputs For each figure id in `simspec.paper.figures[]` the skill expects three artifacts. `ref_path`. The reference PNG extracted from the paper PDF by paper-parser, under `artifacts/figures/.png`. Resolution is whatever the PDF provides; the skill rescales. `sim_path`. The simulation frame for the same figure, exported by the execution stage. The orchestrator picks the frame closest to the figure caption's time anchor ("steady state", "t = 5 ns", "first 100 cycles average"). Multi-panel figures pair frames one-to-one. `claims`. The subset of `paper.numerical_claims[]` whose `section` anchors to this figure (`Fig. 3`, `Fig. 3a`). Each carries value, UCUM unit, confidence, and verbatim source quote. The skill uses them to ground the VLM: rather than "do these look similar", we ask "does the simulation reproduce the quantities this figure claims". Optional `mode` flag: `"fast"` skips the VLM stage (used during the slider loop); `"full"` is the default. ## 5. Output A `paper_diff_report` object per figure, one entry per panel. The shape: ```json { "figure_id": "fig-3", "similarity_score": 0.82, "panels": [ { "label": "panel-a", "ref_path": "artifacts/figures/fig-3.png", "sim_path": "artifacts/sim_frames/fig-3_panel-a.png", "similarity_score": 0.84, "features": { "present": ["five interference peaks", "central maximum brightest", "envelope falls off symmetrically"], "missing": ["axis labels in micrometers"], "spurious": ["faint vertical banding from FFT aliasing"] }, "dominant_color_match": true, "axis_scale_match": true, "peak_count_match": true, "claim_overlay": [ { "claim": "fringe spacing 0.63 mm at 1 m screen distance", "present_in_sim": true, "value_diff_percent": 1.8 }, { "claim": "first minimum at 0.31 mm from center", "present_in_sim": true, "value_diff_percent": 12.4 } ] } ], "summary_md": "Five visible peaks, central maximum brightest, fringe spacing within 2% of paper. First-minimum location off by 12%, likely due to coarser sampling at the aperture edge.", "badge": "match" } ``` `badge` thresholds: `"match"` (similarity >= 0.80 and no missing critical feature); `"approx"` (0.60 ` extractor — keep them equal). `details[]` is recommended and carries the per-figure rich report shape from §5; the UI renders it verbatim. If `ref_path` or `sim_path` is missing, short-circuit Stage 1+2 and emit a fenced JSON with `vlm_status` + `score: 0.0`: ```json { "ssim": 0.0, "vlm_judgment": "uncertain", "diff_score": 0.0, "score": 0.0, "details": [], "vlm_status": "missing-input", "error": "sim_path /runs//scene.png does not exist" } ``` DO NOT emit narration, summaries, or follow-up questions after the closing fence — they break the orchestrator's downstream consumption and get silently dropped. ## 6. Two-stage diff Deterministic first, VLM second. Cheap before expensive, in the order that maximises explainability. **Stage 1: deterministic image metrics.** Over the rescaled image pair (`512 x 512`): - SSIM via `skimage.metrics.structural_similarity` on the luminance channels, mapped to [0, 1]. - Histogram overlap on RGB via `numpy.histogram` with 32 bins per channel, summed per-bin min. Catches dominant-color agreement independently of spatial layout. - Edge IoU via `skimage.feature.canny`, intersection over union of edge pixels. Catches axes and dominant outlines. Stage 1 score = `0.5 * ssim + 0.25 * hist + 0.25 * edges`. Bounded, sub-second, never hallucinates. **Stage 2: Opus 4.7 vision.** One multimodal call to `client.messages.create` with two `image` content blocks (ref and sim, base64-encoded PNG, `media_type: "image/png"`) and a `text` block carrying the schema and claim list. Model: `claude-opus-4-7`. Output is strict JSON matching the panel schema. If the VLM call fails (network, rate limit, parse error), the report falls back to Stage 1 only with `"vlm_status": "unavailable"` plus the error string. The pipeline never crashes on VLM failure. ## 7. VLM prompt (verbatim) The exact system prompt used in Stage 2. ~200 words, no slack: > You are a structural-fidelity comparator for physics simulation outputs. You receive two images: a reference figure from a published paper, and a frame produced by a simulation that claims to reproduce it. Your job is to enumerate, in strict JSON, every structural feature the two images share, every feature present in the reference but missing in the simulation, every feature present in the simulation but spurious. You also report whether dominant colors match, whether axis scales match, and whether the count of dominant peaks or features matches. You compare to a list of numerical claims; for each, mark whether it is present_in_sim and the value_diff_percent if a numerical value can be read from the simulation. > > Forbidden: artistic judgment. Do not say "the simulation looks ugly" or "the colors are not as nice". You are not an art critic. You are reporting structural physics fidelity. Forbidden: speculation beyond what the images show. Do not say "this is probably caused by". Forbidden: prose outside the JSON. The response MUST be a single JSON object matching the panel schema in the user message; no preamble, no postscript, no markdown fences. > > If you cannot determine a field from the images, set it to null and note it in features.missing with the prefix "uncertain:". The user message carries the schema fragment, the claim list, and a 30-word context blurb pulled from the figure caption. Temperature is 0; we do not want creativity in a structural diff. ## 8. Caching Cache key: `sha256(ref_bytes || sim_bytes || claims_canonical_json || prompt_version)`. Cached entries at `/tmp/mobius_paper_diff_cache/.json`. A hit short-circuits both stages. `prompt_version` is bumped manually whenever the system prompt changes, invalidating old entries cleanly. The cache is content-addressed: sliders changing parameters produce new sim bytes and new cache misses, exactly the desired behavior. Reference figures rarely change within a session. Eviction: 200 MB or 500 entries, LRU. The directory is gitignored. ## 9. Visual integration The report renders as a Ciechanowski-style side-by-side panel: - Left: paper reference on `--c4` rose-tinted matte, 1.5 px stroke, IBM Plex Sans caption. - Right: simulation frame on `--c3` blue-tinted matte, identical aspect ratio, caption "Mobius - reproduced". - Center column of annotations, one row per `features.present` (green tick `--c2`), `features.missing` (orange dash `--c1`), `features.spurious` (red cross `--c5`). Hover highlights the corresponding region in both images. - Bottom: `summary_md` as body-type prose, plus an accordion expanding the `claim_overlay` table. The badge is a pill inline with the simulator title bar: `--c2` match, `--c1` approx, `--c5` divergent. Click opens the panel as a modal. All easing follows the project's quintic smoothstep `x*x*x*(x*(x*6-15)+10)`. ## 10. Examples - three worked diff scenarios **Match.** Young's double-slit. Paper shows five fringes with brightest central maximum, fringe spacing 0.63 mm at 1 m. Sim reproduces five fringes, fringe spacing 0.642 mm. SSIM 0.91, hist 0.88, edges 0.86. VLM: "five interference peaks, central maximum brightest, envelope symmetric". `claim_overlay` fringe spacing differs by 1.8%. Badge: `match`. **Approx.** Damped harmonic oscillator. Paper omega_0 = 2.0, gamma = 0.1. Sim 1.94 / 0.097, 3% error from coarser dt. SSIM 0.74. VLM: "correct exponential envelope, period off by 3%". Badge: `approx`. The user sees the honest "off by 3%" pill. **Divergent.** Four-slit grating. Paper shows four primary maxima per order; sim, due to a primitive defaulting slit_count = 2, shows two. SSIM 0.41. VLM: "two primary maxima where four were expected; missing slits 3 and 4". Badge: `divergent`. The orchestrator may re-dispatch `primitive-generator` with the diagnostic; the user sees an unedited red pill explaining the gap. ## 11. Red flags Never lie about a match. If similarity_score .png`; paper-diff reads, never modifies. `primitive-generator` indirectly produces sim frames via executed code; paper-diff sees only the rendered output. `viz-mapper` reads the report and renders the side-by-side panels and badge pill. The report shape is the contract; viz-mapper does not call the VLM. `science-integrity` runs in parallel and writes one perceptual-hash check into `simspec.integrity.checks[]`. The two skills are independent measurements feeding different surfaces. `narration` may quote `summary_md` in an `observation` cue. The orchestrator wires the dependency; paper-diff does not call narration directly. The Stage 1 + Stage 2 script lives at `scripts/run_diff.py`. Invoke as `python scripts/run_diff.py [--claims claims.json]`; it also exposes `run_diff(ref_path, sim_path, claims)` for inline orchestrator use. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [leventilo](https://github.com/leventilo) - **Source:** [leventilo/mobius](https://github.com/leventilo/mobius) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-leventilo-mobius-paper-diff - Seller: https://agentstack.voostack.com/s/leventilo - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.