# Science Integrity > Parallel critic agent that enforces physical invariants on Mobius SimSpecs and simulation outputs. Runs five bundled deterministic checks — units, CFL stability, conservation laws, figure-diff, and claim-match — and blocks or warns upstream generators. Use whenever a SimSpec or simulation artifact is produced. - **Type:** Skill - **Install:** `agentstack add skill-leventilo-mobius-science-integrity` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [leventilo](https://agentstack.voostack.com/s/leventilo) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [leventilo](https://github.com/leventilo) - **Source:** https://github.com/leventilo/mobius/tree/main/skills/science-integrity ## Install ```sh agentstack add skill-leventilo-mobius-science-integrity ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # science-integrity The critic. The thing JutulGPT does not have. The reason a Mobius simulator is trustworthy and a vibe-coded one is not. ## 1. Philosophy — parallel critic, not serial reviewer Mobius generates code. Generated code is a hypothesis. A hypothesis without an external falsifier is just a guess in a costume. JutulGPT (arXiv 2603.00214, §6.1) admits this directly: their authoritative arbiter is *the simulator itself* — if it runs, the answer is "physically valid". This is a category error. A simulator that converges on the wrong PDE, with wrong units, drifting energy, and producing a figure that does not match the paper is still convergent. The simulator certifies *numerical health*, not *paper agreement*. The science-integrity skill is the second arbiter. It runs in parallel with the simulator (not after it as a unit test), it observes the SimSpec at construction time and the telemetry at execution time, and it speaks back to upstream generators in a structured language they can react to. Five orthogonal checks, none of which the simulator-as-arbiter pattern catches: 1. **Units** — SimSpec contains a constant `E = 1.6e-19` declared in eV but used in code as J. Simulator runs cleanly. Output is wrong by 6.24e18×. Only a unit-aware verifier catches this. 2. **CFL** — `dt`, `dx`, and `c_max` jointly violate the Courant condition. Simulator either explodes (caught) or, with implicit smoothing, silently dampens the wave (not caught — this is the dangerous case). 3. **Conservation** — Hamiltonian SimSpec, no dissipation term, energy drifts 2% per period. Simulator converges; physics is broken. 4. **Figure-diff** — paper Fig 3 shows a 2-peak interference pattern; sim renders 4 peaks (slit count off-by-one). Simulator is happy. The user is looking at a different experiment. 5. **Claim-match** — paper claims "visibility 0.78 at λ=500 nm". Sim measures 0.31. Simulator does not know the paper exists. These five layers, run together, do not exist anywhere in the agentic-simulation literature as of April 2026 (verified: JutulGPT, MooseAgent, AutoMOOSE, ChatCFD, FeaGPT, MCP-SIM, InteractScience, ChronoLLM — none combine an external SimSpec verifier + paper-figure VLM diff + paper-claim numerical comparator). This skill is the moat. The choice of *parallel* over *serial* is deliberate. A serial review pattern (generator → checker → re-generator) bottlenecks on the checker and forces the generator to wait for verdicts. Parallel critic runs alongside, signals fail fast, and lets the orchestrator decide whether to interrupt the current generator iteration or queue the failure for the next one. JutulGPT is serial. We are parallel. ## 1.5 Phase routing — pre-render vs. post-render checks **Why a split.** The J5-J6 live run on the Young paper showed Phase D (verify) running for ~36 minutes — the dominant chunk of the 52-minute end-to-end budget. Root cause: every check (including the perceptual VLM-driven `figure_diff`, the rendered-scene `narration_bijection`, and the multi-modal `multimodal_triangulation`) was run pre-render, before the Phase F scene even existed. Three of the eight checks are inherently post-render and were burning model time on absent inputs. **Decision.** Split the eight bundled checks across two phases of the Mobius DAG. The orchestrator routes by phase name; this skill MUST honour the split so it does not waste time on checks whose inputs do not exist yet. ### Phase D-verify (pre-render) — run these 5 Cheap, deterministic, and require ONLY the SimSpec — no rendered scene, no telemetry. | # | Check | Script | Why pre-render | |---|---|---|---| | 1 | `units` | `check_units.py` | Reads `physics.constants[]` and equation strings; pint-only, no scene needed. | | 2 | `cfl` | `check_cfl.py` | Reads `simulation.solver_hints` + `space` + `time` + `c_max`; closed form, no I/O. | | 3 | `conservation` | `check_conservation.py` | Reads telemetry IF available, else returns `pending`. Falls into pre-render slot because telemetry from Phase E is available before Phase F. | | 4 | `claim-match` | `claim_match.py` | Reads `paper.numerical_claims[]` against telemetry; same Phase E/D ordering as conservation. | | 5 | `eligibility` | `check_eligibility.py` | Reads SimSpec gates only; deterministic Python, no LLM. | Total wall-clock target on synthetic input: max_allowed, blocking the primitive-generator. `warn` when `c_max` cannot be extracted (returns the formula and asks the upstream to declare it). ### 2.3 Conservation check (`scripts/check_conservation.py`) **Purpose.** Measure the relative drift of declared conserved quantities across the simulation telemetry. Catches energy/mass/probability/momentum drift in regimes where the governing equations should preserve them. **Input interface.** Reads a telemetry file (CSV or JSONL with at least column `t` and one or more conserved quantity columns) and the SimSpec's `physics.regime` field plus `physics.governing_equations[]` to detect Hamiltonian vs dissipative structure. **Output.** `{"status": ..., "quantities": {"energy": {"drift_pct": ..., "status": ...}, "mass": {...}, ...}, "summary": "..."}`. **Failure behavior.** `fail` when drift exceeds the regime-dependent threshold (1% per 1000 steps for linear/Hamiltonian, 10% for strongly-nonlinear). For stochastic regimes, checks running mean instead of instantaneous value. ### 2.4 Figure-diff check (`scripts/figure_diff.py`) **Purpose.** VLM-based structural comparison between the paper's reference figure and the simulator's rendered output. Catches "wrong slit count", "wrong peak count", "phase inverted", "spatial symmetry broken". **HONEST FLAG.** This check is **not deterministic**. It calls an LLM (Opus 4.7 via the Anthropic vision API) and parses structured JSON back. We retry once on parse failure and cache by image hash to make the same `(ref, sim)` pair return the same answer within a session, but two different VLM invocations on the same images may return slightly different `similarity_score` floats. We accept this because no deterministic algorithm comparing arbitrary scientific figures exists at quality level required (pHash and SSIM fail badly on log-scale plots, on rotated/cropped axes, on different colormaps). We restrict the VLM to a *structural* comparison (count peaks, axis layout, spatial symmetry), not aesthetic judgment. **Input interface.** Two image paths, plus an optional `claim_context` string from `simspec.paper.figure_mapping[i]`. **Output.** `{"status": ..., "similarity_score": 0..1, "features_present": [...], "features_missing": [...], "features_spurious": [...], "confidence": 0..1}`. **Failure behavior.** `fail` when `similarity_score 0.7`. `warn` when `0.5 ≤ similarity_score tolerance` (default 5%). Qualitative claims like "monotonically increases with frequency" return `not-applicable` and do not fail (they are deferred to a future structural-claim checker). ## 2.6 Output format (mandatory) — single emission, no re-run You MUST emit your final output as a **single** fenced ```json block at the END of your reply, with NO prose after the closing fence. **Once emitted, stop all `code_execution` tool calls.** The orchestrator parses that block by regex (`/```(?:json)?\s*\n([\s\S]*?)\n\s*```/`) and any tool invocations after it cause the dispatch to hang on Anthropic's stream pipeline. ### Per-check behaviour contract When running a bundled script: 1. **Call the script once** via `code_execution` (bash, then Python). 2. **Parse the JSON response.** If JSON parse fails, retry **once** more (2 attempts total per check). 3. **If both attempts fail**, emit `{"status": "fail", "summary": "script returned invalid JSON"}` for that check and advance. 4. **If parse succeeds**, read the script's `status`: - `pass` → mark final, advance. - `warn` → mark final, advance. - `fail` → emit final, advance. **Do NOT re-run the script to "verify" or "fix" the failure.** The orchestrator's refine loop is the only legitimate retry mechanism. 5. **Repeat for each check in the phase** (5 checks in D-verify; 3 checks in G-diff). 6. **After all checks complete**, aggregate into the canonical `integrity` JSON. 7. **Emit the fenced ```json``` block ONE TIME.** No `code_execution` calls after this point. If you are tempted to re-run a script "to be sure", resist. Your job is to RUN each check ONCE, not to repeatedly affirm it. Confidence is a single-pass property; double-pass costs minutes of wall-clock and produces zero additional information. Running the bundled scripts via code_execution and aggregating their per-script JSON is fine and recommended, but the canonical orchestrator-facing artifact is the AGGREGATE integrity report inlined as the final fenced JSON block of your reply. The canonical artifact for `science-integrity` (consumed by `orchestrator.ts` Phase D, which reads `result.checks[]` into `simspec.integrity.checks[]`): ```json { "checks": [ { "type": "units", "status": "pass", "message": "all constants have UCUM-parseable units; dimensional homogeneity OK across governing_equations[].", "details": { "n_constants": 4, "n_equations": 2 } }, { "type": "cfl", "status": "warn", "message": "scheme=fft-spectral; CFL not applicable, skipped.", "details": { "scheme": "fft-spectral" } } ], "overall": "ok", "eligibility": { "score": 0.95, "tier": "full", "gates": { "has_equations": true, "has_figures": true, "has_numerical_claims": true, "units_consistent": true, "regime_supported": true }, "reasons": [] } } ``` Required per check: `type` (one of the schema enum values from `simspec.schema.json` §integrity_check.type — `units | cfl | conservation | figure-diff | claim-match | schema-validation | ambiguity | eligibility | narration-bijection | multimodal-triangulation | archetype-routing | narration_audit`) and `status` (`pass | warn | fail | pending`). `message` and `details` are recommended. `overall` MUST be one of `ok | degraded | fail`. `eligibility` is REQUIRED when the SimSpec carries an `eligibility` block (R6); for legacy specs without the eligibility schema, it MAY be omitted. If the input SimSpec is unavailable or every script fails to execute, emit: ```json { "error": "simspec.json missing; cannot run integrity checks" } ``` DO NOT emit narration, summaries, or follow-up questions after the closing fence — they break the orchestrator's downstream consumption and get silently dropped. ## 3. Integration flow ``` paper.pdf │ ▼ ┌──────────────────┐ │ paper-parser │ └────────┬─────────┘ │ parsed_paper.json ▼ ┌──────────────────┐ │ physics-interpret│ └────────┬─────────┘ │ ┌──────────[ check_units ]◄─── governing equations + constants │ │ │ ▼ │ ┌──────────────────┐ │ │ simspec-author │ │ └────────┬─────────┘ │ │ simspec.json │ ▼ ├──────────[ check_units, check_cfl ]◄─── solver_hints, declared constants │ │ │ ▼ │ ┌──────────────────┐ │ │primitive-generate│ │ └────────┬─────────┘ │ │ generated code │ ▼ ├──────────[ check_units (on code) ] │ │ │ ▼ │ ┌──────────────────┐ │ │ Managed Agent │ ◄── execution │ │ (Python sandbox)│ │ └────────┬─────────┘ │ │ telemetry.csv, render.png │ ▼ ├──────────[ check_conservation, claim_match ]◄─── telemetry │ │ │ ▼ └──────────[ figure_diff ]◄────── render.png vs paper figure │ ▼ integrity report │ ▼ orchestrator decides: pass → release to user fail → retry loop (§4) warn → annotate, release ``` The five scripts run at four moments: after `physics-interpreter` (units only), after `simspec-author` (units + cfl), after `primitive-generator` (units on code), and after Managed Agent execution (conservation + claim-match + figure-diff). Each invocation produces a JSON report that the orchestrator merges into `simspec.integrity.checks[]`. ## 4. Retry loop contract When a check fails, the orchestrator receives a JSON message with this exact schema: ```json { "skill": "science-integrity", "check": "check_units" | "check_cfl" | "check_conservation" | "figure_diff" | "claim_match", "status": "fail", "stage": "after-physics-interpreter" | "after-simspec-author" | "after-primitive-generator" | "after-execution", "details": { "...": "..." }, "remediation_hint": "string — natural-language nudge for the upstream generator", "retry_count": 0, "max_retries": 3 } ``` The orchestrator's contract: - On first fail, increment `retry_count`, append `remediation_hint` to the upstream generator's next prompt, re-run only the failing stage. - On second fail, do the same with stronger prompt framing ("the previous attempt also failed because…"). - On third fail, the orchestrator escalates to the user with a structured explanation. The simulator does not get released. - `figure_diff` failures with `confidence = 0.7` → `tier: "full"` → simulator renders normally. - `0.4 dict: ... # CLI: python check_units.py dict: ... # CLI: python check_cfl.py dict: ... # CLI: python check_conservation.py # Exit codes: 0 = pass, 1 = warn, 2 = fail ``` ```python # scripts/figure_diff.py def figure_diff(ref_figure_path: str, sim_output_path: str, claim_context: str = "") -> dict: ... # CLI: python figure_diff.py [claim_context] # Requires ANTHROPIC_API_KEY in env. Caches by hash in /tmp/mobius_figure_diff_cache/. ``` ```python # scripts/claim_match.py def claim_match(simspec: dict, telemetry_path: str) -> dict: ... # CLI: python claim_match.py # Exit codes: 0 = pass, 1 = warn, 2 = fail ``` All five are also re-exported from `scripts/__init__.py` for direct Python import: ```python from scripts import check_units, check_cfl, check_conservation, figure_diff, claim_match ``` The three v0.2 scripts (R5/R6/R8) sit alongside the five originals: ```python # scripts/check_eligibility.py def check_eligibility(simspec: dict) -> dict: ... # CLI: python check_eligibility.py dict: ... # CLI: python check_narration_bijection.py path/to/simspec.json # Exit codes: 0 = pass, 1 = warn, 2 = fail ``` ```python # scripts/check_multimodal_triangulation.py def check_multimodal_triangulation( simspec: dict, render_path: str | None = None, measured: dict[str, float] | None = None, ) -> dict: ... # CLI: python check_multimodal_triangulation.py simspec.json [--render PNG] [--measured k=v ...] # Exit codes: 0 = all verified, 1 = any partially-verified, 2 = any unverified ``` All eight scripts re-export from `scripts/__init__.py`. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [leventilo](https://github.com/leventilo) - **Source:** [leventilo/mobius](https://github.com/leventilo/mobius) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-leventilo-mobius-science-integrity - Seller: https://agentstack.voostack.com/s/leventilo - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.