Science Integrity

science-integrity

The critic. The thing JutulGPT does not have. The reason a Mobius simulator is trustworthy and a vibe-coded one is not.

1. Philosophy — parallel critic, not serial reviewer

Mobius generates code. Generated code is a hypothesis. A hypothesis without an external falsifier is just a guess in a costume. JutulGPT (arXiv 2603.00214, §6.1) admits this directly: their authoritative arbiter is the simulator itself — if it runs, the answer is "physically valid". This is a category error. A simulator that converges on the wrong PDE, with wrong units, drifting energy, and producing a figure that does not match the paper is still convergent. The simulator certifies numerical health, not paper agreement.

The science-integrity skill is the second arbiter. It runs in parallel with the simulator (not after it as a unit test), it observes the SimSpec at construction time and the telemetry at execution time, and it speaks back to upstream generators in a structured language they can react to. Five orthogonal checks, none of which the simulator-as-arbiter pattern catches:

1. Units — SimSpec contains a constant E = 1.6e-19 declared in eV but used in code as J. Simulator runs cleanly. Output is wrong by 6.24e18×. Only a unit-aware verifier catches this.
2. CFL — dt, dx, and c_max jointly violate the Courant condition. Simulator either explodes (caught) or, with implicit smoothing, silently dampens the wave (not caught — this is the dangerous case).
3. Conservation — Hamiltonian SimSpec, no dissipation term, energy drifts 2% per period. Simulator converges; physics is broken.
4. Figure-diff — paper Fig 3 shows a 2-peak interference pattern; sim renders 4 peaks (slit count off-by-one). Simulator is happy. The user is looking at a different experiment.
5. Claim-match — paper claims "visibility 0.78 at λ=500 nm". Sim measures 0.31. Simulator does not know the paper exists.

These five layers, run together, do not exist anywhere in the agentic-simulation literature as of April 2026 (verified: JutulGPT, MooseAgent, AutoMOOSE, ChatCFD, FeaGPT, MCP-SIM, InteractScience, ChronoLLM — none combine an external SimSpec verifier + paper-figure VLM diff + paper-claim numerical comparator). This skill is the moat.

The choice of parallel over serial is deliberate. A serial review pattern (generator → checker → re-generator) bottlenecks on the checker and forces the generator to wait for verdicts. Parallel critic runs alongside, signals fail fast, and lets the orchestrator decide whether to interrupt the current generator iteration or queue the failure for the next one. JutulGPT is serial. We are parallel.

1.5 Phase routing — pre-render vs. post-render checks

Why a split. The J5-J6 live run on the Young paper showed Phase D (verify) running for ~36 minutes — the dominant chunk of the 52-minute end-to-end budget. Root cause: every check (including the perceptual VLM-driven figure_diff, the rendered-scene narration_bijection, and the multi-modal multimodal_triangulation) was run pre-render, before the Phase F scene even existed. Three of the eight checks are inherently post-render and were burning model time on absent inputs.

Decision. Split the eight bundled checks across two phases of the Mobius DAG. The orchestrator routes by phase name; this skill MUST honour the split so it does not waste time on checks whose inputs do not exist yet.

Phase D-verify (pre-render) — run these 5

Cheap, deterministic, and require ONLY the SimSpec — no rendered scene, no telemetry.

| # | Check | Script | Why pre-render | |---|---|---|---| | 1 | units | check_units.py | Reads physics.constants[] and equation strings; pint-only, no scene needed. | | 2 | cfl | check_cfl.py | Reads simulation.solver_hints + space + time + c_max; closed form, no I/O. | | 3 | conservation | check_conservation.py | Reads telemetry IF available, else returns pending. Falls into pre-render slot because telemetry from Phase E is available before Phase F. | | 4 | claim-match | claim_match.py | Reads paper.numerical_claims[] against telemetry; same Phase E/D ordering as conservation. | | 5 | eligibility | check_eligibility.py | Reads SimSpec gates only; deterministic Python, no LLM. |

Total wall-clock target on synthetic input: max_allowed, blocking the primitive-generator. warn when c_max cannot be extracted (returns the formula and asks the upstream to declare it).

2.3 Conservation check (scripts/check_conservation.py)

Purpose. Measure the relative drift of declared conserved quantities across the simulation telemetry. Catches energy/mass/probability/momentum drift in regimes where the governing equations should preserve them.

Input interface. Reads a telemetry file (CSV or JSONL with at least column t and one or more conserved quantity columns) and the SimSpec's physics.regime field plus physics.governing_equations[] to detect Hamiltonian vs dissipative structure.

Output. {"status": ..., "quantities": {"energy": {"drift_pct": ..., "status": ...}, "mass": {...}, ...}, "summary": "..."}.

Failure behavior. fail when drift exceeds the regime-dependent threshold (1% per 1000 steps for linear/Hamiltonian, 10% for strongly-nonlinear). For stochastic regimes, checks running mean instead of instantaneous value.

2.4 Figure-diff check (scripts/figure_diff.py)

Purpose. VLM-based structural comparison between the paper's reference figure and the simulator's rendered output. Catches "wrong slit count", "wrong peak count", "phase inverted", "spatial symmetry broken".

HONEST FLAG. This check is not deterministic. It calls an LLM (Opus 4.7 via the Anthropic vision API) and parses structured JSON back. We retry once on parse failure and cache by image hash to make the same (ref, sim) pair return the same answer within a session, but two different VLM invocations on the same images may return slightly different similarity_score floats. We accept this because no deterministic algorithm comparing arbitrary scientific figures exists at quality level required (pHash and SSIM fail badly on log-scale plots, on rotated/cropped axes, on different colormaps). We restrict the VLM to a structural comparison (count peaks, axis layout, spatial symmetry), not aesthetic judgment.

Input interface. Two image paths, plus an optional claim_context string from simspec.paper.figure_mapping[i].

Output. {"status": ..., "similarity_score": 0..1, "features_present": [...], "features_missing": [...], "features_spurious": [...], "confidence": 0..1}.

Failure behavior. fail when similarity_score 0.7. warn when 0.5 ≤ similarity_score tolerance (default 5%). Qualitative claims like "monotonically increases with frequency" return not-applicable and do not fail (they are deferred to a future structural-claim checker).

2.6 Output format (mandatory) — single emission, no re-run

You MUST emit your final output as a single fenced ``json block at the END of your reply, with NO prose after the closing fence. **Once emitted, stop all code_execution tool calls.** The orchestrator parses that block by regex (/``(?:json)?\s*\n([\s\S]*?)\n\s*`/) and any tool invocations after it cause the dispatch to hang on Anthropic's stream pipeline.

Per-check behaviour contract

When running a bundled script:

1. Call the script once via code_execution (bash, then Python).
2. Parse the JSON response. If JSON parse fails, retry once more

(2 attempts total per check).

1. If both attempts fail, emit `{"status": "fail", "summary": "script

returned invalid JSON"}` for that check and advance.

1. If parse succeeds, read the script's status:

• pass → mark final, advance.
• warn → mark final, advance.
• fail → emit final, advance. **Do NOT re-run the script to "verify"

or "fix" the failure.** The orchestrator's refine loop is the only legitimate retry mechanism.

1. Repeat for each check in the phase (5 checks in D-verify; 3 checks

in G-diff).

1. After all checks complete, aggregate into the canonical integrity

JSON.

1. **Emit the fenced ``json` block ONE TIME.** No code_execution`

calls after this point.

If you are tempted to re-run a script "to be sure", resist. Your job is to RUN each check ONCE, not to repeatedly affirm it. Confidence is a single-pass property; double-pass costs minutes of wall-clock and produces zero additional information.

Running the bundled scripts via code_execution and aggregating their per-script JSON is fine and recommended, but the canonical orchestrator-facing artifact is the AGGREGATE integrity report inlined as the final fenced JSON block of your reply.

The canonical artifact for science-integrity (consumed by orchestrator.ts Phase D, which reads result.checks[] into simspec.integrity.checks[]):

{
  "checks": [
    {
      "type": "units",
      "status": "pass",
      "message": "all constants have UCUM-parseable units; dimensional homogeneity OK across governing_equations[].",
      "details": { "n_constants": 4, "n_equations": 2 }
    },
    {
      "type": "cfl",
      "status": "warn",
      "message": "scheme=fft-spectral; CFL not applicable, skipped.",
      "details": { "scheme": "fft-spectral" }
    }
  ],
  "overall": "ok",
  "eligibility": {
    "score": 0.95,
    "tier": "full",
    "gates": {
      "has_equations": true,
      "has_figures": true,
      "has_numerical_claims": true,
      "units_consistent": true,
      "regime_supported": true
    },
    "reasons": []
  }
}

Required per check: type (one of the schema enum values from simspec.schema.json §integrity_check.type — units | cfl | conservation | figure-diff | claim-match | schema-validation | ambiguity | eligibility | narration-bijection | multimodal-triangulation | archetype-routing | narration_audit) and status (pass | warn | fail | pending). message and details are recommended.

overall MUST be one of ok | degraded | fail. eligibility is REQUIRED when the SimSpec carries an eligibility block (R6); for legacy specs without the eligibility schema, it MAY be omitted.

If the input SimSpec is unavailable or every script fails to execute, emit:

{ "error": "simspec.json missing; cannot run integrity checks" }

DO NOT emit narration, summaries, or follow-up questions after the closing fence — they break the orchestrator's downstream consumption and get silently dropped.

3. Integration flow

                      paper.pdf
                          │
                          ▼
               ┌──────────────────┐
               │   paper-parser   │
               └────────┬─────────┘
                        │ parsed_paper.json
                        ▼
               ┌──────────────────┐
               │ physics-interpret│
               └────────┬─────────┘
                        │
       ┌──────────[ check_units ]◄─── governing equations + constants
       │                │
       │                ▼
       │       ┌──────────────────┐
       │       │  simspec-author  │
       │       └────────┬─────────┘
       │                │ simspec.json
       │                ▼
       ├──────────[ check_units, check_cfl ]◄─── solver_hints, declared constants
       │                │
       │                ▼
       │       ┌──────────────────┐
       │       │primitive-generate│
       │       └────────┬─────────┘
       │                │ generated code
       │                ▼
       ├──────────[ check_units (on code) ]
       │                │
       │                ▼
       │       ┌──────────────────┐
       │       │  Managed Agent   │ ◄── execution
       │       │  (Python sandbox)│
       │       └────────┬─────────┘
       │                │ telemetry.csv, render.png
       │                ▼
       ├──────────[ check_conservation, claim_match ]◄─── telemetry
       │                │
       │                ▼
       └──────────[ figure_diff ]◄────── render.png vs paper figure
                        │
                        ▼
                   integrity report
                        │
                        ▼
            orchestrator decides:
            pass → release to user
            fail → retry loop (§4)
            warn → annotate, release

The five scripts run at four moments: after physics-interpreter (units only), after simspec-author (units + cfl), after primitive-generator (units on code), and after Managed Agent execution (conservation + claim-match + figure-diff). Each invocation produces a JSON report that the orchestrator merges into simspec.integrity.checks[].

4. Retry loop contract

When a check fails, the orchestrator receives a JSON message with this exact schema:

{
  "skill": "science-integrity",
  "check": "check_units" | "check_cfl" | "check_conservation" | "figure_diff" | "claim_match",
  "status": "fail",
  "stage": "after-physics-interpreter" | "after-simspec-author" | "after-primitive-generator" | "after-execution",
  "details": { "...": "..." },
  "remediation_hint": "string — natural-language nudge for the upstream generator",
  "retry_count": 0,
  "max_retries": 3
}

The orchestrator's contract:

• On first fail, increment retry_count, append remediation_hint to the upstream generator's next prompt, re-run only the failing stage.
• On second fail, do the same with stronger prompt framing ("the previous attempt also failed because…").
• On third fail, the orchestrator escalates to the user with a structured explanation. The simulator does not get released.
• figure_diff failures with confidence = 0.7 → tier: "full" → simulator renders normally.
• `0.4 dict: ...

CLI: python check_units.py dict: ...

CLI: python check_cfl.py dict: ...

CLI: python check_conservation.py

Exit codes: 0 = pass, 1 = warn, 2 = fail

```python
# scripts/figure_diff.py
def figure_diff(ref_figure_path: str, sim_output_path: str, claim_context: str = "") -> dict: ...
# CLI: python figure_diff.py   [claim_context]
# Requires ANTHROPIC_API_KEY in env. Caches by hash in /tmp/mobius_figure_diff_cache/.

# scripts/claim_match.py
def claim_match(simspec: dict, telemetry_path: str) -> dict: ...
# CLI: python claim_match.py  
# Exit codes: 0 = pass, 1 = warn, 2 = fail

All five are also re-exported from scripts/__init__.py for direct Python import:

from scripts import check_units, check_cfl, check_conservation, figure_diff, claim_match

The three v0.2 scripts (R5/R6/R8) sit alongside the five originals:

# scripts/check_eligibility.py
def check_eligibility(simspec: dict) -> dict: ...
# CLI: python check_eligibility.py  dict: ...
# CLI: python check_narration_bijection.py path/to/simspec.json
# Exit codes: 0 = pass, 1 = warn, 2 = fail

# scripts/check_multimodal_triangulation.py
def check_multimodal_triangulation(
    simspec: dict,
    render_path: str | None = None,
    measured: dict[str, float] | None = None,
) -> dict: ...
# CLI: python check_multimodal_triangulation.py simspec.json [--render PNG] [--measured k=v ...]
# Exit codes: 0 = all verified, 1 = any partially-verified, 2 = any unverified

All eight scripts re-export from scripts/__init__.py.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: leventilo
• Source: leventilo/mobius
• License: MIT

Install and usage instructions live in the source repository linked above.