Autoverifier

AutoVerifier

A structured, agentic protocol for verifying the claims in technical literature. It turns a raw paper plus a question ("does this result actually hold up?") into a traceable, evidence-linked assessment, by forcing the analysis through six progressive layers instead of trusting free-text inference.

The core idea: decompose every assertion into claim triples (Subject, Predicate, Object), attach provenance and evidence links to each one, and then progressively test those triples against (a) the document's own evidence, (b) the independent literature, and (c) real-world commercial/financial signals — aggregating the result into a scored hypothesis matrix. The discipline is the point: it catches overclaims, citation distortion, and undisclosed conflicts that a one-shot "summarize this paper" pass would repeat verbatim.

What this skill is — and is not

• **It is a verification protocol, not an oracle.** It encodes how a careful, skeptical analyst with no domain expertise should investigate a claim. Output quality depends on the evidence actually retrieved and the reasoning applied at each step. Treat every verdict as "best-supported assessment given what was found," not ground truth.
• Do not let the verifier overclaim. This tool's whole purpose is to flag conclusions that outrun their evidence. Hold the output to the same standard: every verdict must cite the evidence behind it, confidence must be hedged, and "not found" / "insufficient evidence" is a valid and important result. A confident verdict with no traceable support is itself the failure mode this skill exists to catch.
• Named individuals and organizations require extra care (see Guardrails). Conflict-of-interest and financial findings (Layer 5) must be tied to a citable public record and stated as "potential / per public filings," never asserted as fact.

Tools this skill expects

The protocol assumes an agent with: web search and page fetch (to find rebuttals, benchmarks, filings, news), the ability to read uploaded documents (PDF/HTML), and optionally code execution (for embeddings, a small vector store, or metric math on a large corpus). It runs end-to-end as a single agent calling these tools; no special infrastructure is required for a handful of papers. A real vector database only earns its keep on large corpora or in the reproducible multi-model variant — for a single target paper plus its critics, search + the agent's own context is enough.

Step 0 — Scope the task

Before running anything, fix three things:

1. Target artifact(s). The paper / preprint / report / claim under examination. If the user uploaded a file, that's the seed. If they named a claim or a "breakthrough," find the primary source first.
2. Target query. Phrase the actual question as a falsifiable statement to test — e.g. "Does method X achieve the [speedup / accuracy / efficacy] it claims?" rather than "tell me about X." The query defines what "supported / refuted" means downstream.
3. Domain calibration. This protocol is field-neutral and must be tuned to the artifact's field before extraction begins — domain expertise enters the pipeline here. The field's core concepts and terminology, what counts as strong evidence, its standard performance metrics and how they're defined, the reference benchmarks and current state of the art a claim must be measured against, what a genuinely strong baseline is, how comparisons get gamed, and where financial/COI signals live all differ by field. Run the calibration in references/domain_calibration.md first; the resulting domain profile provisions every downstream layer and is carried through the whole pipeline.

Then pick a mode matching the stakes and the user's appetite for effort. State which mode you're running and why.

| Mode | Layers | Use when | |---|---|---| | Quick scan | 1–3 | "Is this paper internally sound / is it overclaiming?" Single-document audit. Fast. | | Standard | 1–4 | "Does the literature corroborate this claim?" Adds independent triangulation. Good default. | | Full assessment | 1–6 | High-stakes, contested, or commercially-tied claims. Adds external signals + scored hypothesis matrix. This is the case-study mode. |

Layers are modular and ordered: each consumes the structured output of the previous one. You may also run a single layer in isolation if the user asks (e.g. "just do an overclaim check" → Layer 3 on the uploaded paper).

The pipeline

Run the layers in order for the chosen mode. Each layer has a dedicated reference file with the detailed protocol, prompts, and output schema — read the reference file for a layer before executing it.

1. Corpus Construction & Ingestion → references/layer1_corpus.md

Collect the target plus related sources (rebuttals, follow-ups, benchmarks, surveys, author profiles). Score sources for quality/bias. Extract clean text with section structure, process figures, capture metadata. Produces the evidence base.

1. Entity & Claim Extraction → references/layer2_extraction.md

Extract entities and decompose every assertion into (Subject, Predicate, Object) triples. Tag each with a 5-level provenance label and normalize quantitative metrics to comparable units. Produces a claim knowledge graph.

1. Intra-Document Verification → references/layer3_intradoc.md

For each triple, locate in-document evidence and label it supports / contradicts / neutral (NLI-style). Check methodology→result coherence (scope, baseline fairness, reproducibility). Flag overclaims (abstract/conclusion exceeding the body). Produces a per-document consistency score.

1. Cross-Source Verification → references/layer4_crosssource.md

Discover related claims via citation networks, semantic similarity, and shared-entity traversal; route every new source back through Layers 1–3. Align claim pairs, check citation fidelity, run contradiction root-cause analysis, and weight by source independence. Produces cross-source consensus labels.

1. External Signal Corroboration → references/layer5_external_signals.md

Build a financial/COI profile per entity (filings, funding, officer records), map supply-chain dependencies via multi-hop reasoning, and assemble a strategic-event timeline. Distinguishes real R&D from announcement-driven positioning. (Optional; highest care required.)

1. Hypothesis Matrix & Reporting → references/layer6_hypothesis_matrix.md

Aggregate provenance (L2) + consistency (L3) + consensus (L4) + signals (L5) into testable hypotheses, generate adversarial counter-hypotheses, estimate confidence from answer-stability + multi-model agreement, and assign each a label of Supported / Needs Review / Likely Hallucination, plus an overall technology-readiness rating and alpha-signal flag.

Output

The report must present a detailed, clearly-labeled output for every layer that was run — not only the final verdict. Each layer produces a real deliverable (the corpus inventory, the entities and claim-triple table, the intra-document verdicts and overclaim table with the consistency score, the cross-source contradiction analysis with independence weighting, the per-entity signal profiles, and the hypothesis matrix), and each gets its own section in the report, so the chain of reasoning is auditable end to end and any final conclusion can be traced back through the intermediate evidence that produced it. A report that shows only the bottom line is a failure — the traceability is the product. Use the per-layer tables in references/output_templates.md and the section-per-layer skeleton in assets/report_template.md.

Bookend the per-layer detail with an executive frame: open with a bottom-line answer that directly answers the target query — stating the alpha-signal determination (present / absent) first, then a concrete, consolidated summary of the reasons collected from every layer that ran, and a one-line split of what holds up vs what does not (e.g. "the underlying system demonstrably works and that finding is high-confidence; the headline performance claim does not survive independent evaluation") — so the reader gets the full, traceable verdict before the detail; then close with a plain-language recommendation. Conflating "the thing exists / runs" with "the headline claim is true" in that summary is the single most common error in technical assessment.

Include a short summary of the Step 0 domain profile up front so the reader can see which field-specific lens was applied. Match the report's depth to the mode: a Quick scan reports Layers 1–3, Standard reports 1–4, and the Full assessment reports all six.

Guardrails

• Evidence-link everything. No verdict without a source-text link or citation. If a claim can't be tied to retrieved evidence, label it "unverified," not "false."
• Layer 5 / conflicts of interest. State COI, financial, or supply-chain findings only against a citable public record (filings, company pages, news), and phrase them as potential and per [source]. Never assert that a named person has a conflict, stake, or motive as established fact. When public records are thin, say so and stop — do not infer motive.
• Calibrate confidence, don't manufacture it. Use Low / Medium / High and tie the level to agreement and answer-stability. Disagreement among sources or among generations is a finding to report, not noise to suppress.
• Provenance honesty. A projection ("we expect...") is Level 5 even when it sounds like a result. A single best-case number is not a typical result; report the distribution (median/range), not just the maximum.
• Stay domain-agnostic. The protocol carries no field-specific assumptions. When a field has an accepted external rubric for what makes this class of claim credible, retrieve it during domain calibration and inject it as a Layer-4 evaluation framework rather than hard-coding judgments — see references/domain_calibration.md and references/layer4_crosssource.md.

Customizing the pipeline for a new domain

Each layer is a self-contained module with declarative prompts and a structured deliverable, so analysts can select, reorder, or extend layers. The protocol itself is field-neutral; everything domain-specific is established once, up front, by the Step 0 calibration in references/domain_calibration.md, which feeds a domain profile into the layers — what counts as experimental vs simulated vs theoretical evidence (Layer 2), the field's credibility rubric (Layer 4), its characteristic ways of gaming comparisons (Layer 3), and its relevant external-signal sources (Layer 5, e.g. trial registries and regulators in biomedicine instead of securities filings in commercial tech). The provenance taxonomy and the hypothesis-matrix labels are domain-independent and should not be changed.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yuntaod
• Source: yuntaod/AutoVerifier
• License: Apache-2.0
• Homepage: https://arxiv.org/abs/2604.02617

Install and usage instructions live in the source repository linked above.