Veracity V2 Experimental

Veracity v2 Experimental — Enhanced Parallel Fact-Check

SAFE-style claim decomposition → quality gate → 3 waves of verification agents per run. Each wave attacks from a different angle. Multiple runs shift perspectives to reduce blind spots. Between runs, findings go through user review; approved fixes are applied before the next run.

What's New in v2 (12 Gap Fixes)

| # | Gap | Fix | Where | |---|-----|-----|-------| | 1 | No Citation Verification | Citation Existence + Support Verifier | A1 | | 2 | Decomposition Quality Unchecked | Disambiguation Auditor + Recall Auditor | Wave 0.5 | | 3 | No Confidence Calibration | Calibration protocol + weighted synthesis | All + C5 | | 4 | Model Homogeneity | Tool chain diversification + acknowledgment | All | | 5 | No Self-Consistency/SelfCheckGPT | Stochastic consistency in Blind Re-Derivation | B6 | | 6 | No Chain-of-Verification | Blind Re-Derivation agent | B6 | | 7 | No Logical/Causal Composition | Logical Composition Verifier | C6 | | 8 | No Conflicting Evidence Resolution | Conflict Resolution Protocol | C3 | | 9 | Temporal Validity too late | A4 enhanced with "still true today?" | A4 | | 10 | Consensus too simplistic | C5 rewritten with weighted aggregation | C5 | | 11 | No Error Propagation Control | Abstention instruction | All | | 12 | No Recall Audit | Recall Auditor | Wave 0.5 |

Methodology

Published methods:

• SAFE (Google DeepMind, arXiv:2403.18802): Atomic fact decomposition — outperforms crowdsourced human annotators (wins 76% of disagreement cases) at lower cost
• FActScore (Min et al., EMNLP 2023): Popularized atomic fact decomposition; SAFE extends similar approaches (citing Gao et al. and Wang et al.)
• Claimify (Microsoft, ACL 2025, arXiv:2502.10855): Disambiguation-aware claim extraction — 99% entailment, 87.6% recall, 96.7% precision [GAP-2]
• GhostCite (arXiv:2602.06718): Citation hallucination detection — 14-94% hallucination rates across 13 LLMs, 98.75% for recent-year citations [GAP-1]
• SourceCheckup (Stanford, Nature Communications 2025): 50-90% of LLM-cited sources don't fully support claims [GAP-1]
• 6-Point Veracity Scale: TRUE / MOSTLY TRUE / MIXED / MOSTLY FALSE / FALSE / UNVERIFIABLE (adapted from PolitiFact)
• Source Reliability Tiering: T1 (DOIs, databases) > T2 (institutional) > T3 (secondary)
• Tool-MAD Adversarial Debate (arXiv:2601.04742): Pro/con agents before a judge — up to 5.5% over prior SOTA
• CoVe — Chain-of-Verification (Meta, ACL 2024 Findings): Independent re-verification without seeing originals reduces hallucinated entities from 2.95 to 0.68 [GAP-6]
• LoCal (ACM Web Conference 2025): Logical composition — verified sub-facts may not logically entail original claims [GAP-7]
• CONFACT (IJCAI 2025, arXiv:2505.17762): Conflicting evidence resolution for varying-credibility sources [GAP-8]
• ChronoFact (IJCAI 2025, arXiv:2410.14964): 82% of temporal errors involve implicit temporal info [GAP-9]
• SelfCheckGPT (arXiv:2303.08896): NLI variant AUC-PR 92.50 — if model "knows" a fact, samples agree; hallucinations diverge [GAP-5]
• VeriFact/FactRBench (EMNLP 2025): First precision+recall benchmark — high precision ≠ high recall [GAP-12]
• Demystifying MAD (arXiv:2601.19921): Calibrated confidence improves Brier scores 0.217→0.069 [GAP-3]
• Scaling Truth (arXiv:2509.08803): AI Dunning-Kruger — smaller LLMs show high confidence despite lower accuracy; larger models better calibrated [GAP-3]
• FREE-MAD (arXiv:2509.11035): Trajectory-based scoring +13-16.5% over baselines [GAP-10]
• Beyond Majority Voting (arXiv:2510.01499): Optimal Weight aggregation outperforms majority voting [GAP-10]
• Voting or Consensus? (ACL 2025 Findings): Voting better for reasoning, consensus for knowledge [GAP-10]
• Alignment Bottleneck (arXiv:2602.10380): Conservative abstention reduces error propagation [GAP-11]

Custom practices (not published):

• Supermajority weighted consensus: 75% weighted agreement threshold, enhanced with calibration (Demystifying MAD)
• Evidence chain logging: Source URL/DOI, tier, relevant quote, confirms/contradicts
• Counter-evidence prompting: Agents try to disprove before marking VERIFIED
• Tool chain diversification: Different agents use different primary tools to reduce correlated errors [GAP-4]

Known limitation [GAP-4]: All agents use the same Claude model. Research (Stop Overvaluing MAD, arXiv:2502.08788; A-HMAD, 2025) shows model heterogeneity yields 4-6% absolute gains. We partially mitigate via tool chain diversification and prompt variation.

Input

$ARGUMENTS: file path or URL to audit, optionally runs=N (default: 3). If no target, ask user.

Architecture

Each run = 1 decomposition + 1 quality gate + 3 waves of verification = 20 agents/run.

• Run 1: Wave 0 → Wave 0.5 → Waves A, B, C (foundational)
• Run 2: Wave 0 (refresh) → Wave 0.5 → Waves D, E, F (adversarial)
• Run 3+: Wave 0 (refresh) → Wave 0.5 → Waves G, H, I (meta-analysis)
• Run 4+: Cycle (4→A/B/C, 5→D/E/F, 6→G/H/I) with all prior context

Total agents = runs × 20.

Shared Protocols

When constructing subagent prompts, include these definitions wherever agents reference [VERACITYSCALE], [EVIDENCECHAIN], [THINKVERIFY], [OUTPUTFORMAT], [CALIBRATION], [ABSTENTION].

Veracity Scale (6-point, adapted from PolitiFact):

• TRUE: Fully accurate, confirmed by T1 source
• MOSTLY TRUE: Accurate with minor discrepancy
• MIXED: Partially accurate
• MOSTLY FALSE: Significant inaccuracy
• FALSE: Factually incorrect
• UNVERIFIABLE: Cannot confirm or deny

Source Tiers: T1 (DOIs, databases, registrars, original data) > T2 (institutional, Google Scholar) > T3 (news, abstracts, secondary)

Evidence Chain (required per rated fact): Source URL/DOI | Source tier | Relevant quote | Confirms/contradicts

Output Format:

F### [CATEGORY] — **RATING** (Raw: N% → Calibrated: N%)
  Claim: "..." | Evidence: ... | Source: URL (Tier N) | Note: ...

Think & Verify: Before marking FALSE, double-check your source. Before marking VERIFIED, attempt to find contradicting evidence.

Confidence Calibration [GAP-3] (required for every rating): Report TWO measures per fact:

1. RAW CONFIDENCE: initial gut-level (0-100%)
2. CALIBRATED CONFIDENCE: adjusted by: T1 source (+20) / T2 (+10) / T3 (+0) / none (-20); 2+ sources agree (+15) / single (+0) / conflict (-15); designated domain (+10) / adjacent (+0) / outside (-10); reproducible (+10) / maybe (+0) / unlikely (-10). Calibrated = clamp(Raw + adjustments, 0, 100).

Report: "Raw: 85% → Calibrated: 70% (single T2, no corroboration)"

Abstention [GAP-11]: If CALIBRATED confidence 70% confidence, keep compound and flag for manual review.

1. QUALITY CHECK: Completeness, Correctness, Atomicity.
2. NUMBER sequentially (F001, F002, ...).
3. TRACK PROVENANCE [GAP-7]: Record original sentence (line/paragraph). Facts from same sentence share a provenance group ID (PG001, PG002, ...) for logical composition checking.

Output: F001 [CATEGORY] (PG001) "claim text" — Source: paragraph N, line M Report: total facts, category breakdown, ambiguous claims flagged, difficulties.

Wait for completion. Fact list → Wave 0.5.

---

## Wave 0.5 — Decomposition Quality Gate (2 parallel agents) `[GAP-2, GAP-12]`

Launch both simultaneously (Task tool, `subagent_type: general-purpose`):

**Agent 0.5a: Disambiguation & Atomicity Auditor `[GAP-2]`**

Audit decomposition quality using Claimify (ACL 2025) and Decomposition Dilemmas (NAACL 2025) insights. You have Wave 0 fact list and [TARGET].

For each fact:

1. ATOMICITY: Truly one verifiable claim? If 2+, flag for re-decomposition.
2. DISAMBIGUATION: Multiple plausible interpretations? Flag with confidence per interpretation.
3. MEANING PRESERVATION: Compare against provenance group — did decomposition change meaning?
4. OVER-DECOMPOSITION: Too fine-grained to verify independently?

Output: PASS/FAIL per fact. If >15% fail, recommend re-running Wave 0.

**Agent 0.5b: Recall Auditor `[GAP-12]`**

Using VeriFact/FactRBench insight: "high precision ≠ high recall." Read [TARGET] independently, paragraph by paragraph. For each, list every verifiable claim YOU identify. Cross-reference against Wave 0 list.

Focus on: implicit claims (timelines implying duration, lists implying counts), claims in visualizations/data structures/footnotes/tooltips, hedged but verifiable claims, multi-sentence claims.

Output: Missed facts (M001, M002, ...), recall estimate = Wave0/(Wave0+missed)×100. If recall 25% disputed, flag document-level concern. Apply [CALIBRATION], [ABSTENTION].

**C4: Comparative Claims Assessor**

Check all [COMPARATIVE] facts: "first"/"largest"/"novel" claims, adoption/influence claims, implied uniqueness, sample sizes vs field norms, effect sizes supporting narrative. WebSearch for prior work predating claims. Apply [VERACITY_SCALE], [CALIBRATION], [ABSTENTION]. PRIMARY TOOL: WebSearch.

**C5: Weighted Synthesis & Calibrated Consensus `[GAP-3, GAP-10]`**

Using FREE-MAD, Beyond Majority Voting, Voting vs Consensus insights. ALL findings: [ALLPRIORFINDINGS].

STEP 1: Collect all ratings + CALIBRATED confidence per fact.

STEP 2 — WEIGHTED AGGREGATION [GAP-10]:

• QUANTITATIVE/TEMPORAL facts: VOTING weighted by calibrated confidence (better for reasoning)
• NARRATIVE/COMPARATIVE facts: CONSENSUS 75% WEIGHTED agreement (better for knowledge)
• Domain-relevant agents get 2x weight (A1 on PUBLICATION, B1 on COMPARATIVE, A4 on TEMPORAL, B5 on CREDENTIAL)
• Other facts: Weighted supermajority 75% threshold

STEP 3 — TRAJECTORY [GAP-10]: Rating changes across waves are more informative than static. Agent that updated on new evidence → weight HIGHER. Agent that maintained despite contradiction → weight LOWER.

STEP 4 — CALIBRATION QUALITY [GAP-3]: Flag when avg calibrated is >20pts below avg raw (overconfidence), calibrated varies >40pts across agents, or B6 contradicts majority.

STEP 5: Categorize — FALSE→CRITICAL, MOSTLY FALSE→HIGH, MIXED→MEDIUM, MOSTLY TRUE→LOW, TRUE→VERIFIED, irreconcilable→DISPUTED [GAP-8]. Provide exact fix for CRITICAL/HIGH.

STEP 6: Executive summary: total facts, breakdown, % verified, confidence 0-100, calibration quality score, consensus rate, top 3 concerns, B6 agreement rate.

**C6: Logical Composition Verifier `[GAP-7]`**

Using LoCal (ACM Web Conference 2025). You have fact list with provenance groups, all results, [TARGET].

For each provenance group (facts from same original sentence):

1. COMPOSITION: Do verified sub-facts logically entail the original? (e.g., "Led a team of 5 that published 3" — "Led" never verified = orphan claim)
2. COUNTERFACTUAL: Flip one sub-fact to FALSE — does original still hold? If yes, something was missed.
3. ORPHAN CLAIMS: Causal ("because"), relational ("led", "managed"), conditional ("if") aspects uncovered by sub-facts.
4. EMERGENT CLAIMS: Combined sub-facts create an implied claim never verified.

Output per group: Original sentence, sub-facts+ratings, verdict (ENTAILED/PARTIAL/NOT ENTAILED), orphans, emergent claims, recommendation. Apply [CALIBRATION], [ABSTENTION].

---

## Run 2+ Waves

Reuse Wave 0 (refresh if changed), re-run Wave 0.5, shift perspectives. All agents receive [CALIBRATION] and [ABSTENTION].

### Wave D — Domain Expert Simulation (5 agents)

Adapt domain experts to the target document's subject matter. Example defaults:
- D1: Subject-matter domain expert — methodology, sample sizes, statistics
- D2: Technical/ML researcher — technical claims, benchmarks, implementation details
- D3: Adjacent-field expert — cross-disciplinary connections, broader context
- D4: Standards reviewer — evidence standards, citation norms, formatting
- D5: Investigative journalist — two-source rule, public records verification

### Wave E — Regression & Drift Detection (5 agents)
- E1: Internal document consistency (cross-references, embedded data vs standalone)
- E2: Quantitative data consistency (counts, percentages, totals)
- E3: Cross-section self-consistency (fact numbering, ordering)
- E4: Current vs prior versions (git history, changelogs)
- E5: Stale data detection (dates, links, version numbers)

### Wave F — Red Team (5 agents)
- F1: Prior work invalidating "first"/"pioneered"/"novel" claims
- F2: Methodology claims vs actual implementation
- F3: Quoted sources vs original archives
- F4: Repository/artifact contents vs documented claims (not just existence)
- F5: Retractions, corrections, errata for cited works

### Wave G — Meta-Analysis (5 agents, Run 3+)
- G1: Cross-run agreement — consistently flagged facts
- G2: Confidence trends (calibrated) across runs
- G3: Evidence gaps — facts still lacking T1 after 2 runs
- G4: Fix verification — applied fixes resolved issues?
- G5: New fact discovery — missed in prior Wave 0 passes

### Wave H — Stress Testing (5 agents, Run 3+)
- H1: Steelman/strawman — harder disproof of VERIFIED facts
- H2: Context checker — true in specific context or only general?
- H3: Temporal validity (deep) — additional sources beyond A4
- H4: Scope creep — claims within evidence support?
- H5: Statistical rigor — significant figures, rounding, units

### Wave I — Final Cross-Run Synthesis (5 agents, Run 3+)
- I1: Master consensus — weighted supermajority across ALL agents/runs
- I2: Reliability assessor — 0-100 with breakdown
- I3: Fix prioritizer — impact-to-effort ratio
- I4: Pattern reporter — systematic patterns
- I5: Executive summary — 1-page for decision-makers

**Convergence**: Stop when delta =4 to observe two consecutive deltas. After Run 4+: delta -run", "type": "run", "version": "v2-experimental",
  "session_id": "", "run_number": 1, "timestamp": "",
  "target": "...", "project": "...", "project_path": "...",
  "branch": "...", "commit_sha": "...",
  "veracity_score": 0, "prior_score": null, "score_delta": null,
  "agents_deployed": 20,
  "claims_total": 0, "claims_verified": 0, "claims_flagged": 0,
  "claims_unverifiable": 0, "claims_disputed": 0, "consensus_rate": 0,
  "veracity_distribution": { "true": 0, "mostly_true": 0, "mixed": 0, "mostly_false": 0, "false": 0, "unverifiable": 0, "disputed": 0 },
  "severity_counts": { "critical": 0, "high": 0, "medium": 0, "low": 0, "verified": 0, "unverifiable": 0, "disputed": 0 },
  "evidence_quality": { "tier1_sources": 0, "tier2_sources": 0, "tier3_sources": 0, "multi_source_claims": 0 },
  "fact_categories": { "quantitative": 0, "publication": 0, "temporal": 0, "credential": 0, "technical": 0, "link": 0, "comparative": 0, "narrative": 0, "missing": 0 },
  "v2_metrics": {
    "decomposition_quality_pct": 0, "decomposition_recall_pct": 0, "missed_facts_added": 0,
    "calibration_quality": 0.0, "blind_rederivation_agreement_pct": 0,
    "logical_composition_pct": 0, "conflicts_resolved": 0, "abstention_rate_pct": 0,
    "stochastic_consistency_high_pct": 0, "stochastic_consistency_low_count": 0
  },
  "review_decisions": { "critical_batch": "...", "high_batch": "...", "medium_batch": "...", "low_batch": "...", "disputed_batch": "..." },
  "findings": [{
    "fact_id": "F###", "severity": "...", "veracity_rating": "...",
    "raw_confidence": 0, "calibrated_confidence": 0,
    "category": "...", "claim": "...", "evidence_summary": "...",
    "source_tier": "T#", "source_url": "...",
    "citation_support": "supports|partially|does_not_support|contradicts|n_a",
    "blind_rederivation": "match|mismatch|partial|not_checked",
    "stochastic_consistency": "high|medium|low|not_checked",
    "logical_composition": "entailed|partial|not_entailed|n_a",
    "agents_agreed": "N/M", "weighted_agreement": 0,
    "location": "...", "fix": "...", "status": "...", "disposition": null
  }]
}

6c. Session entry: Same base fields plus type: "session", runs_requested, `runs_compl

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: joonchungpersonal-dev
• Source: joonchungpersonal-dev/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.