# Evidentia > Open-source medical citation verifier: catches fabricated DOI/PMID/NCT references with CLI, MCP, and Claude Code skill. - **Type:** MCP server - **Install:** `agentstack add mcp-kgraph57-evidentia` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [kgraph57](https://agentstack.voostack.com/s/kgraph57) - **Installs:** 0 - **Category:** [Integrations](https://agentstack.voostack.com/c/integrations) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [kgraph57](https://github.com/kgraph57) - **Source:** https://github.com/kgraph57/evidentia ## Install ```sh agentstack add mcp-kgraph57-evidentia ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Evidentia ### Catch AI-fabricated medical citations before you publish. **AIが捏造した医学引用を、公開前に捕まえる。** The `evidentia` command verifies every citation in a piece of medical writing against **CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov** and grades each one in a 4-tier classification. The companion agent skill adds a full 15-criteria evidence appraisal on top. Built by a board-certified pediatrician. [](https://www.npmjs.com/package/evidentia) [](https://github.com/kgraph57/evidentia/actions/workflows/ci.yml) [](LICENSE) [](https://code.claude.com/docs/en/skills) [](https://code.claude.com/docs/en/skills) [](README.md) [](README.ja.md) 📖 日本語で読む方は **[日本語版 README → README.ja.md](README.ja.md)** へ --- > **Why now:** A *Lancet* audit of 2.5 million biomedical papers (Topaz et al., May 2026; [doi:10.1016/S0140-6736(26)00603-3](https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(26)00603-3/fulltext)) found that **1 in 277 papers published in early 2026 contained a fabricated reference** — up from 1 in 2,828 in 2023, a **12-fold rise** that tracks the spread of AI writing tools. (The audit screened the PubMed Central open-access subset.) Coverage: [STAT](https://www.statnews.com/2026/05/07/lancet-study-finds-steep-rise-fraudulent-citations-academic-papers/) · [Nature](https://www.nature.com/articles/d41586-026-00748-w) · [Columbia Nursing](https://www.nursing.columbia.edu/news/nearly-3-000-peer-reviewed-medical-papers-have-fake-citations-columbia-nursing-ai-assisted-audit-finds) · [Retraction Watch](https://retractionwatch.com/2026/05/07/one-in-277-pubmed-indexed-papers-in-2026-shows-fabricated-references-says-analysis/). > > A fabricated DOI looks exactly like a real one. Evidentia is the open-source tool that resolves each one and tells you which is which. > ⚕️ **Scope:** Evidentia is a pre-publication aid for **writers, editors, and researchers — not clinical decision support.** It does not diagnose, treat, or replace professional medical judgment. ## 30-second start **As a command-line tool** (no install, no API key): ```bash npx evidentia check your-article.md ``` **As a Claude Code skill** (full 15-criteria appraisal): ```bash /plugin marketplace add kgraph57/evidentia /plugin install evidentia@evidentia ``` Then just say: *"Fact-check this article"* / *「この記事をファクトチェックして」*. **As a Codex Desktop plugin**: ```text Marketplace source: https://github.com/kgraph57/evidentia.git Plugin: evidentia ``` ## What it catches Here is Evidentia run on a real AI-generated answer about vitamin D and childhood infections — four citations, formatted perfectly, all plausible: ```text $ npx evidentia check examples/inputs/ai-generated-answer.md Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrication rate) [OK ] doi:10.1136/bmj.i6583 — Paper exists and the cited metadata matches the registry record. [HAL] doi:10.1056/nejmoa2105512 — DOI does not resolve in CrossRef or OpenAlex, and no matching paper was found. [HAL] pmid:18768876 — Identifier resolves to a different paper ("Trafficking of antigen-specific CD8+ T lymphocytes…") than the one cited. [MIS] doi:10.1002/14651858.cd012734 — Paper exists, but cited metadata disagrees with the record (year). ``` One citation was real. One DOI was invented. One PMID pointed to an unrelated paper. One had the wrong year. **A human reviewer would have to check all four by hand.** Evidentia did it in seconds. See the [full report](examples/reports/ai-generated-answer.report.md). > This is a deliberately tough example. Most carefully written articles score far lower — Evidentia's value is catching the handful that slip through, every time, without fatigue. ## The 4-tier classification Most "citation checkers" stop at *"could not verify."* Evidentia keeps going — it resolves the identifier and tells you **why** a citation is suspect: | Tier | Verdict | Meaning | |:----:|---------|---------| | ✅ **1** | **Verified** | The paper exists and the cited title/authors/year/journal match the registry record. | | ⚠️ **3** | **Bibliographic mismatch** | A real paper exists, but the DOI/PMID is wrong, or the metadata disagrees (a real source cited carelessly — or a fabricated identifier bolted onto a real title). | | ❌ **4** | **Hallucination** | The identifier resolves to nothing, or resolves to a *completely different* paper. This is the signature of AI-generated text. | | 🔍 **2** | **Content review needed** | The paper is real, but whether it's used *in the right context* needs a human or an LLM. Handled by the Evidentia skill, below. | ## Two layers: deterministic engine + LLM appraisal Evidentia is deliberately split into a part a computer can do perfectly and a part that needs judgment: **1. The engine (CLI + MCP server)** — pure, deterministic citation verification. No API key, no LLM, no hallucination of its own. It answers one question with certainty: *does this cited paper actually exist, and does the identifier point to it?* Use it in a terminal, in CI, or as an MCP tool inside any agent. **2. The skill (Claude Code)** — wraps the engine in a full **15-criteria critical-appraisal rubric**: evidence level, statistical interpretation (relative vs. absolute risk, NNT), causation vs. correlation, conflicts of interest, exaggeration, population fit, ethics, and more — producing an **A–F report** with concrete fixes. This is the Tier-2 "is it used correctly?" layer the engine can't do alone. You can use either on its own. Together they cover citation *existence* (deterministic) and citation *honesty* (appraisal). ## Use it as an MCP tool Give any agent the ability to verify citations: ```bash claude mcp add evidentia -- npx -y evidentia-mcp ``` The server exposes one tool, `verify_citations(text)`, returning the tiered report as Markdown or JSON. JSON responses include both the public 4-tier verdict and a machine-readable `lookupVerified` / `resolverOutcomes` trace for agents and CI. ## Use it in CI Block a pull request that introduces a fabricated citation. Drop [`.github/workflows/evidentia.yml`](examples/ci/evidentia.yml) into any medical-content repo: ```yaml - run: npx evidentia check content/**/*.md --fail-on-fabrication ``` `--fail-on-fabrication` exits non-zero if any citation is a mismatch or hallucination. ## The 15-criteria skill (Claude Code) When invoked as a skill, Evidentia evaluates medical content across 15 dimensions and adapts to the media type — research paper, news article, social post, patient leaflet, conference slide, guideline, pharma marketing, or AI-generated text. The 15 criteria 1. Evidence level & study design 2. Citation & source accuracy *(powered by the engine above)* 3. Statistical interpretation 4. Causation vs. correlation 5. Bias & conflicts of interest 6. Exaggeration & overclaiming 7. Target population fit 8. Temporal validity 9. Jargon–readability balance 10. Ethical considerations 11. Logical consistency 12. Images & figures 13. Alternative explanations 14. Clinical relevance 15. Information completeness Each item is rated **Excellent / Good / Fair / Poor**, aggregated into an overall **A–F** score with a **public-health risk level** (LOW / MEDIUM / HIGH). See [`skills/medical-fact-check/SKILL.md`](skills/medical-fact-check/SKILL.md). ## Works with your agent The skill follows the open [Agent Skills](https://code.claude.com/docs/en/skills) `SKILL.md` standard, so it loads in Claude Code today and in any agent that adopts the format. The engine is a plain npm package and an MCP server — usable from Claude Code, Codex CLI, Cursor, or your own scripts. ## Install ### CLI ```bash # one-off, no install npx evidentia check article.md # or install globally npm install -g evidentia evidentia check article.md --format md --out report.md ``` ```text evidentia check Verify citations in a file, web page, or stdin --format Output format (default: text) --out Write the report to a file --mailto Contact email for the CrossRef/OpenAlex polite pool --cache Reuse registry HTTP responses from a local JSON cache --fail-on-fabrication Exit 1 if any citation is mismatch/hallucination (CI) --offline Extraction only, no network ``` ### Claude Code skill / plugin ```bash /plugin marketplace add kgraph57/evidentia /plugin install evidentia@evidentia ``` Or copy the skill manually: ```bash git clone https://github.com/kgraph57/evidentia.git cp -r evidentia/skills/medical-fact-check ~/.claude/skills/ ``` ## How verification works For each citation, Evidentia extracts every identifier (DOI, PMID, arXiv, **NCT trial ID, ISBN**) and any nearby title/author/year, then: 1. **Resolves the DOI** against CrossRef, falling back to OpenAlex. 2. **Resolves the PMID** against PubMed E-utilities. 3. **Resolves arXiv IDs** against the arXiv API. 4. **Resolves NCT trial IDs** against ClinicalTrials.gov. 5. If the identifier doesn't resolve, **searches by title** in OpenAlex — this is how it distinguishes *"real paper, wrong DOI"* (Tier 3) from *"this paper does not exist"* (Tier 4). 6. **Compares** the cited title/authors/year against the registry record to catch an identifier that silently points to a different paper. Each JSON citation also includes `lookupVerified` (`true`, `false`, or `unresolvable`) and `resolverOutcomes` (`matched`, `unmatched`, `unreachable`, or `skipped`, with whether the lookup was keyed by an identifier or a title). This keeps agent workflows auditable without changing the human-facing 4-tier verdict. It is deliberately careful about **what it does *not* flag**: a book (ISBN), a clinical guideline, or any source that isn't indexed in these registries is marked *"verify manually"* (Tier 2), never *"hallucination"* — only a failing DOI/PMID/arXiv/NCT identifier (which is *supposed* to resolve) earns a fabrication verdict. Identifier-less entries in a reference list are surfaced for review rather than silently skipped. All registries are free and keyless. Pass `--mailto` to join the faster "polite pool." ## Examples | Input | Result | |-------|--------| | [AI-generated answer](examples/inputs/ai-generated-answer.md) (real + fabricated mix) | [75% fabrication rate](examples/reports/ai-generated-answer.report.md) | | [Clean reference list](examples/inputs/clean-references.md) (all real) | [0% — all verified](examples/reports/clean-references.report.md) | ## Limitations - The **engine** verifies citation *existence and bibliographic accuracy* — not whether a real paper is summarized correctly. That semantic check is the skill's job (Tier 2), and even then it relies on what's publicly indexed (abstracts, open-access full text, metadata). - This is **decision support for writers, editors, and researchers — not clinical decision support.** It does not diagnose, treat, or replace professional medical judgment. - Very new papers may not yet be indexed, which can surface as a false "unverified." Re-run later, or pass `--mailto` for the freshest index. ## Roadmap - [x] Batch input — `evidentia check a.md b.md …` with an aggregate report ([benchmark: 22 cases](benchmark/)) - [x] arXiv ID verification, resolver traces, and optional local lookup cache - [ ] `evidentia-bench` — grow to 100+ cases and publish per-model fabrication rates - [ ] CrossRef/OpenAlex abstract retrieval to assist Tier-2 context checks - [ ] More media presets in the skill Track it on the [pinned roadmap issue](https://github.com/kgraph57/evidentia/issues). ## Contributing Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md). ## About the author Built by **Ken Okamoto, MD** — a board-certified pediatrician and medical-AI entrepreneur. Evidentia comes out of the daily problem of separating real evidence from confident-sounding fabrication in AI-assisted medical writing. ## License [MIT](LICENSE) > *Evidentia* (the citation-verification tool documented here) is unrelated to "Evidentia" genealogy software. ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [kgraph57](https://github.com/kgraph57) - **Source:** [kgraph57/evidentia](https://github.com/kgraph57/evidentia) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-kgraph57-evidentia - Seller: https://agentstack.voostack.com/s/kgraph57 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.