Checkapp

CheckApp

> AI content quality gate for marketing teams. CLI + source-repo web dashboard that returns plagiarism, AI-detection, SEO score, fact-check, tone-of-voice, legal risk, brief matching, and content summary — before you publish. Supports context management (tone guides, briefs, legal policies), MCP server for AI agent integration, batch checking, CI mode, JSON output, tags, search, report export, and a local dashboard for browsing results and managing skills when running from the source repo.

[](https://github.com/sharonds/checkapp/actions/workflows/ci.yml) [](https://www.npmjs.com/package/checkapp) [](https://github.com/sharonds/checkapp) [](LICENSE) [](https://bun.sh) [](CONTRIBUTING.md)

---

What Is CheckApp?

CheckApp is a pluggable CLI tool that runs a configurable set of quality checks on any article — a Google Doc URL or a local .md/.txt file — before it goes live.

Each check is a skill you can enable or disable. Results appear in the terminal and are automatically saved as an HTML report and to a local SQLite history database.

---

Phase 7 — Research-Backed Editor

Every flagged issue ships with evidence + rewrite + citation:

• Fact-check now carries sources[] (Exa highlights with url/title/quote) on every finding. Upgrade to deep-reasoning with --deep-fact-check.
• Grammar & Style (LanguageTool + LLM fallback) produces a rewrite per finding. LLM-fallback rewrites are grammar-checked a second time to catch mechanical errors.
• Academic Citations (OpenAlex recommended, Semantic Scholar legacy) merges citations onto matching fact-check findings with scientific/medical/financial claim types. Free, no API key — see [Academic Citations](#academic-citations) below.
• Self-Plagiarism (Cloudflare Vectorize + OpenRouter embeddings) flags overlap with your past articles. Run checkapp index once to ingest your archive.

Pick a provider per skill from the Settings → Providers dashboard. CheckApp stores API tokens only in the local user config or reads them from environment variables — users bring their own keys.

Pre-flight cost estimate: checkapp --estimate-cost article.md or the Run Check page in the dashboard shows "Estimated cost: $0.0320" before spending anything.

See [docs/security.md](docs/security.md) for the BYOK-alpha threat model.

---

Skills

| Skill | Engine | Cost/check | Enabled by default | |-------|--------|-----------|-------------------| | Plagiarism | Copyscape · Gemini Grounded Plagiarism | ~$0.09 / ~$0.04 estimate | ✅ | | AI Detection | Copyscape (English) · Gemini 3.1 Pro Preview (multilingual) | ~$0.03 / ~$0.01 estimate | ✅ | | SEO | Offline (no API) | free | ✅ | | Grammar & Style | LanguageTool + LLM fallback | free tier / ~$0.002 | ❌ disabled by default (enable in Settings; LanguageTool free tier works without any API key) | | Academic Citations | OpenAlex (default) / Semantic Scholar (legacy) | free | ❌ disabled by default (augments fact-check findings when enabled; OpenAlex/SS both free) | | Self-Plagiarism | Cloudflare Vectorize + OpenRouter embeddings | ~$0.0001 | ❌ disabled by default — requires index (checkapp index ), OPENROUTER_API_KEY, and Cloudflare Vectorize provider config | | Fact Check | Tiered: Basic = Exa + LLM; Standard = Gemini + Google Search; Deep Audit = Gemini Deep Research | varies | Basic is available by default; Standard is opt-in; Deep Audit is async | | Tone of Voice | Claude/MiniMax | ~$0.002 | ❌ requires LLM key + tone guide file | | Legal Risk | Claude/MiniMax | ~$0.002 | ❌ requires LLM key | | Content Summary | Claude/MiniMax | ~$0.002 | ❌ requires LLM key | | Brief Matching | MiniMax/Claude | ~$0.002 | ❌ requires LLM key + brief context | | Content Purpose | MiniMax/Claude | ~$0.002 | ❌ requires LLM key |

All enabled skills run in parallel. Adding more skills does not increase total time significantly.

Academic Citations

CheckApp finds peer-reviewed supporting papers for scientific, medical, and financial claims.

Recommended provider: OpenAlex. Free, ~250M indexed works, no API key required. Set OPENALEX_MAILTO=your@email.com in your .env to enable it — this both activates the routing (skill is skipped if unset and no explicit provider is configured) and joins the polite pool (100k req/day).

Legacy provider: Semantic Scholar. Users with an explicit providers.academic = { provider: "semantic-scholar" } config continue to hit SS. Note: the free tier of SS has aggressive per-IP rate limiting and is effectively unusable on shared IPs — that's why OpenAlex is the new default. Authenticated (paid) SS requests are not currently wired in the client; support for a paid SS API key is a separate workstream.

OpenAlex is the recommended provider because it is free, broad, and avoids Semantic Scholar's aggressive unauthenticated rate limits.

---

Fact-Check Tiers

Standard is opt-in and stays off by default until Gate 2 passes. Basic remains the default tier unless factCheckTierFlag is explicitly enabled.

| Tier | Engine | Cost per article | Typical time | Notes | |------|--------|------|------|-------| | Basic (default) | Exa + LLM | $0.04 | ~15s | Requires Exa plus an LLM key for claim extraction/assessment | | Standard (opt-in) | Gemini 3.1 Pro + Google Search grounding | $0.16 | ~45s | Requires Gemini. Enable with factCheckTierFlag=true and factCheckTier="standard", or set providers["fact-check"].provider = "gemini-grounded". | | Deep Audit (async) | Gemini Deep Research | $1.50 | 5–15 min | Premium async audit workflow. The normal sync check still runs Basic unless Standard is selected. Initiate via dashboard button or deep_audit_article MCP tool. |

Basic, Standard, and sync deep-reasoning fact-checking default to checking up to 4 extracted claims per article to keep latency and provider cost predictable. Advanced users can adjust factAudit.standardMaxClaims, factAudit.deepMaxClaims, and provider-call budgets in config; structured reports show skipped-claim counts and budget reasons such as claim_cap when a cap stops verification.

Research basis: the Standard tier was selected based on an internal benchmark on a 20-claim synthetic corpus. That benchmark is directional, not definitive - see its LIMITATIONS.md before relying on the results for your own decisions.

Confidence and limitations

CheckApp reports evidence confidence, not certainty. Gemini grounded fact-check has the strongest signal on concrete dates, statistics, named entities, and claims where Google Search returns source URLs. If Gemini or Exa marks a claim as supported but no source URL is attached, CheckApp downgrades that claim to unverified. “No issues found” does not prove the article is accurate or original across the whole web.

Gemini grounded plagiarism is stricter than a general similarity prompt: matched URLs that appear in Gemini grounding metadata are treated as grounded matches. If Gemini returns plausible match JSON without grounding metadata, CheckApp reports a reduced-confidence review finding instead of a clean pass. Exact public-source English and Hebrew copying is the highest-confidence path we validate. Translated or paraphrased plagiarism is lower confidence and should still receive human review.

Provider failures (HTTP 429/500/502/503/504 and network errors) are retried within a configurable retry budget; claims that still fail after retries are reported as provider errors and force a warn verdict, never a pass.

Coverage audit details

Fact-check and plagiarism checks now attach structured audit details when provider-backed checks run. Each actionable finding can include the exact article quote, section/paragraph/sentence location, character offsets when available, language/direction metadata for Hebrew, English, or mixed text, confidence rationale, evidence sources, provider/search metadata, and a suggested rewrite in the article language. Character offsets are JavaScript/UTF-16 code-unit offsets. Fuzzy token-overlap locations are labeled approximate and omit exact character offsets. Coverage metadata records checked/skipped claims and budget stop reasons. Dashboard and HTML detail reports show this context; list/search APIs return only summaries so quotes and evidence are not exposed outside detail views.

HTML reports and dashboard audit drilldowns localize CheckApp-owned labels for English and Hebrew. Hebrew audit reports use RTL layout and Hebrew labels for locations, evidence, suggested rewrites, verdicts, and audit summaries. Provider names, URLs, titles, and quoted source/article text are preserved rather than translated.

When the same quote appears multiple times and the provider does not return source offsets, CheckApp resolves the quote to the first deterministic matching location. Reports preserve this behavior as a known limitation; provide more specific quoted text or offsets when exact duplicate disambiguation matters.

---

Features

| Feature | Details | |---------|---------| | Pluggable skills | Enable/disable any skill via config. Add custom skills by implementing one TypeScript interface. | | Plagiarism check | Checks against Copyscape's indexed web data by default, or Gemini Grounded Plagiarism when selected. Returns 0–100% similarity + matched sources. | | AI detection | Copyscape AI detector by default, or Gemini 3.1 Pro Preview when explicitly configured for multilingual checks. Returns 0–100% probability per sentence and an overall verdict. | | SEO analysis | Offline. Checks word count (800–2500 ideal), H1/H2 headings, average sentence length, Flesch-Kincaid readability. | | Fact check | Basic extracts specific claims, searches each with Exa AI, and uses the configured LLM to assess evidence. Standard uses Gemini 3.1 Pro with Google Search grounding. | | Tone of voice | Loads your brand voice guide (.md file), sends article + guide to the configured LLM, returns violations with quotes and rewrite suggestions in your brand voice. | | Legal risk | Scans for unsubstantiated health claims, defamation, false promises, GDPR risks, price misrepresentation. Findings include actionable "Fix:" suggestions. | | Content summary | Analyzes topic, main argument, target audience, and tone (informational/persuasive/conversational/technical/promotional). | | SEO keyword detection | Extracts the top keyword and checks whether it appears in the first paragraph. | | Fact-check confidence | Each claim now shows high/medium/low confidence based on the number of supporting sources found. | | Batch checking | Check all .md/.txt files in a directory with checkapp --batch ./articles/. | | Configurable thresholds | Custom pass/warn/fail score cutoffs per skill via config.json. | | HTML report | Self-contained, no-dependency HTML file. Score bars, verdict badges, per-finding citations. Opens in browser automatically. | | SQLite history | Every check is saved to ~/.checkapp/history.db. Query with --history. | | Google Doc support | Paste a publicly-shared Google Doc URL including ?tab=t.xxx for specific tabs. No Google auth required. | | Local file support | Pass a .md or .txt file path. Works offline for the fetch step. | | Single binary | CLI binaries can run checks without Node.js, Bun, or a runtime. The dashboard is not bundled into the binary. | | Web dashboard | Local Next.js UI — overview stats, report browser, run checks, manage skills and settings, in-app docs. Run from a source checkout. | | --ui flag | Launches the dashboard dev server when the sibling dashboard/ directory is present, such as in a source checkout. Published npm/binary installs expose CLI/MCP features but do not bundle the dashboard app. | | --output export | --output report.md or --output report.html — save the report to a file. | | Tags + search | Attach tags to checks, search check sources and tags via dashboard or API. | | JSON API | RESTful API at localhost:3000/api for running checks, managing tags, toggling skills. See [docs/api.md](docs/api.md). | | Context system | Upload tone guides, content briefs, legal policies, and style guides. Contexts are stored in SQLite and automatically loaded by relevant skills. Manage via CLI (checkapp context add/list/show/remove) or the dashboard Contexts page. | | MCP server | 10 tools for AI agent integration (Claude Code, Cursor, Windsurf). Start with checkapp --mcp. Tools include check_article, list_reports, get_report, upload_context, list_contexts, get_skills, toggle_skill, regenerate_article, deep_audit_article, and get_deep_audit_result. | | CI mode (--ci) | Exits with code 1 if any skill returns a fail verdict. Designed for CI/CD pipelines. | | JSON output (--json) | Outputs structured JSON instead of the Ink terminal UI. Ideal for scripts, agents, and piping. | | Brief matching | Checks article against an uploaded content brief. Verifies coverage of required topics, audience alignment, and tone match. Requires a brief context. | | Content purpose detection | Detects article type (tutorial, product announcement, case study, thought leadership, etc.) and provides purpose-specific recommendations for missing elements. | | Regenerate/fix | checkapp --fix runs all checks then generates AI-suggested rewrites for every flagged sentence, using tone guide and legal policy contexts. | | Cross-platform | Mac (Apple Silicon + Intel), Linux, Windows. |

---

Real Results — What It Finds

Example 1 — English article with Wikipedia passages

An article about Vitamin D with 3 verbatim sentences from Wikipedia. Live output with the default 3 skills:

────────────────────────────────────────────────
Words checked:  310
API cost:        $0.080

❌   Plagiarism Check:  33% similarity — 18 sources matched  (34/100)
✅   AI Detection:  10% AI probability — human  (90/100)
❌   SEO:  310 words · avg 17-word sentences · readability: Medium  (49/100)
────────────────────────────────────────────────
Overall: 58/100
Report: checkapp-report.html
────────────────────────────────────────────────

HTML report: Each skill gets a card with a circular score indicator, engine badge, and a list of findings. The report links to all engines used (Copyscape, Gemini, Exa AI, MiniMax) and includes an MIT disclaimer.

---

Example 2 — Hebrew article with Wikipedia passages

Three sentences from the Hebrew Wikipedia article on Vitamin D:

────────────────────────────────────────────────
Words checked:  119
Plagiarism:      39%  (46 / 118 words matched)

Top match: he.wikipedia.org/wiki/ויטמין_D
  ↳ "ויטמין D הוא קבוצה של חמש תרכובות מסיסות בשמן..."
  ↳ "המחלה הנפוצה ביותר הנגרמת כתוצאה ממחסור בוויטמין D..."

AI detection:    12%  probability AI-generated
────────────────────────────────────────────────
❌  REWRITE — similarity too high
✍️  HUMAN — 12% AI probability
────────────────────────────────────────────────

Hebrew content and RTL rendering are supported. Provider configuration still matters: Copyscape AI detection is English-only, so use gemini-ai-detection for Hebrew AI detection; use gemini-grounded-plagiarism for Hebrew plagiarism checks when Copyscape cannot cover the language.

---

Example 3 — Hebrew article with 2 sentences from an Israeli news site (Ynet)

Only 2 sentences copied out of ~220 words of original Hebrew content:

────────────────────────────────────────────────
Words checked:  222
Similarity:      33%  (76 / 224 words matched)

Top match: ynet.co.il/articles/0,7340,L-4870486,00.html  76 words
  (syndicated copy also found at news08.net)
────────────────────────────────────────────────
❌  REWRITE — similarity too high
──

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [sharonds](https://github.com/sharonds)
- **Source:** [sharonds/checkapp](https://github.com/sharonds/checkapp)
- **License:** MIT

Install and usage instructions live in the source repository linked above.