Citecheck

CiteCheck — Paper Citation Verification

Overview

CiteCheck verifies academic paper citations by combining structured parsing with agent-native LLM evaluation. It supports LaTeX source files (preferred) and PDF fallback.

Workflow

1. Parse paper: Call citecheck CLI to read LaTeX (preferred) or PDF, extract references and body text
2. Format check: Call citecheck CLI to validate bibliography entries
3. Queryability verification: Call citecheck CLI to verify existence via Crossref / Semantic Scholar
4. Evaluate thematic relevance: Evaluate directly using the agent's reasoning capabilities — compare cited paper title/abstract/venue against the citing paper
5. Evaluate semantic accuracy: Evaluate directly using the agent's reasoning capabilities — compare in-text citation context against cited source content
6. Generate report: Aggregate all results into a Markdown report

> Why matching steps are not done by the CLI > The citecheck CLI can run standalone with optional --api-key for external LLM-powered matching. When used as a Skill, the host agent itself possesses LLM reasoning capabilities. Direct evaluation is faster, more consistent, and requires no additional API keys from the user.

---

1. Parse Paper (CLI)

LaTeX source (preferred)

citecheck path/to/latex_project/ --skip-verification --skip-semantic -o parsed_report.md

Or parse a single file:

citecheck main.tex --skip-verification --skip-semantic -o parsed_report.md

PDF (fallback)

citecheck paper.pdf --skip-verification --skip-semantic -o parsed_report.md

> --skip-verification and --skip-semantic are required in Skill mode because steps 3–5 are performed directly by the agent.

---

2. Format Check (CLI)

Call citecheck to check and report format issues:

| Check item | Description | |------------|-------------| | Required fields | Author, title, year, venue completeness | | Format consistency | Punctuation, capitalization, abbreviation uniformity | | DOI/URL | If present, whether format is correct and accessible | | Year sanity | No 202x placeholders, not in the future |

Detailed rules: see [./references/format-check-rules.md](./references/format-check-rules.md).

---

3. Queryability Verification (CLI + Agent Supplement)

Call citecheck to query Crossref and Semantic Scholar public APIs for citation existence.

API details: see [./references/api-reference.md](./references/api-reference.md).

If the CLI fails due to network/SSL issues, use WebSearch to directly query suspicious citations (especially those marked "unverifiable"), supplementing the verification results.

---

4. Evaluate Thematic Relevance (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each reference, extract:

• Citing paper: title, abstract, keywords
• Cited paper: title, abstract (from API results or WebSearch), venue

Evaluation prompt template: see [./references/thematic-scoring-prompt.md](./references/thematic-scoring-prompt.md).

---

5. Evaluate Semantic Accuracy (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each in-text citation position:

1. Extract 1–2 sentences before and after the citation marker as citing_text
2. Obtain the cited paper's abstract via Semantic Scholar or WebSearch as cited_text
3. Evaluate semantic consistency using the prompt template: see [./references/semantic-matching-prompt.md](./references/semantic-matching-prompt.md).

---

6. Output Report

Generate a Markdown report containing:

Summary

• Total references
• Format issues count
• Query failures / suspicious count
• Average thematic relevance score
• Average semantic accuracy score

Detailed Results Table

| No. | Title | Format | Queryable | Thematic | Semantic | Notes | |-----|-------|--------|-----------|----------|----------|-------| | 1 | ... | ✅/⚠️/❌ | ✅/❌ | 0.85 | 0.90 | DOI mismatch |

Issue Summary

• List all findings and recommendations by severity

---

Dependencies

pip install CiteCheck

For PDF support:

pip install CiteCheck[pdf]

---

Notes

• No --api-key is required in Skill mode. Thematic and semantic matching are performed directly by the agent, with no external LLM API calls.
• Prioritize LaTeX source files; parsing accuracy is far higher than PDF.
• Add delays and retries to API calls to avoid rate limiting.
• When Semantic Scholar abstracts are missing, fall back to Crossref + title keywords.
• Semantic matching requires the cited paper's abstract; mark "source unreachable" when unavailable.
• If placeholder citations like =? or [?, ?] are found, mark them as "unverifiable references".

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: color4-alt
• Source: color4-alt/CiteCheck
• License: MIT

Install and usage instructions live in the source repository linked above.