# Paper Extraction > > - **Type:** Skill - **Install:** `agentstack add skill-lightconeresearch-lightcone-cli-paper-extraction` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [LightconeResearch](https://agentstack.voostack.com/s/lightconeresearch) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** BSD-3-Clause - **Upstream author:** [LightconeResearch](https://github.com/LightconeResearch) - **Source:** https://github.com/LightconeResearch/lightcone-cli/tree/main/claude/lightcone/skills/paper-extraction - **Website:** https://docs.lightconeresearch.org/ ## Install ```sh agentstack add skill-lightconeresearch-lightcone-cli-paper-extraction ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # paper-extraction Turn a DOI or arXiv ID into a standardized, indexed `work/reference/` directory. One entry-point, idempotent, self-contained. The output is a predictable surface anyone can rely on without re-parsing LaTeX. What a consumer does with that surface is their concern — paper-extraction's job ends at the index. ## When to use - "Read [paper] end-to-end" / "I want to verify a claim in [paper]" — full source plus structured artifacts so you're reading the actual paper, not a flattened PDF - "Set up reading materials for [paper]" — when the next thing you'll do involves browsing figures, citations, or section structure and you don't want to grep the tarball every time - Any workflow where another skill or process needs a known directory shape per paper ## Outputs Under `work/reference/` (idempotent — skips work already done): ``` work/reference/ ├── index.json # structural index — figures, tables, outline, citations (with DOIs), paths ├── astra.yaml # ASTRA-shape representation: the paper as an ASTRA artifact, including findings ├── paper.pdf # always ├── paper.tex # Path A — symlink to the main .tex file │ (or) ├── document.md # Path B — Docling-extracted markdown ├── source/ # Path A — extracted arXiv tarball (full source tree) ├── figures/ # figure files (copied from LaTeX or rendered by Docling) ├── tables/ # one .tex file per `\begin{table}` block (Path A) ├── bibliography-source.bib # Path A only — copy of any .bib found in source/ ├── bibliography-source.bbl # Path A only — copy of any .bbl found in source/ └── .doi-cache.json # Crossref/ADS lookup cache for re-run idempotency ``` The skill produces only the paper's own reading materials. Anything not contained in or derived from the paper itself — code repositories, supplementary datasets, related papers — is out of scope; the caller handles those. ### Two surfaces: `index.json` (structural) and `astra.yaml` (semantic) **`index.json` is structural and machine-friendly.** Everything the script could mechanically extract: figures, tables, section outline with line numbers, citation keys with every location *plus the cited paper's full citation text and resolved DOI*, abstract, paths. Read this when you want to know "what's in this paper, where do I find it." Sample shape: ```json { "schema_version": 1, "path": "A", // or "B" "paper_pdf": "paper.pdf", "paper_tex": "paper.tex", // null on Path B "source_dir": "source", // null on Path B "document_md": null, // "document.md" on Path B "bibliography_source_bib": "bibliography-source.bib", "bibliography_source_bbl": null, "astra_yaml": "astra.yaml", "title": "UNIONS-3500 Weak Lensing: B-mode validation", "abstract": "At Stage-III sensitivities, cosmic shear B modes ...", "figures": [ {"id": "fig1", "label": "fig:bao", "caption": "...", "source_path": "fig_bao", "file": "figures/fig_bao.pdf", "block_origin": "main.tex", "line": 412} ], "tables": [ {"id": "tab1", "label": "tab:cosmo", "caption": "...", "file": "tables/tab-cosmo.tex", "block_origin": "main.tex", "line": 487} ], "outline": [ {"level": 1, "title": "Introduction", "label": "sec:intro", "source_file": "main.tex", "line": 157} ], "citations": { "asgari17": { "locations": [{"file": "main.tex", "line": 178}, {"file": "main.tex", "line": 561}], "citation": "Asgari, M., et al. (2017) KiDS-450: Tomographic cross-correlation cosmic shear results. MNRAS 464, 1676-1692", "doi": "10.1093/mnras/stw2606" }, "planck18_lensing": { "locations": [{"file": "main.tex", "line": 92}], "citation": "Planck Collaboration, et al. (2020) Planck 2018 results. VIII. Gravitational lensing. A&A 641, A8", "doi": "10.1051/0004-6361/201833886" } }, "extraction_warnings": [ "figure fig3: \\includegraphics{...} could not resolve to a file in source/", "citation kuijken:2011: could not resolve DOI; tried doi-field, eprint-field, Crossref, ADS" ] } ``` The `citations:` block maps each cited paper's BibTeX key (Path A) or synthetic `_` key (Path B) to `{locations, citation, doi}`. Downstream consumers (e.g. lc-from-paper's SPECIFY when authoring `prior_insights:` placeholders, LITERATURE when discovering which DOIs to fetch) read the DOI directly from `citations[key].doi`. Unresolvable entries keep `citation: null` and/or `doi: null` and are flagged in `extraction_warnings`. **`astra.yaml` is semantic and ASTRA-validating.** Treats the paper as an ASTRA artifact: `id`, `version`, `name`, a top-level `description:` (from the abstract), and `findings:` carrying the paper's claimed numerical results in ASTRA's Insight + Evidence shape. Read this when you want to know "what does this paper claim, with quote evidence anchored to the source." The script writes a stub (id, version, name, `description` from abstract, empty findings); Step 5 fills in `findings:`. Why both: the structural index is queryable by any consumer (`grep`, `jq`, agent code) without needing to know about ASTRA. The ASTRA file composes directly into reproductions, MySTRA, and any other ASTRA-aware tool — and the verbosity of the Insight + Evidence shape *is* the back-pressure against hallucinated numerical claims (the agent has to find and quote the actual text). ## Workflow ### Step 1 — Survey Always start with `ls work/reference/` and read `index.json` if present. Skip the work that's already done: | File present | Step to skip | |---|---| | `source/` (Path A) or `document.md` (Path B) + `paper.pdf` | Substrate acquired (Step 2) | | `index.json` with non-empty figures/tables/outline | Structural extraction done (Step 3) | | `astra.yaml` exists | Stub written; never overwritten on re-run (preserves agent edits) | | `astra.yaml` has non-empty `findings:` populated | Findings step done (Step 5, optional) | If nothing is present, run the full workflow. ### Step 2 — Acquire substrate Pick the path on entry from the input form: - **arXiv ID** (e.g. `2503.19441`) → **Path A** (LaTeX source primary) - **DOI** for an arXiv paper (e.g. `10.48550/arXiv.2503.19441`) → Path A (resolve to arXiv ID first) - **Journal DOI** without arXiv preprint → **Path B** (PDF + Docling fallback) Read [`references/arxiv-source.md`](references/arxiv-source.md) for Path A; [`references/pdf-fallback.md`](references/pdf-fallback.md) for Path B. Both end with `work/reference/paper.pdf` and a structured-text representation under `work/reference/`. ### Step 3 — Run the extraction script `scripts/extract-paper-substrate.py` does the deterministic structural pass and writes the `astra.yaml` stub: ```bash python3 .claude/skills/paper-extraction/scripts/extract-paper-substrate.py \ --arxiv-id # or --doi ``` The script detects the path automatically and produces: - `figures/` populated with copied figure files (Path A) or untouched (Path B — Docling already populated it) - `tables/.tex` — one file per `\begin{table}` block (Path A only) - `bibliography-source.{bib,bbl}` if present in the source tarball (Path A only) - `index.json` — the unified structural index, including the enriched `citations:` block (each cited key carries `{locations, citation, doi}`; DOI resolution covers ~96% of typical-paper bibliographies) - `astra.yaml` — stub ASTRA representation: id, version, name (from `\title{}`), `description` (from abstract), empty `findings: {}` for Step 5 - `.doi-cache.json` — Crossref/ADS lookup cache; re-runs skip the network for already-seen entries The `--arxiv-id` / `--doi` argument populates the `id` and the evidence `doi:` field in `astra.yaml`. If neither is provided, the script writes placeholder text the agent can fix. The DOI resolver tries, in order: the entry's `doi:` field → an `eprint:`-derived arXiv DOI → Crossref bibliographic query (free, no API key needed) → ADS title search (only if `ADS_API_TOKEN` env var or `~/.ads/dev_key` is present — graceful skip when absent). Title hits from Crossref are gated by a similarity check against the queried title to drop noisy false matches. ### Step 4 — Review the script's output and fix structural gaps The script is purely deterministic. It walks the structural surface but does not understand the paper. Read `index.json`'s `extraction_warnings` and address each: - **`figure figN: \includegraphics{X} could not resolve`** — the LaTeX referenced a file the script couldn't find. Search the source tree manually (sometimes figures live in non-standard subdirectories with non-standard extensions); copy the file into `figures/` and update the corresponding `index.json` entry's `file` so it's no longer null. - **`figure figN: no \caption found`** — composite figures (subfloats) sometimes lack a top-level caption; verify the figure block in source and either record the per-subfigure captions in `caption` or note that the figure is composite. - **`table tabN: no \label`** — verify the table is intentional (some `\begin{table}` blocks are non-tabular layout); rename or annotate as needed. - **`citation : could not resolve DOI`** — the entry has no `doi:` / `eprint:` field, and neither Crossref nor ADS (when available) returned a match. The entry stays in `citations:` with `doi: null`; a downstream consumer can flag it for human resolution or skip it. If many entries are unresolved, check that the title field is clean (sometimes `.bib` titles carry uncleaned LaTeX commands that drag down the Crossref similarity gate). Delete `.doi-cache.json` to force re-resolution. - **`citation : cited in source but no matching entry in bibliography-source.{bib,bbl}`** — a `\cite{}` invocation has no corresponding bib record. Usually a typo in the LaTeX source; flag it and move on. The entry stays in `citations:` with `citation: null, doi: null`, locations preserved. - **Path B caveat** — outline extraction is not yet implemented for the Docling fallback. Bibliography resolution works on Path B by parsing the references section at the tail of `document.md` and synthesizing keys (`_`), but citation *invocations* from rendered prose aren't yet extracted — Path B citations carry empty `locations: []`. The warnings list flags this. Also eyeball `astra.yaml`'s `name:` and `description:`. The title or abstract may contain unresolved custom `\newcommand` macros (defined elsewhere in the source); the script doesn't expand macros, so they pass through verbatim. Clean them up if you need pretty rendering downstream — none of this blocks validation. ### Step 5 — *(Optional)* Walk the paper for findings, append to `astra.yaml` **Skip unless a downstream consumer needs `findings:` populated.** Steps 1–4 produce a complete `work/reference/` and a valid (empty-findings) `astra.yaml` on their own. Reproductions and diff workflows need findings; reading and browsing don't. When you do run Step 5: for each **central numerical claim the paper makes about its results** — headline measurements, structural conclusions ("we detect X at Y σ"), validated null-test outcomes — append a finding to `astra.yaml`'s `findings:` map. *Not* methodology choices or dataset descriptions; those live elsewhere. Shape (per ASTRA's [Insight + Evidence](https://w3id.org/ASTRA/insight) classes): ```yaml findings: s8_constraint: id: s8_constraint claim: "S_8 = sigma_8 (Omega_m / 0.3)^0.5 = 0.795 ± 0.014 from the fiducial pure E/B analysis" created_at: "2026-04-04T00:00:00Z" evidence: - id: abstract_quote doi: "10.48550/arXiv.2604.03227" version: 1 quote: exact: "we find $S_8 = 0.795 \\pm 0.014$" ``` **Discipline:** - **Read the abstract and conclusions first.** Most central findings can be quoted from one of those two surfaces. - **`quote.exact` is verbatim.** Copy LaTeX as it appears in `paper.tex` — don't paraphrase, don't expand macros, don't normalize math. `astra validate --verify-evidence` searches for this string in the cached PDF; paraphrasing breaks the gate. If the quote isn't unique, add `prefix:` / `suffix:` (~20–100 chars) per W3C TextQuoteSelector. - **Every evidence carries `doi:`** (the paper's own DOI, e.g. `10.48550/arXiv.2604.03227`) and `version:` (the arXiv version: `1` for v1, `2` for v2). - **Validate.** `astra validate work/reference/astra.yaml` confirms shape; `--verify-evidence` confirms each `quote.exact` is actually findable in the cached PDF. ## Inputs The skill accepts: 1. An **arXiv ID** (`YYMM.NNNNN` or pre-2007 form like `astro-ph/0607021`) 2. A **DOI** — either an arXiv DOI (`10.48550/arXiv.`) or a journal DOI The slash-command form is `/paper-extraction `. ## What the script does vs what the agent does **Script (`extract-paper-substrate.py`):** walks LaTeX (Path A) or Docling output (Path B) and emits two things: 1. `index.json` — figures (with copied files + line numbers + multi-graphic panels), tables (one `.tex` per block, including AAS `deluxetable`), section outline (with line numbers, in paper-reading order), citation keys (with every file+line they appear on, including biblatex commands, *plus the cited paper's full citation text and resolved DOI*), abstract, title, paths. 2. `astra.yaml` — a stub ASTRA artifact: `id` (derived from arxiv-id/DOI), `version`, `name` (from `\title{}`), `description` (from abstract), empty `inputs:`/`outputs:`/`findings:`. Validates as-is. The script handles a few realities of LaTeX papers automatically: - **Comments are stripped** before regex passes, so commented-out `\includegraphics` / `\cite` / `\section` don't leak into extraction. Newlines are preserved so line numbers stay accurate. - **Multi-file source** (`\input{}` / `\include{}` chains) is read in **paper-reading order** by walking `main.tex`'s input tree, not alphabetical filename order. - **Simple `\newcommand{\name}{body}` macros** are expanded in extracted titles, abstracts, captions, and section names. Macros with arguments (`\newcommand{\foo}[1]{...}`) pass through unexpanded — handling those would require evaluating arbitrary LaTeX. - **Standard table envs** (`table`, `table*`, `deluxetable`, `deluxetable*`) and **standard citation commands** (natbib family + biblatex `\autocite` / `\textcite` / `\parencite` / `\footcite` / `\smartcite`) are all recognized. - **Bibliography parsed in-script.** `.bib` files (preferred — `@type{key, field = value}` entries with brace-protected lastnames recognized) and `.bbl` files (rendered `\bibitem{key}` blocks) are parsed for Path A; the references section at the tail of `document.md` is parsed for Path B (synthesizing `_` keys with letter-suffix disambiguation). DOIs are resolved against Crossref + (optionally) ADS, cached for idempotency, and joined back against `\cite{}`-extracted locations. What the script does *not* do: understand what figures show, identify findings, infer methodology, or handle substrate acquisition (Step 2). It also doesn't expand macros with arguments, resolve `\graphicspath{}` overrides, parse non-LaTeX abstract metadata blocks, or extract citation invocations from rendered prose (Path B `locations:` arrays are empty as a result). **Agent (Steps 4 + 5):** reads `index.json`'s `extraction_warnings` and fixes structural gaps (Step 4), then walks the paper and writes `findings:` into `astra.yaml` with quote-anchored evidence (Step 5). The verbosity of the Insight + Evidence shape *is* the back-pressure: the agent has to find and quote actual paper text, not invent. ## Discipline - **One entry-point.** `/paper-extraction ` is the whole surface. Don't have callers reach into `scripts/` or `references/` directly. The skill orchestrates; consumers trust `index.json`. - **Self-contained.** This skill takes a DOI and produces a standardized directory. It doesn't know who calls … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [LightconeResearch](https://github.com/LightconeResearch) - **Source:** [LightconeResearch/lightcone-cli](https://github.com/LightconeResearch/lightcone-cli) - **License:** BSD-3-Clause - **Homepage:** https://docs.lightconeresearch.org/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-lightconeresearch-lightcone-cli-paper-extraction - Seller: https://agentstack.voostack.com/s/lightconeresearch - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.