# Paper Fetch

> Use whenever the user wants to obtain, download, or fetch a paper's PDF — given a DOI, an arXiv id, a paper title, a citation, or a list of DOIs. Trigger on phrases like "download this paper", "find the PDF for [DOI]", "grab me the [Nature/bioRxiv/arXiv] paper on X", "get the open-access version", "I need this article", or any bulk/batch paper download request, even when the user doesn't explicit…

- **Type:** Skill
- **Install:** `agentstack add skill-agents365-ai-paper-fetch-paper-fetch`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Agents365-ai](https://agentstack.voostack.com/s/agents365-ai)
- **Installs:** 0
- **Category:** [Content & Media](https://agentstack.voostack.com/c/content-and-media)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Agents365-ai](https://github.com/Agents365-ai)
- **Source:** https://github.com/Agents365-ai/paper-fetch/tree/main/skills/paper-fetch
- **Website:** https://agents365-ai.github.io/paper-fetch/

## Install

```sh
agentstack add skill-agents365-ai-paper-fetch-paper-fetch
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# paper-fetch

Fetch the PDF for a paper given a DOI (or title). Tries multiple sources in priority order and stops at the first hit.

## Resolution order

1. **Unpaywall** — `https://api.unpaywall.org/v2/{doi}?email=$UNPAYWALL_EMAIL`, read `best_oa_location.url_for_pdf` (skipped if `UNPAYWALL_EMAIL` not set)
2. **Semantic Scholar** — `https://api.semanticscholar.org/graph/v1/paper/DOI:{doi}?fields=openAccessPdf,externalIds`
3. **arXiv** — if `externalIds.ArXiv` present, `https://arxiv.org/pdf/{arxiv_id}.pdf`
4. **PubMed Central OA** — if PMCID present, `https://www.ncbi.nlm.nih.gov/pmc/articles/{pmcid}/pdf/`
5. **bioRxiv / medRxiv** — if DOI prefix is `10.1101`, query `https://api.biorxiv.org/details/{server}/{doi}` for the latest version PDF URL
6. **Publisher direct** *(institutional mode only — `PAPER_FETCH_INSTITUTIONAL=1`)* — DOI-prefix → publisher PDF template (Nature / Science / Wiley / Springer / ACS / PNAS / NEJM / Sage / T&F / Elsevier). The caller's own subscription IP / cookies / EZproxy are what authorize the fetch; unauthorized responses fail the `%PDF` check and fall through to step 7.
7. **Sci-Hub mirrors** *(on by default; disable with `PAPER_FETCH_NO_SCIHUB=1`)* — last-resort fallback. Tries the mirror list in `PAPER_FETCH_SCIHUB_MIRRORS` (or built-in defaults `sci-hub.ru`, `sci-hub.st`, `sci-hub.su`, `sci-hub.box`, `sci-hub.red`, `sci-hub.al`, `sci-hub.mk`, `sci-hub.ee`) in order; on full miss, scrapes `https://www.sci-hub.pub/` once per process for fresh mirrors. CAPTCHA / missing-paper pages have no PDF iframe and fall through silently.
8. Otherwise → report failure with title/authors so the user can request via ILL

**CloakBrowser fallback (download layer, opt-in — `PAPER_FETCH_CLOAK=1`).** This is not a separate source: it sits at the download chokepoint, so it applies to *any* of the sources above. When a resolved PDF URL is blocked by Cloudflare — HTTP 403/429, or a "Just a moment…" HTML interstitial served in place of the file — and the operator opted in, the URL is retried through [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) (a stealth Chromium that passes the JS challenge) via the `cloak_pdf.py` companion. Bytes it returns are re-validated through the same `%PDF` magic-byte + 50 MB checks; on success the result carries `via: "cloak"`. Off by default, fails closed (missing CloakBrowser → silent fall-through), and the agent cannot opt in — see *CloakBrowser access* below.

If only a title is given, pass it directly via `--title ""`. Resolution chain:

1. **Crossref** `query.title` — primary; covers all major journal/conference DOIs
2. **Semantic Scholar `/paper/search/match`** — fallback when Crossref's top match is low-confidence (`match_score ` is synthesized so the download chain stays uniform.
3. **Crossref's best guess (low-confidence)** — used only when both resolvers struggled. The result envelope sets `meta.title_resolution.low_confidence: true` plus a `low_confidence_reason` (`score_below_threshold` / `ambiguous_runner_up`) so an agent can either bail or confirm via `--dry-run`.

Either way the resolved DOI, the winning resolver, the full `resolvers_tried` list, and the top candidate matches are all surfaced under `meta.title_resolution`.

**If `semanticscholar-skill` is registered**, it can serve as a richer pre-step for title → DOI resolution — useful when you also need relevance ranking, snippet search, or citation context, not just a DOI. The agent writes a Python script using the skill's `match_title()` to read `externalIds.DOI`, then runs `paper-fetch `. When the result has only an `ArXiv` id (no DOI), synthesize `10.48550/arXiv.` and pass that to paper-fetch.

When only the DOI is needed, `--title` is the single-command path — paper-fetch's built-in Crossref → S2 chain handles most cases.

## Usage

```bash
python scripts/fetch.py  [options]
python scripts/fetch.py --title "" [options]
python scripts/fetch.py --batch  [options]
python scripts/fetch.py schema           # machine-readable self-description
```

### Flags

The flags below are the ones an agent composes in normal use. For the complete contract — including `--dry-run`, `--pretty`, `--stream`, `--overwrite`, `--timeout`, `--version`, plus parameter types and exit-code mappings — run `python scripts/fetch.py schema` (machine-readable, drift-checked via `schema_version`).

| Flag | Default | Description |
|------|---------|-------------|
| `doi` | — | DOI to fetch (positional). Use `-` to read a single DOI from stdin |
| `--title TITLE` | — | Paper title; resolved to a DOI via Crossref before download. Mutually exclusive with positional DOI / `--batch` |
| `--batch FILE` | — | File with one DOI per line for bulk download. Use `-` to read from stdin |
| `--out DIR` | `pdfs` | Output directory |
| `--format` | auto | `json` for agents, `text` for humans. Auto-detects: `json` when stdout is not a TTY, `text` when it is |
| `--idempotency-key KEY` | — | Safe-retry key. Re-running with the same key replays the original envelope from `/.paper-fetch-idem/` without network I/O |

### Agent discovery: `schema` subcommand

```bash
python scripts/fetch.py schema
```

Emits a complete machine-readable description of the CLI on stdout (no network). Includes `cli_version`, `schema_version`, parameter types, exit codes, error codes, envelope shapes, and environment variables. Agents should read this once, cache it against `schema_version`, and re-read when the cached version drifts.

### Output contract

**stdout** emits a single JSON envelope. Every envelope carries a `meta` slot.

**Success** (all DOIs resolved):

```json
{
  "ok": true,
  "data": {
    "results": [
      {
        "doi": "10.1038/s41586-021-03819-2",
        "success": true,
        "source": "unpaywall",
        "pdf_url": "https://www.nature.com/articles/s41586-021-03819-2.pdf",
        "file": "pdfs/Jumper_2021_Highly_accurate_protein_structure_predic.pdf",
        "meta": {"title": "Highly accurate protein structure prediction with AlphaFold", "year": 2021, "author": "Jumper"},
        "sources_tried": ["unpaywall"]
      }
    ],
    "summary": {"total": 1, "succeeded": 1, "failed": 0},
    "next": []
  },
  "meta": {
    "request_id": "req_a908f5156fc1",
    "latency_ms": 2036,
    "schema_version": "1.9.0",
    "cli_version": "0.13.1",
    "sources_tried": ["unpaywall"]
  }
}
```

**Partial** (batch mode — some DOIs failed, exit code reflects the failure class):

```json
{
  "ok": "partial",
  "data": {
    "results": [
      { "doi": "10.1038/s41586-021-03819-2", "success": true, "source": "unpaywall", ... },
      {
        "doi": "10.1234/nonexistent",
        "success": false,
        "source": null,
        "pdf_url": null,
        "file": null,
        "meta": {},
        "sources_tried": ["unpaywall", "semantic_scholar"],
        "error": {
          "code": "not_found",
          "message": "No open-access PDF found",
          "retryable": true,
          "retry_after_hours": 168,
          "reason": "OA availability changes over time; retry after embargo lifts or preprint appears"
        }
      }
    ],
    "summary": {"total": 2, "succeeded": 1, "failed": 1},
    "next": ["paper-fetch 10.1234/nonexistent --out pdfs"]
  },
  "meta": { ... }
}
```

The `next` slot is an array of suggested follow-up commands: re-invoking them retries only the failed subset. Combine with `--idempotency-key` to make the whole batch safely retriable without re-downloading the already-succeeded items.

**Failure** (bad arguments, exit code 3):

```json
{
  "ok": false,
  "error": {
    "code": "validation_error",
    "message": "Provide a DOI or --batch file",
    "retryable": false
  },
  "meta": { ... }
}
```

**Per-item skipped** (destination already exists, no `--overwrite`):

```json
{
  "doi": "10.1038/s41586-021-03819-2",
  "success": true,
  "source": "unpaywall",
  "pdf_url": "https://...",
  "file": "pdfs/Jumper_2021_...pdf",
  "skipped": true,
  "skip_reason": "file_exists",
  "sources_tried": ["unpaywall"]
}
```

**Idempotency replay** (re-run with the same `--idempotency-key`):

The cached envelope is returned verbatim, but `meta.request_id` and `meta.latency_ms` are re-stamped for the current call, and `meta.replayed_from_idempotency_key` is set. No network I/O occurs.

### Stderr progress (NDJSON)

When `--format json`, stderr emits one JSON object per line for liveness:

```
{"event": "session",     "request_id": "req_...", "elapsed_ms": 0,    "cli_version": "0.13.1", "schema_version": "1.9.0"}
{"event": "start",       "request_id": "req_...", "elapsed_ms": 2,    "doi": "10.1038/..."}
{"event": "source_try",  "request_id": "req_...", "elapsed_ms": 2,    "doi": "...", "source": "unpaywall"}
{"event": "source_hit",  "request_id": "req_...", "elapsed_ms": 2036, "doi": "...", "source": "unpaywall", "pdf_url": "..."}
{"event": "download_ok", "request_id": "req_...", "elapsed_ms": 4120, "doi": "...", "file": "..."}
```

Event types: `session`, `start`, `source_try`, `source_hit`, `source_miss`, `source_skip`, `source_enrich`, `source_enrich_failed`, `download_ok`, `download_error`, `download_skip`, `dry_run`, `not_found`, `resolve_error`. All events share `request_id` and `elapsed_ms`, letting an orchestrator correlate progress across stderr and the final stdout envelope. The `session` event fires once per invocation, before any DOI work or network I/O, and carries `cli_version` / `schema_version` so agents can detect schema drift against a cached copy without waiting for the final envelope.

`source_enrich` fires when Semantic Scholar is called purely to backfill missing `author` / `title` after another source already provided the PDF URL; its `fields` array lists exactly which fields were filled in. `source_enrich_failed` fires when that enrichment call fails — the Unpaywall PDF URL is still used and the filename falls back to `unknown__…`.

When `--format text`, stderr emits human-readable prose.

### Exit codes

| Code | Meaning | Retryable class |
|------|---------|-----------------|
| `0` | All DOIs resolved / previewed | — |
| `1` | Unresolved — one or more DOIs had no OA copy; no transport failure | Not now (retry after `retry_after_hours`) |
| `2` | Reserved for auth errors (currently unused) | — |
| `3` | Validation error (bad arguments, missing input) | No |
| `4` | Transport error (network / download / IO failure) | Yes |

The taxonomy lets an orchestrator route failures deterministically: exit 4 is worth retrying immediately, exit 1 is not, exit 3 is a bug in the caller.

### Error codes in JSON

Every retryable error carries a `retry_after_hours` hint in the error object, so an orchestrator can schedule retries without guessing.

| Code | Meaning | Retryable | `retry_after_hours` |
|------|---------|-----------|---------------------|
| `validation_error` | Bad arguments or empty input | No | — |
| `title_resolve_failed` | Crossref returned no items for the given `--title` query (try a longer / cleaner title, or pass the DOI directly) | No | — |
| `not_found` | No open-access PDF found | Yes | `168` (one week — OA lands on embargo / preprint timescale) |
| `resolve_network_error` | Metadata resolvers failed with transport errors (timeout / 5xx / 403); OA availability is unknown — retry rather than treating as `not_found`. Maps to exit `4`. | Yes | `1` |
| `download_network_error` | Network failure during download | Yes | `1` |
| `download_not_a_pdf` | Response was not a PDF (HTML landing page) | No | — |
| `download_host_not_allowed` | PDF URL failed SSRF safety check (private IP / non-http(s) / non-80,443 / blocked metadata host / hostname resolving into private space / unsafe redirect hop) | No | — |
| `download_size_exceeded` | Response exceeded 50 MB limit | Yes | `24` |
| `download_io_error` | Local filesystem write failed | Yes | `1` |
| `internal_error` | Unexpected error | No | — |

The canonical mapping lives in `RETRY_AFTER_HOURS` in `scripts/fetch.py` and is surfaced in `schema.error_codes`.

### Examples

```bash
# Single DOI (JSON output when piped; text when in a terminal)
python scripts/fetch.py 10.1038/s41586-020-2649-2

# Single title (resolved to DOI via Crossref, then downloaded)
python scripts/fetch.py --title "Highly accurate protein structure prediction with AlphaFold"

# Dry-run preview (resolve without downloading)
python scripts/fetch.py 10.1038/s41586-020-2649-2 --dry-run

# Title + dry-run — preview the resolved DOI and candidate matches
python scripts/fetch.py --title "Attention Is All You Need" --dry-run

# Force JSON (for agents even inside a terminal)
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format json

# Human-readable with pretty colors in a pipeline
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format text

# Batch download, safely retriable
python scripts/fetch.py --batch dois.txt --out ./papers \
    --idempotency-key monday-review-batch

# Pipe DOIs from another tool
zot -F ids.json query ... | jq -r '.[].doi' | python scripts/fetch.py --batch -

# Agent discovery
python scripts/fetch.py schema --pretty

# Streaming mode — one result per line as each DOI resolves
python scripts/fetch.py --batch dois.txt --stream

# Works without UNPAYWALL_EMAIL (skips Unpaywall, uses remaining 4 sources)
python scripts/fetch.py 10.1038/s41586-020-2649-2
```

## Environment

| Variable | Default | Purpose |
|---|---|---|
| `UNPAYWALL_EMAIL` | unset | Contact email for Unpaywall API. Optional but recommended. Without it, Unpaywall is skipped (remaining sources still work). |
| `PAPER_FETCH_INSTITUTIONAL` | unset | Set to any value (e.g. `1`) to opt into **institutional mode** — activates a 1 req/s rate limiter and the publisher-direct fallback. See below. |
| `PAPER_FETCH_NO_SCIHUB` | unset | Set to any value to disable the Sci-Hub fallback (step 7). |
| `PAPER_FETCH_SCIHUB_MIRRORS` | unset | Comma-separated mirror hostnames to try in priority order (e.g. `sci-hub.ru,sci-hub.st,sci-hub.su`). Overrides built-in defaults. |
| `PAPER_FETCH_CLOAK` | unset | Set to any value to enable the **CloakBrowser fallback** — Cloudflare-blocked PDFs (HTTP 403/429 or a non-PDF interstitial) are retried through a stealth Chromium. See *CloakBrowser access* below. |
| `CLOAKBROWSER_PYTHON` | auto | Path to a Python that can `import cloakbrowser`, used by the cloak fallback. Auto-detect order: this var → `~/github/CloakBrowser/.venv/bin/python` → the current interpreter. |
| `PAPER_FETCH_CLOAK_HEADED` | unset | Set to any value to launch a **headed** (visible) browser instead of headless. Harder Cloudflare challenges (e.g. `science.org`) defeat headless mode and only clear in a real window — set this when the cloak fallback keeps returning HTTP 403 / "Just a moment…". Requires a display. |

## CloakBrowser access (opt-in)

Some publishers (e.g. `science.org`) sit behind Cloudflare, which answers a plain HTTP client with a `403`/`429` or a "Just a moment…" JS-challenge page instead of the PDF — so the default `urllib` download can't get through even when the URL is legitimately accessible from a browser. [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) is a stealth Chromium that passes those challenges. This skill borrows the approach from [cloakFetch](https://github.com/Agents365-ai/cloakFetch).

**Opt in:** `export PAPER_FETCH_CLOAK=1` (plus a `cloakbrowser`-importable Python — see `CLOAKBROWSER_PYTHON`).

**How it works:** the fallback lives at the **download layer**, not as a new source — so it applies to any resolved URL (Unpaywall, publisher-direct, Sci-Hub, …). On a Cloudflare block, `fetch.py` shells out to the `cloak_pdf.py` companion via the resolved Python; CloakBrowser loads the PDF host's origin to solve the JS challenge, then fetches the PDF with an **in-page `fetch()`** (so the request carries the browser's real fingerprint and `cf_clearance` cookie) and returns the bytes on stdout. `fetch.py` itself stays stdlib-only — it never imports `cloakbrowser`.

**Headless vs. h

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Agents365-ai](https://github.com/Agents365-ai)
- **Source:** [Agents365-ai/paper-fetch](https://github.com/Agents365-ai/paper-fetch)
- **License:** MIT
- **Homepage:** https://agents365-ai.github.io/paper-fetch/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-agents365-ai-paper-fetch-paper-fetch
- Seller: https://agentstack.voostack.com/s/agents365-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.