Ref Downloader

Ref Downloader — Agent Runbook

> Slim entry for agent mode. The full 8-step manual runbook with code > snippets for Mode A debug + PUBLISHER_MAP extension procedure > lives in [references/agent-runbook.md](references/agent-runbook.md). > Human users see [../../README.md](../../README.md).

` = this folder (skills/ref-downloader in the source repo, or wherever the user copied this skill — e.g. ~/.claude/skills/ref-downloader/). Python scripts live in /scripts/; config files (config.example.toml, config.local.toml) live at /`.

Mode router

This skill handles two flows. Pick before running.

• Mode A — Reference-list download (original use case). User

provides ONE paper (DOI or local PDF) and wants "all of its references". Pipeline: extract_refs.py → validate_refs.py → download_refs.py.

• Mode B — Custom batch download. User provides their own batch

of papers — DOIs, paper titles, non-DOI identifiers (arXiv / PMID / Semantic Scholar IDs), OR an abstract query ("Smith 在 Google Scholar 上的文章" / "Nature Energy 2023 papers"). The agent resolves whatever was given to DOIs, then runs validate_refs.py → download_refs.py directly. Skip the wrapper — run_ref_downloader.py assumes a parent DOI and will fail.

Both modes share install, config, per-publisher strategies, failure modes, output layout, and the CloakBrowser opt-in backend.

Trigger family

| User input shape | Mode | Sub-flow | |---|---|---| | One DOI/PDF + "all refs of" / "全部参考文献" / "把这篇引用都下了" | A | — | | ≥2 DOIs in input (any wrapping: bare / {} / https://doi.org/… / dx.doi.org/…; ASCII or full-width slashes) | B | B.1 (after canonicalize) | | Non-DOI IDs only: arXiv: / PMID: / S2: / corpusId: | B | B.0 normalize → B.1 | | Title list ("下载这几篇：title1, title2, …") | B | B.2 | | Abstract query (author / topic / journal+year / "Google Scholar 上 …") | B | B.3 | | Mixed (DOIs + titles + IDs + queries) | B | run each, merge | | Single DOI without "of refs" qualifier | B | B.1 single-item | | Title + author + year for ONE paper ("Smith 2024 Nature paper on X") | B | B.2 (specific paper, lookup) | | Open-ended query for a corpus ("Smith 2024 之后所有的 Nature 文章") | B | B.3 (discovery) | | Insufficient resolvable content ("上次给你的那 5 篇" / pure pronouns) | — | Ask user to repaste / attach file; do NOT guess | | Genuinely ambiguous A vs B | — | ask user |

Key disambiguators:

• "refs OF a paper" (Mode A) vs "these papers themselves" (Mode B).
• Identifies a specific paper (B.2) vs describes a set to discover

(B.3). If user names a specific paper with enough metadata to uniquely identify it (title + author + year), it's B.2 (lookup exact); if they describe a class of papers ("all Smith's 2024 Nature papers"), it's B.3 (discover then filter).

Mode A — Reference-list download

When to invoke

Trigger phrases:

• "帮我下载 [paper / DOI] 的参考文献" / "批量下载引用文献" / "把这篇论文的所有引用下载下来"
• "Download all refs of [paper / DOI]" / "Batch-download every reference"
• User provides a DOI (10.x/y form) or local PDF path and asks for

"all references" / "全部参考文献"

Don't invoke for:

• Downloading one arbitrary PDF (not a reference list) → user wants

one paper itself, route to Mode B as a single-item B.1

• Generic web scraping
• Paper search / Zotero import — different tools

Primary entry

python "/scripts/run_ref_downloader.py" 

The wrapper handles DOI resolution (Zotero → fitz fallback), output-dir layout, sequential 3-stage pipeline (extract_refs.py → validate_refs.py → download_refs.py), and end-of-run cleanup.

Useful flags:

• --yes — non-interactive (CI/batch), overwrite prompts default-yes
• --auto — forwarded to download_refs.py: skip "press Enter"

confirm + shorter challenge wait + async retry queue for manual_pending refs (60s delay, single retry, max 3 concurrent). Use for CI / overnight runs; not for sessions where you want to drive captchas yourself.

• --fail-fast — terminate after first actionable unresolved ref

(useful in CI to surface real failures fast)

• --output-dir — override default output location
• --config — alternate TOML config (overrides

config.local.toml)

Pre-flight checklist (confirm before running)

1. DOI correct? Echo back to user:

即将下载参考文献：DOI=

1. Edge fully closed? All msedge.exe processes killed (Task

Manager check). The script claims the user's persistent Edge profile and needs exclusive access. (Cloak backend skips this.)

1. Config set? First-run users need

/config.local.toml with [crossref].mailto. Missing config → wrapper prints a WARNING but continues with placeholder defaults.

1. Output location agreed? Default for DOI input:

/_refs/. For PDF input: /_refs/. Override with --output-dir.

Mode B — Custom batch download

Input variability is the point. Don't refuse — route. The agent handles whatever shape the user gave (paste, file, prose, BibTeX, RIS, abstract query) and resolves it to a clean DOI list before handing off to the pipeline.

Step 0 — Canonicalize input (always runs first)

Before any extraction or routing, normalize the input string:

1. Unwrap delimiters around DOIs: strip leading/trailing

{}, <>, (), [], and quote marks.

1. Strip URL prefixes: https://doi.org/, http://doi.org/,

https://dx.doi.org/, http://dx.doi.org/ → bare DOI.

1. Full-width / Unicode punctuation to ASCII:

• ／ (U+FF0F) → /
• ： (U+FF1A) → :
• ． (U+FF0E) → .
• Smart quotes "" '' → straight " '

1. Trim trailing punctuation: .,;)}"' (note }).

Apply Step 0 BEFORE the regex pass in B.1 AND before the canonical dedupe compare in step 5 of the main flow.

B.0 — Normalize non-DOI identifiers

For each non-DOI identifier the user gave:

| Input | Action | |---|---| | arXiv:2401.12345 or bare arXiv ID | Use 10.48550/arXiv. (canonical), but prefer a journal DOI if the agent can discover one via Crossref query.bibliographic= | | PMID:12345678 or pubmed.ncbi.nlm.nih.gov/12345678 | Hit eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id= → grab articleids[type=doi] | | Semantic Scholar paper ID (S2:abc... / corpusId:N) | Hit api.semanticscholar.org/graph/v1/paper/?fields=externalIds → grab externalIds.DOI | | Anything else non-DOI shaped | Leave for B.1 regex pass to ignore; if it survives B.1+B.2 unresolved, drop with skipped (unresolvable_identifier) in the confirm table — do NOT auto-fire requests on garbage |

Network preflight: before firing B.0 lookups, do one cheap sanity probe (e.g. HEAD https://api.crossref.org/). If it fails, tell the user "no network — Mode B can't resolve non-DOI identifiers or do discovery; only direct DOI extraction will work" and let them decide to proceed with B.1 only.

Semantic Scholar rate limit: unauthenticated ≤ 1 req/sec. Pace batches; on 429 back off 30s then retry once; on second 429, drop the entry with skipped (ss_rate_limited).

B.1 — DOI extraction

After Step 0 canonicalization:

1. Regex 10\.\d{4,9}/[^\s,;<>"'{}]+ on the canonicalized input

(or file contents). The character class explicitly excludes {} so wrappers like BibTeX doi = {10.x/y} don't leak braces.

1. Strip trailing punctuation .,;)}"'.
2. Dedupe via canonical lowercase compare (see flow step 5).
3. Fallback rule: if regex yields DOIs for less than 50% of the

apparent entries (e.g. BibTeX with 20 @article entries but only 5 have ASCII DOIs), send the remaining entries to B.2 title lookup rather than silently dropping them.

B.2 — Title → DOI lookup

For each title:

1. Query Crossref:

GET https://api.crossref.org/works?query.title=&rows=5&mailto=

1. Pick top by score. Note: Crossref score is unbounded

relevance, NOT 0–100 — absolute thresholds across queries don't compare. Use relative + content rules:

• If `top1.score / top2.score / ...)

`` Default: download confidence=high rows only. confidence=low` excluded unless user explicitly includes. Unresolvables dropped.

1. Propose project name (context-aware: "组会文献" →

groupmtg_; "Smith 综述补充" → smith_review_extras; nothing topical → custom_). Ask user confirm.

1. Append vs new — if //refs_raw.json

exists, ask append / new / rename (default: ask again on any other input — DO NOT default-append). Append rules:

• Canonicalize new DOIs (Step 0) BEFORE compare.
• Drop new DOIs already present in existing entries

(case-insensitive canonical-form compare).

• New ids start at id = max(existing_ids) + 1.
• Never renumber existing entries — validate_refs.py keys

its incremental skip on id, renumbering re-assigns prior verified metadata to the wrong DOI. Only verified rows are skipped; failed/pending rows revalidate on re-run.

1. Build refs_raw.json (heredoc the agent runs):

``python import json from datetime import datetime dois = [...] # finalized canonical-lowercase list start_id = 1 # or max(existing_ids)+1 in append mode data = { "parent_doi": "", # empty string for clean report labels; # validate_refs.py reads as raw JSON, null # would also work but "" is preferred. "parent_title": f"Custom batch — {user_label}", "extracted_at": datetime.now().isoformat(timespec="seconds"), "total": len(dois), "with_doi": len(dois), "without_doi": 0, "references": [ {"id": i, "doi": d, "key": "", "unstructured": "", "author": "", "year": "", "journal": "", "volume": "", "first_page": ""} for i, d in enumerate(dois, start=start_id) ], } with open(f"{project_name}/refs_raw.json", "w", encoding="utf-8") as f: json.dump(data, f, ensure_ascii=False, indent=2) ` All metadata fields stay empty — validate_refs.py fills them from Crossref per DOI on success. Rows whose DOI Crossref can't resolve become status=failed` with empty metadata (not partially enriched).

1. Run pipeline (skip the wrapper):

``bash cd python /scripts/validate_refs.py python /scripts/download_refs.py [--auto] [--fail-fast] ``

Mode B pre-flight checklist

1. Full confirm table approved? User saw EVERY row, not just top 5.
2. Total count + estimated runtime? Roughly: 0.35s × N for

validate, ~30-90s per ref for download (publisher-dependent).

1. Project name + new/append decided? First-write = new project;

second-write = append OR rename.

1. Edge fully closed? (Edge backend only — cloak skips.)
2. Config set? [crossref].mailto for polite-pool latency.
3. Network reachable? (api.crossref.org HEAD probe).

Compatibility with v0.4 flags

• --auto: works with Mode B (manual_pending refs go to async

retry queue same as Mode A).

• --fail-fast: works with Mode B (stops on first actionable

unresolved ref).

• [user].verified_no_si_dois: works with Mode B (matches by

lowercase DOI — independent of how refs_raw.json was produced).

• CloakBrowser backend (REF_DOWNLOADER_BROWSER=cloak): works

with Mode B (browser backend is decided by env var, independent of input mode).

Install prerequisites (before first invocation)

The skill protocol can't manage Python deps. If python -c "import playwright" fails, the user needs:

cd ""
pip install playwright pymupdf
playwright install msedge          # downloads Edge driver
cp config.example.toml config.local.toml   # then user edits [crossref].mailto

If the user is developing from the source repo instead of an installed skill copy, they can also install from the repo root with pip install -r requirements.txt -r requirements-dev.txt.

Alternative backend: CloakBrowser (Cloudflare-heavy sites)

Default backend is Microsoft Edge. For sites that keep blocking ordinary Playwright (Cloudflare Turnstile, Radware, persistent Just a moment / 安全验证 pages), switch to the CloakBrowser stealth Chromium backend.

What CloakBrowser is. Third-party MIT-licensed Python package by CloakHQ (github.com/CloakHQ/CloakBrowser, pypi:cloakbrowser). Ships a patched Chromium build with anti-fingerprint changes. Its launch_persistent_context_async() is Playwright-API-compatible, which is why ref-downloader can swap it in with one env var. NOT a dependency of ref-downloader — if the user doesn't pip install cloakbrowser, it's never imported and the default Edge path runs as normal. Beta software; user installs it at their own discretion.

# One-time setup (separate from ref-downloader's `pip install playwright pymupdf`)
pip install cloakbrowser

# Switch backend (env vars; no CLI flag changes)
$env:REF_DOWNLOADER_BROWSER = "cloak"
$env:REF_DOWNLOADER_CLOAK_HUMAN_PRESET = "careful"   # optional: slower mouse/scroll
# Optional overrides:
# $env:REF_DOWNLOADER_CLOAK_PROFILE = ""  # default: ~/.local/cloakbrowser/profiles/ref-downloader
# $env:REF_DOWNLOADER_CLOAK_PROXY = "http://..."
# $env:REF_DOWNLOADER_CLOAK_GEOIP = "1"
# $env:CLOAKBROWSER_PYTHONPATH = ""      # sys.path hint if cloakbrowser is checked out, not pip-installed

python "/scripts/download_refs.py" 

Caveats:

• cloakbrowser uses its own Chromium with a separate profile —

Edge does NOT need to be closed.

• The separate profile means your institutional cookies are NOT

carried — best for open-Cloudflare sites, less useful for paywalled refs your institution licenses.

• A fresh cloak profile may still show Cloudflare/security-verification

pages on first visit; warm it manually by opening the target site once with the same REF_DOWNLOADER_CLOAK_PROFILE and finishing any verification before running the downloader.

• human_preset=careful lowers behavior-detection trigger rates but

is not a captcha solver.

Output layout

Same for both modes:

/
├── /
│   ├── refs_raw.json           # extract_refs.py output (Mode A) or
│   │                           # hand-built JSON (Mode B)
│   ├── refs_validated.json     # validate_refs.py output
│   ├── download_report.csv     # per-ref status (only on graceful
│   │                           # completion; OVERWRITTEN each run —
│   │                           # NOT historical truth)
│   ├── *.pdf                   # reference PDFs
│   └── *_SI.pdf                # supplementary files (where supported)
└── runs/-round-03/
    └── events.jsonl            # full event trace per ref
                                # (append-only across runs;
                                #  THIS is the authoritative history)

Interruption note: if the run is interrupted (Ctrl+C / Edge crash / VPN drop), the root download_report.csv may be stale. Trust the latest runs//events.jsonl + actual files in /.

Common failure modes

| Status / symptom | Meaning | Action | |---|---|---| | manual_pending (auth_redirect) | Bounced to institution SSO | User signs in via live Edge tab; re-run (incremental skips done refs) | | manual_pending (challenge_timeout) | Cloudflare / publisher challenge unsolved in time | Re-run interactively; solve captcha when prompted | | manual_pending (elsevier_crasolve_shell) | Elsevier viewer stuck in transition | In --auto mode the async retry queue picks it up ~60s later; in interactive mode the hot-session retry usually catches it, else manual click in live page | | failed (auto) | Generic auto path failed | Check events.jsonl for that ref; may need a publisher-specific patch | | ignored (ignored_institution_access) | DOI listed in [institution].ignored_access_dois | Skip-by-design; remove from config to retry | | Edge won't launch | Background msedge.exe still holding profile | Kill all

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ltczding-gif
• Source: ltczding-gif/ref-downloader
• License: MIT

Install and usage instructions live in the source repository linked above.