# Ref Downloader

> >

- **Type:** Skill
- **Install:** `agentstack add skill-ltczding-gif-ref-downloader-ref-downloader`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ltczding-gif](https://agentstack.voostack.com/s/ltczding-gif)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ltczding-gif](https://github.com/ltczding-gif)
- **Source:** https://github.com/ltczding-gif/ref-downloader/tree/main/skills/ref-downloader

## Install

```sh
agentstack add skill-ltczding-gif-ref-downloader-ref-downloader
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Ref Downloader — Agent Runbook

> Slim entry for agent mode. The full 8-step manual runbook with code
> snippets for Mode A debug + `PUBLISHER_MAP` extension procedure
> lives in [references/agent-runbook.md](references/agent-runbook.md).
> Human users see [../../README.md](../../README.md).

`` = this folder (`skills/ref-downloader` in the source repo,
or wherever the user copied this skill — e.g.
`~/.claude/skills/ref-downloader/`). Python scripts live in
`/scripts/`; config files (`config.example.toml`,
`config.local.toml`) live at `/`.

## Mode router

This skill handles two flows. **Pick before running.**

- **Mode A — Reference-list download** (original use case). User
  provides ONE paper (DOI or local PDF) and wants "all of its
  references". Pipeline: `extract_refs.py` → `validate_refs.py` →
  `download_refs.py`.

- **Mode B — Custom batch download**. User provides their own batch
  of papers — DOIs, paper titles, non-DOI identifiers (arXiv / PMID
  / Semantic Scholar IDs), OR an abstract query ("Smith 在 Google
  Scholar 上的文章" / "Nature Energy 2023 papers"). The agent
  resolves whatever was given to DOIs, then runs `validate_refs.py`
  → `download_refs.py` directly. **Skip the wrapper** —
  `run_ref_downloader.py` assumes a parent DOI and will fail.

Both modes share install, config, per-publisher strategies, failure
modes, output layout, and the CloakBrowser opt-in backend.

### Trigger family

| User input shape | Mode | Sub-flow |
|---|---|---|
| One DOI/PDF + "all refs of" / "全部参考文献" / "把这篇引用都下了" | A | — |
| ≥2 DOIs in input (any wrapping: bare / `{}` / `https://doi.org/…` / `dx.doi.org/…`; ASCII or full-width slashes) | B | B.1 (after canonicalize) |
| Non-DOI IDs only: `arXiv:` / `PMID:` / `S2:` / `corpusId:` | B | B.0 normalize → B.1 |
| Title list ("下载这几篇：title1, title2, …") | B | B.2 |
| Abstract query (author / topic / journal+year / "Google Scholar 上 …") | B | B.3 |
| Mixed (DOIs + titles + IDs + queries) | B | run each, merge |
| Single DOI without "of refs" qualifier | B | B.1 single-item |
| Title + author + year for ONE paper ("Smith 2024 Nature paper on X") | B | **B.2** (specific paper, lookup) |
| Open-ended query for a corpus ("Smith 2024 之后所有的 Nature 文章") | B | **B.3** (discovery) |
| **Insufficient resolvable content** ("上次给你的那 5 篇" / pure pronouns) | — | **Ask user to repaste / attach file**; do NOT guess |
| Genuinely ambiguous A vs B | — | ask user |

**Key disambiguators**:

- *"refs OF a paper"* (Mode A) vs *"these papers themselves"* (Mode B).
- *Identifies a specific paper* (B.2) vs *describes a set to discover*
  (B.3). If user names a specific paper with enough metadata to
  uniquely identify it (title + author + year), it's B.2 (lookup
  exact); if they describe a class of papers ("all Smith's 2024
  Nature papers"), it's B.3 (discover then filter).

## Mode A — Reference-list download

### When to invoke

**Trigger phrases**:
- "帮我下载 [paper / DOI] 的参考文献" / "批量下载引用文献" / "把这篇论文的所有引用下载下来"
- "Download all refs of [paper / DOI]" / "Batch-download every reference"
- User provides a DOI (`10.x/y` form) or local PDF path and asks for
  "all references" / "全部参考文献"

**Don't invoke for**:
- Downloading one arbitrary PDF (not a reference list) → user wants
  one paper itself, route to Mode B as a single-item B.1
- Generic web scraping
- Paper search / Zotero import — different tools

### Primary entry

```bash
python "/scripts/run_ref_downloader.py" 
```

The wrapper handles DOI resolution (Zotero → fitz fallback),
output-dir layout, sequential 3-stage pipeline (`extract_refs.py` →
`validate_refs.py` → `download_refs.py`), and end-of-run cleanup.

**Useful flags**:
- `--yes` — non-interactive (CI/batch), overwrite prompts default-yes
- `--auto` — forwarded to `download_refs.py`: skip "press Enter"
  confirm + shorter challenge wait + async retry queue for
  `manual_pending` refs (60s delay, single retry, max 3 concurrent).
  Use for CI / overnight runs; not for sessions where you want to
  drive captchas yourself.
- `--fail-fast` — terminate after first actionable unresolved ref
  (useful in CI to surface real failures fast)
- `--output-dir ` — override default output location
- `--config ` — alternate TOML config (overrides
  `config.local.toml`)

### Pre-flight checklist (confirm before running)

1. **DOI correct?** Echo back to user:
   `即将下载参考文献：DOI=`
2. **Edge fully closed?** All `msedge.exe` processes killed (Task
   Manager check). The script claims the user's persistent Edge
   profile and needs exclusive access. (Cloak backend skips this.)
3. **Config set?** First-run users need
   `/config.local.toml` with `[crossref].mailto`. Missing
   config → wrapper prints a WARNING but continues with placeholder
   defaults.
4. **Output location agreed?** Default for DOI input:
   `/_refs/`. For PDF input:
   `/_refs/`. Override with `--output-dir`.

## Mode B — Custom batch download

Input variability is the point. Don't refuse — route. The agent
handles whatever shape the user gave (paste, file, prose, BibTeX,
RIS, abstract query) and resolves it to a clean DOI list before
handing off to the pipeline.

### Step 0 — Canonicalize input (always runs first)

Before any extraction or routing, normalize the input string:

1. **Unwrap delimiters around DOIs**: strip leading/trailing
   `{}`, `<>`, `()`, `[]`, and quote marks.
2. **Strip URL prefixes**: `https://doi.org/`, `http://doi.org/`,
   `https://dx.doi.org/`, `http://dx.doi.org/` → bare DOI.
3. **Full-width / Unicode punctuation to ASCII**:
   - `／` (U+FF0F) → `/`
   - `：` (U+FF1A) → `:`
   - `．` (U+FF0E) → `.`
   - Smart quotes `""` `''` → straight `"` `'`
4. **Trim trailing punctuation**: `.,;)}"'` (note `}`).

Apply Step 0 BEFORE the regex pass in B.1 AND before the canonical
dedupe compare in step 5 of the main flow.

### B.0 — Normalize non-DOI identifiers

For each non-DOI identifier the user gave:

| Input | Action |
|---|---|
| `arXiv:2401.12345` or bare arXiv ID | Use `10.48550/arXiv.` (canonical), but prefer a journal DOI if the agent can discover one via Crossref `query.bibliographic=` |
| `PMID:12345678` or pubmed.ncbi.nlm.nih.gov/12345678 | Hit `eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=` → grab `articleids[type=doi]` |
| Semantic Scholar paper ID (`S2:abc...` / `corpusId:N`) | Hit `api.semanticscholar.org/graph/v1/paper/?fields=externalIds` → grab `externalIds.DOI` |
| Anything else non-DOI shaped | Leave for B.1 regex pass to ignore; if it survives B.1+B.2 unresolved, drop with `skipped (unresolvable_identifier)` in the confirm table — do NOT auto-fire requests on garbage |

**Network preflight**: before firing B.0 lookups, do one cheap
sanity probe (e.g. `HEAD https://api.crossref.org/`). If it fails,
tell the user "no network — Mode B can't resolve non-DOI identifiers
or do discovery; only direct DOI extraction will work" and let them
decide to proceed with B.1 only.

**Semantic Scholar rate limit**: unauthenticated ≤ 1 req/sec. Pace
batches; on `429` back off 30s then retry once; on second 429, drop
the entry with `skipped (ss_rate_limited)`.

### B.1 — DOI extraction

After Step 0 canonicalization:

1. Regex `10\.\d{4,9}/[^\s,;<>"'{}]+` on the canonicalized input
   (or file contents). The character class explicitly excludes `{}`
   so wrappers like BibTeX `doi = {10.x/y}` don't leak braces.
2. Strip trailing punctuation `.,;)}"'`.
3. Dedupe via canonical lowercase compare (see flow step 5).
4. **Fallback rule**: if regex yields DOIs for less than 50% of the
   apparent entries (e.g. BibTeX with 20 `@article` entries but only
   5 have ASCII DOIs), send the remaining entries to **B.2 title
   lookup** rather than silently dropping them.

### B.2 — Title → DOI lookup

For each title:

1. Query Crossref:
   `GET https://api.crossref.org/works?query.title=&rows=5&mailto=`
2. Pick top by `score`. **Note**: Crossref `score` is unbounded
   relevance, NOT 0–100 — absolute thresholds across queries don't
   compare. Use **relative + content** rules:
   - If `top1.score / top2.score  / ...)
   ```
   Default: download `confidence=high` rows only. `confidence=low`
   excluded unless user explicitly includes. Unresolvables dropped.
4. **Propose project name** (context-aware: "组会文献" →
   `groupmtg_`; "Smith 综述补充" → `smith_review_extras`;
   nothing topical → `custom_`). Ask user confirm.
5. **Append vs new** — if `//refs_raw.json`
   exists, ask `append / new / rename` (default: ask again on any
   other input — DO NOT default-append). **Append rules**:
   - Canonicalize new DOIs (Step 0) BEFORE compare.
   - Drop new DOIs already present in existing entries
     (case-insensitive canonical-form compare).
   - New ids start at `id = max(existing_ids) + 1`.
   - **Never renumber existing entries** — `validate_refs.py` keys
     its incremental skip on `id`, renumbering re-assigns prior
     verified metadata to the wrong DOI. Only verified rows are
     skipped; failed/pending rows revalidate on re-run.
6. **Build `refs_raw.json`** (heredoc the agent runs):
   ```python
   import json
   from datetime import datetime
   dois = [...]  # finalized canonical-lowercase list
   start_id = 1  # or max(existing_ids)+1 in append mode
   data = {
     "parent_doi": "",  # empty string for clean report labels;
                        # validate_refs.py reads as raw JSON, null
                        # would also work but "" is preferred.
     "parent_title": f"Custom batch — {user_label}",
     "extracted_at": datetime.now().isoformat(timespec="seconds"),
     "total": len(dois),
     "with_doi": len(dois),
     "without_doi": 0,
     "references": [
       {"id": i, "doi": d,
        "key": "", "unstructured": "",
        "author": "", "year": "", "journal": "",
        "volume": "", "first_page": ""}
       for i, d in enumerate(dois, start=start_id)
     ],
   }
   with open(f"{project_name}/refs_raw.json", "w", encoding="utf-8") as f:
       json.dump(data, f, ensure_ascii=False, indent=2)
   ```
   All metadata fields stay empty — `validate_refs.py` fills them
   from Crossref per DOI on success. Rows whose DOI Crossref can't
   resolve become `status=failed` with empty metadata (not partially
   enriched).
7. **Run pipeline** (skip the wrapper):
   ```bash
   cd 
   python /scripts/validate_refs.py 
   python /scripts/download_refs.py  [--auto] [--fail-fast]
   ```

### Mode B pre-flight checklist

1. **Full confirm table approved?** User saw EVERY row, not just top 5.
2. **Total count + estimated runtime?** Roughly: 0.35s × N for
   validate, ~30-90s per ref for download (publisher-dependent).
3. **Project name + new/append decided?** First-write = new project;
   second-write = append OR rename.
4. **Edge fully closed?** (Edge backend only — cloak skips.)
5. **Config set?** `[crossref].mailto` for polite-pool latency.
6. **Network reachable?** (`api.crossref.org` HEAD probe).

### Compatibility with v0.4 flags

- `--auto`: works with Mode B (manual_pending refs go to async
  retry queue same as Mode A).
- `--fail-fast`: works with Mode B (stops on first actionable
  unresolved ref).
- `[user].verified_no_si_dois`: works with Mode B (matches by
  lowercase DOI — independent of how `refs_raw.json` was produced).
- CloakBrowser backend (`REF_DOWNLOADER_BROWSER=cloak`): works
  with Mode B (browser backend is decided by env var,
  independent of input mode).

## Install prerequisites (before first invocation)

The skill protocol can't manage Python deps. If
`python -c "import playwright"` fails, the user needs:

```bash
cd ""
pip install playwright pymupdf
playwright install msedge          # downloads Edge driver
cp config.example.toml config.local.toml   # then user edits [crossref].mailto
```

If the user is developing from the source repo instead of an installed
skill copy, they can also install from the repo root with
`pip install -r requirements.txt -r requirements-dev.txt`.

## Alternative backend: CloakBrowser (Cloudflare-heavy sites)

Default backend is Microsoft Edge. For sites that keep blocking
ordinary Playwright (Cloudflare Turnstile, Radware, persistent
`Just a moment` / 安全验证 pages), switch to the CloakBrowser stealth
Chromium backend.

**What CloakBrowser is.** Third-party MIT-licensed Python package by
CloakHQ ([github.com/CloakHQ/CloakBrowser](https://github.com/CloakHQ/CloakBrowser),
[pypi:cloakbrowser](https://pypi.org/project/cloakbrowser/)). Ships a
patched Chromium build with anti-fingerprint changes. Its
`launch_persistent_context_async()` is Playwright-API-compatible,
which is why ref-downloader can swap it in with one env var. **NOT a
dependency of ref-downloader** — if the user doesn't
`pip install cloakbrowser`, it's never imported and the default Edge
path runs as normal. Beta software; user installs it at their own
discretion.

```powershell
# One-time setup (separate from ref-downloader's `pip install playwright pymupdf`)
pip install cloakbrowser

# Switch backend (env vars; no CLI flag changes)
$env:REF_DOWNLOADER_BROWSER = "cloak"
$env:REF_DOWNLOADER_CLOAK_HUMAN_PRESET = "careful"   # optional: slower mouse/scroll
# Optional overrides:
# $env:REF_DOWNLOADER_CLOAK_PROFILE = ""  # default: ~/.local/cloakbrowser/profiles/ref-downloader
# $env:REF_DOWNLOADER_CLOAK_PROXY = "http://..."
# $env:REF_DOWNLOADER_CLOAK_GEOIP = "1"
# $env:CLOAKBROWSER_PYTHONPATH = ""      # sys.path hint if cloakbrowser is checked out, not pip-installed

python "/scripts/download_refs.py" 
```

Caveats:
- cloakbrowser uses its **own Chromium with a separate profile** —
  Edge does NOT need to be closed.
- The separate profile means your institutional cookies are NOT
  carried — best for open-Cloudflare sites, less useful for
  paywalled refs your institution licenses.
- A fresh cloak profile may still show Cloudflare/security-verification
  pages on first visit; warm it manually by opening the target site
  once with the same `REF_DOWNLOADER_CLOAK_PROFILE` and finishing
  any verification before running the downloader.
- `human_preset=careful` lowers behavior-detection trigger rates but
  is **not** a captcha solver.

## Output layout

Same for both modes:

```
/
├── /
│   ├── refs_raw.json           # extract_refs.py output (Mode A) or
│   │                           # hand-built JSON (Mode B)
│   ├── refs_validated.json     # validate_refs.py output
│   ├── download_report.csv     # per-ref status (only on graceful
│   │                           # completion; OVERWRITTEN each run —
│   │                           # NOT historical truth)
│   ├── *.pdf                   # reference PDFs
│   └── *_SI.pdf                # supplementary files (where supported)
└── runs/-round-03/
    └── events.jsonl            # full event trace per ref
                                # (append-only across runs;
                                #  THIS is the authoritative history)
```

**Interruption note**: if the run is interrupted (Ctrl+C / Edge
crash / VPN drop), the root `download_report.csv` may be stale.
Trust the latest `runs//events.jsonl` + actual files in
`/`.

## Common failure modes

| Status / symptom | Meaning | Action |
|---|---|---|
| `manual_pending (auth_redirect)` | Bounced to institution SSO | User signs in via live Edge tab; re-run (incremental skips done refs) |
| `manual_pending (challenge_timeout)` | Cloudflare / publisher challenge unsolved in time | Re-run interactively; solve captcha when prompted |
| `manual_pending (elsevier_crasolve_shell)` | Elsevier viewer stuck in transition | In `--auto` mode the async retry queue picks it up ~60s later; in interactive mode the hot-session retry usually catches it, else manual click in live page |
| `failed (auto)` | Generic auto path failed | Check `events.jsonl` for that ref; may need a publisher-specific patch |
| `ignored (ignored_institution_access)` | DOI listed in `[institution].ignored_access_dois` | Skip-by-design; remove from config to retry |
| Edge won't launch | Background `msedge.exe` still holding profile | Kill all

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ltczding-gif](https://github.com/ltczding-gif)
- **Source:** [ltczding-gif/ref-downloader](https://github.com/ltczding-gif/ref-downloader)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-ltczding-gif-ref-downloader-ref-downloader
- Seller: https://agentstack.voostack.com/s/ltczding-gif
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.