Download Gated Pdfs

Download bot-gated PDFs via Wayback id_

Problem

Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML.

Context / Trigger Conditions

• curl -o file.pdf succeeds but the file starts with `id_/" -o out.pdf

`` is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The id_` suffix after the timestamp is what requests the untouched original.

1. Verify the download with pypdf — a bot page fails immediately:

``python from pypdf import PdfReader r = PdfReader("out.pdf"); print(len(r.pages), "pages") ``

1. If Wayback has no capture, fall back to: another mirror found via search

(Exa/Firecrawl), or Firecrawl scrape for the parsed text when the binary is not strictly needed.

Verification

PdfReader opens the file and reports a plausible page count; first-page text matches the expected title.

Example

Verified 2026-07-15: taxpolicycenter.org/sites/default/files/publication/165884/ssrn-id4797771.pdf and urban.org/sites/default/files/publication/80621/2000790-...pdf both bot-gated to direct curl (with UA), both downloaded intact via https://web.archive.org/web/2024id_/ and .../web/2023id_/ (18 and 12 pages).

Notes

• Government data hosts (e.g. ticdata.treasury.gov) are usually NOT gated — try direct

curl first; Wayback is the fallback, not the default.

• Wayback captures can be stale for frequently-revised documents; check the snapshot date

if currency matters.

• See also: the pdf skill (parsing/extraction after download) and

firecrawl:firecrawl-scrape (parsed markdown when the binary is unnecessary).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kennethkhoocy
• Source: kennethkhoocy/legal-scholarship-skills
• License: MIT

Install and usage instructions live in the source repository linked above.