Ultrasearch

ultrasearch — thesis-level literature research

⛔ Rules that override everything else

1. NEVER fabricate DOIs or citations. Every DOI in the final report MUST exist in papers.doi of corpus.db. A regex-validated check is in the Checklist section; run it before delivering output.
2. NEVER skip the env-var pre-flight. OPENALEX_API_KEY and UNPAYWALL_EMAIL are not optional. Failing fast on missing env vars is correct behavior, not a bug.
3. NEVER bypass the venv. Every script is invoked through ensure_env.py, which resolves the skill's venv (a global state dir outside the code — ADR-008). Calling extract/index scripts with system python3 will fail — sentence-transformers, pymupdf4llm, and sqlite-vec live only in that venv.
4. NEVER deliver a report whose paragraphs lack [Sn] citation markers. synthesize.validate_report() returns the offenders — fix or remove them.
5. NEVER delete corpus.db. It is user content (ADR-008), stored in a global state dir outside the code; reinstall never touches it. If the user explicitly asks to reset, resolve the path as in Persistent corpus and rm "$DB"*.

Required environment variables

export OPENALEX_API_KEY="..."    # free key — https://openalex.org/account
export OPENALEX_EMAIL="you@example.com"   # polite-pool identifier
export UNPAYWALL_EMAIL="you@example.com"  # mandatory for Tier 2 fetch
export S2_API_KEY="..."           # optional — dedicated 1 RPS pool
# Stage 2 additions:
export CORE_API_KEY="..."         # optional — enables CORE source (5 of 6 work without)
export ZOTERO_API_KEY="..."       # optional — enables --export-zotero remote push
export ZOTERO_USER_ID="..."       # required if ZOTERO_API_KEY set

| Var | Required? | What breaks without it | |---|---|---| | OPENALEX_API_KEY | YES (post 13 Feb 2026) | OpenAlex returns 401 for new users | | OPENALEX_EMAIL | Recommended | drops to anonymous quota | | UNPAYWALL_EMAIL | YES | Tier 2 (Unpaywall PDF lookup) skipped; cascade loses ~30% recall | | S2_API_KEY | Optional | shares anonymous 5000/5min bucket; traversal hits it harder | | CORE_API_KEY | Optional (Stage 2) | CORE source disabled; other 5 sources cover | | ZOTERO_API_KEY | Optional (Stage 2) | --export-zotero writes .bib only, no remote push | | ZOTERO_USER_ID | Required if ZOTEROAPIKEY set | API push fails fast |

Pre-flight one-liner:

for v in OPENALEX_API_KEY OPENALEX_EMAIL UNPAYWALL_EMAIL S2_API_KEY; do
    printf "%-22s %s\n" "$v" "${!v:+set}"
done

When to use

Triggers (any language):

• "literature review on X", "найди все статьи про X"
• "related work for my thesis / paper / RFC"
• "систематический обзор по теме X", "ВКР по нейроинтерфейсам"
• "what do papers say about X / какие исследования есть по теме Y"
• "thesis prep", "обзор источников", "deep research on X"

NOT for:

• single-paper summarization → use Anthropic's pdf skill
• generating new papers/code from a topic → use gost-report (RU) / your own writing
• code documentation → use repomix / gitingest
• general web research without academic grounding → use WebSearch / WebFetch directly

Canonical invocation

Run it with us (installed on PATH with the skill). It bootstraps the skill's venv on first call (~3-5 min cold, .bib | Stage 2: path for .bib file | | --refresh-retractions | off | Stage 2: pull latest Retraction Watch CSV | | --no-docling | off | Stage 2: disable docling fallback (pymupdf4llm only) | | --sources LIST | all 8 | comma-separated subset of {openalex,s2,arxiv,crossref,europepmc,core,datacite,hf} | | --profile NAME | academic | v2: pipeline profile (auto, academic, dev, docs). auto = classifier picks. academic = v1 pipeline (default). Other profiles use the generic flat orchestrator. | | --retrieval-profile fast\|full | fast | Renamed from v1 --profile fast/full. Old --profile=fast/--profile=full still works via deprecation shim. full = cross-encoder rerank + RCS-cached scoring + (planned) multi-section synth | | --profiles-dir PATH | /profiles | override profile YAML directory | | --no-classifier | off | skip classifier; require explicit --profile | | --output-template NAME | (profile default) | force a template (library_matrix.md.j2, adr.md.j2, ...) | | --lang auto\|en\|ru | auto | Stage 3: query-language detection + EN parallel for non-EN queries (argos-translate) | | --render-graph | off | Stage 3: embed Mermaid citation graph in the report | | --grey` | off | Stage 3: sci-hub opt-in fallback (LEGAL GRAY ZONE — see disclaimer) |

v2 profiles (v0.15+)

v2 introduces profile-routed pipelines. The academic profile (default) is the v1 pipeline below, byte-identical. Other profiles use a generic flat orchestrator (orchestrate.py) over source clients in scripts/sources/.

| Profile | Sources | Output template | Status | |---|---|---|---| | academic | OpenAlex, S2, arXiv, Crossref, EuropePMC, CORE | literature_review.md.j2 | v1 (full, ships) | | dev | GitHub, Stack Exchange, HN Algolia, deps.dev, PyPI, docs_crawl | library_matrix.md.j2 (alt: adr.md.j2) | Stage 1 (ships) | | docs | llms.txt → sitemap → Crawl4AI BFS → trafilatura | literature_review.md.j2 | Stage 1 (ships; crawl4ai lazy-installed on first use) | | auto | (classifier picks) | (per primary profile) | Stage 1 ships keyword fallback; subagent integration is Stage 1.5 |

Query style by profile — this matters. Only the academic profile does semantic (SPECTER-embedded) matching and handles full natural-language topics. The dev profile hits GitHub repo search, Stack Exchange, HN Algolia, and package registries — engines that match short keyword / library-name queries, not prose. Feed it terse terms or a tool name:

• ✅ "rust async runtime", "pdf table extraction python", "BFG git secrets", redis
• ❌ "what are the best practices for safely cleaning up a git repository and removing stale artifacts" → returns ~zero hits; these APIs don't do sentence-level semantic search, and the output is a library-comparison matrix, not a how-to.

So reach for dev when the question is "which library/tool for X". For how-to / best-practice questions, use a few keywords, the docs profile against known doc roots, or plain WebSearch / WebFetch. The auto classifier can still mis-route a long prose query to dev; pass --profile explicitly (or shorten the query) when you know you want tool discovery.

Routing logic: ultrasearch.py resolves --profile:

• omitted / academic / legacy fast|full → v1 _pipeline() (academic)
• auto → classifier (keyword fallback for now) → picks primary profile
• named profile with YAML → v2 orchestrate.run_pipeline(profile)
• named profile without YAML (e.g. --profile=regulatory in Stage 1) → falls back to academic with a log line

Customizing: drop a .yaml into /profiles/, validated against profiles/_schema.json. Sources reference modules in scripts/sources/.py. Scoring is a scoring/.py: reference. Output template is a Jinja2 file in templates/.

SSRF / XXE hardening: crawl/__init__.py:_url_is_safe() rejects loopback / private / link-local / cloud-metadata IPs; follow_redirects=False; sitemap parsing uses regex (no XML parser). Source-client HTTP error logs sanitize URLs (type(e).__name__ only).

v2 example invocations

# dev profile (force, skip classifier)
us "best Python web framework 2026" \
    --profile=dev --max-papers 25 --top-k 8 \
    --out /tmp/web-frameworks.md

# auto-classify, pure-academic query routes to v1 pipeline unchanged
us "literature review on SSVEP BCI" --profile=auto --max-papers 30

# docs profile with explicit root URLs — uses the orchestrate.py entry, so call it
# through ensure_env.py directly (us maps to ultrasearch.py only)
python3 ~/.claude/skills/ultrasearch/scripts/ensure_env.py orchestrate.py \
    "API reference" --profile=docs \
    --max-items 50 --top-k 20 --out /tmp/docs.md
# Requires: ensure_env.py runs ensure_crawl4ai() to lazy-install Playwright

Pipeline (academic profile — v1 / Stage 2)

discover (async, 6 sources concurrent)
   ├─ OpenAlex    /works              ← abstracts + referenced_works (traversal seed)
   ├─ Semantic Scholar /paper/search   ← citation_count, externalIds
   ├─ arXiv       /api/query           ← preprints + bare-ID lookup
   ├─ Crossref    /works               ← Stage 2 — DOI metadata + backward refs
   ├─ Europe PMC  /search              ← Stage 2 — biomed full-text manifests
   └─ CORE        /v3/search/works     ← Stage 2 — 449M OA articles
       ↓
   dedup by normalize_doi(doi) or 'T:' + casefold_title(title)
       ↓
   fetch (async, sem=4, 7-tier cascade)
   ├─ Tier 1: OpenAlex primary_location.pdf_url
   ├─ Tier 2: Unpaywall best_oa_location.url_for_pdf
   ├─ Tier 3: arXiv direct PDF
   ├─ Tier 4: Europe PMC PMC fulltext
   ├─ Tier 5: publisher OA templates (PLOS, Frontiers)
   ├─ Tier 6: Zenodo records
   └─ Tier 7: arXiv HTML (ar5iv)  ← Stage 3 hook
       ↓
   parse (subprocess per PDF, pymupdf4llm + docling fallback)
   ├─ default: pymupdf4llm only
   └─ per-page heuristic → docling re-parses table/math pages, stitch back in
       ↓
   index (serial, BEGIN IMMEDIATE writes)
   ├─ chunk_text → ~500-token paragraph-aware windows
   ├─ allenai-specter encode (MPS on Apple Silicon, ~50 chunks/s)
   └─ vec_chunks INSERT (rowid == chunks.chunk_id invariant)
       ↓
   retrieve (top-20 via sqlite-vec MATCH)
       ↓
   traverse (Stage 2 — PaperQA2 Algorithm 1)
   ├─ seeds = top-10 hits with score ≥ 0.8
   ├─ 3 API calls per seed: S2 refs + Crossref refs + S2 citers
   ├─ SPECTER cosine gate against corpus centroid (≥ 0.55)
   └─ overlap filter + ell=12 cap → list of new candidates
       ↓
   quality (Stage 2 — Retraction Watch join)
   ├─ optional --refresh-retractions: pull CSV from gitlab.com/crossref/retraction-watch-data
   └─ mark_retractions: UPDATE papers.retracted_at on DOI match
       ↓
   synthesize → markdown with [S1]..[Sn] markers
   ├─ ⚠ RETRACTED footnotes on flagged refs
   ├─ Open Questions section pulled from traversal candidates
   └─ validate_report: every paragraph has [Sn]; every [Sn] in References
       ↓
   (optional) export → .bib via bibtexparser + Zotero push via pyzotero

First-run cost

• torch wheel ~150 MB on macOS arm64
• sentence-transformers + transformers ~80 MB
• allenai-specter model ~440 MB (downloaded on first index.py call, NOT during pip install)
• docling model bundle ~358 MB (downloaded on first docling fallback trigger, NOT during pip install) — Stage 2
• git clone of Retraction Watch repo ~50 MB (Stage 2 — only if --refresh-retractions)
• Cold install: 5-8 minutes
• Warm second run: — research brief

TL;DR

Findings

() —

... [S1]

...

Open questions / related directions

• ** — [S9]

References

• [S1] 10.xxxx/yyyy — **, , , .

...

## What ships in v1.0 (Stages 1+2+3)

**Stage 1 (MVP):** OpenAlex + Semantic Scholar + arXiv discovery, Unpaywall + arXiv 2-tier fetch, pymupdf4llm parsing, SPECTER embedding in sqlite-vec, deterministic markdown report with `[Sn]` citations.

**Stage 2 (v0.5):** Crossref/Europe PMC/CORE added (6 sources total), 7-tier PDF cascade with publisher OA templates and Zenodo, docling fallback on math/table-heavy pages, PaperQA2 Algorithm 1 citation traversal (`--depth`), Retraction Watch CSV ingest + DOI join (`--refresh-retractions`), BibTeX/Zotero export (`--export-zotero`), Open Questions section in synthesis.

**Stage 3 (v1.0):**
- 3-stage retrieval (`--profile full`): sqlite-vec top-50 → bge-reranker top-10 → RCS-cached scoring (≥5 keeps)
- Multilingual query expansion (`--lang auto`): argos-translate EN↔RU; abstract translations stored in `papers.abstract_translated`
- Composite Q-score: `0.3·log(cit+1) + 0.2·venue + 0.2·h_index + 0.3·recency` (populated lazily by `quality.populate_q_scores`)
- MMR diversity selection (λ=0.7) + institution≤25% / primary-author≤15% caps in `quality.apply_diversity_caps`
- Mermaid citation graph rendering (`--render-graph`, capped at 50 nodes)
- DataCite + HuggingFace Papers added to discovery; КиберЛенинка / OATD / BASE stubs in place for v1.1
- CriticAgent prompt template at `prompts/critic.txt` (5 issue types, severity rubric; orchestrator-side validation loop in roadmap)
- Sci-hub `--grey` opt-in tier 8 with explicit disclaimer (legal gray zone — research §14 line 300)
- New tables: `rcs_scores` (PaperQA2 RCS score cache keyed by `sha256(query)[:16]`)

## Known deferrals (post-v1.0)

- specter2 adapter framework / multilingual-e5-large / mlx-embeddings backend (alt embedders)
- PostToolUse hook for auto-indexing PDFs you Read (per research §13 line 289)
- CriticAgent loop integration into `synthesize.py` (the prompt is shipped; the orchestrator currently uses deterministic synth — wrapping the critic into a retry loop is Stage 3.1)
- LanceDB migration when corpus crosses ~500k chunks
- mlx-lm offline LLM for RCS scoring (research §11 line 240)
- BEIR-SciFact / TREC-COVID benchmark suite (research §recommendations item 5)
- `/ultrasearch-update` slash command for periodic Retraction Watch + corpus refresh
- КиберЛенинка / OATD / BASE actual scrapers (stubs registered; OAI-PMH parsing pending)

## References (lazy-loaded)

- `references/apis.md` — endpoint specs, rate budgets, env vars (all 11 sources)
- `references/parsing-troubleshooting.md` — pymupdf4llm AGPL, MPS device, docling fallback notes
- `references/methodology.md` — Wohlin 2014 snowballing + PaperQA2 Algorithm 1 + SPECTER cosine gate
- `references/ru-sources.md` — КиберЛенинка status, eLIBRARY exclusion, РГБ note, translation flow (Stage 3)
- `prompts/perspective-questions.txt` — STORM-style perspective generation (Stage 2)
- `prompts/rcs-summary.txt` — PaperQA2 RCS chunk-scorer prompt (Stage 3)
- `prompts/section-writer.txt` — section-draft subagent prompt (Stage 3)
- `prompts/polisher.txt` — final stitch + TL;DR + References writer prompt (Stage 3)
- `prompts/critic.txt` — CriticAgent integrity-gate prompt (Stage 3)

## Dependencies

`scripts/requirements.txt`: pyalex, semanticscholar, arxiv, httpx[http2], aiometer, tenacity, unpywall, pymupdf4llm, torch, sentence-transformers, sqlite-vec, numpy, tiktoken. No system deps beyond Python 3.10+.

**AGPL note:** `pymupdf4llm` and its transitive `pymupdf` are AGPL v3 (Artifex). Fine for an open-source skill; commercial reuse needs a license from Artifex. See `references/parsing-troubleshooting.md`.

## License

See LICENSE — MIT.

## Checklist (before delivering output)

1. Report is non-empty: `wc -c ` > 1000
2. Every paragraph cites at least one `[Sn]` marker:
   ```bash
   python3 -c "import re,sys; t=open('$REPORT').read(); body=t.split('## References',1)[0]; paras=[p for p in re.split(r'\\n\\s*\\n', body) if p.strip() and not p.lstrip().startswith(('#','>','-','*','|','\`'))]; bad=[p for p in paras if '[S' not in p]; print('bad paragraphs:', len(bad))"
   ```
3. Every `[Sn]` resolves to a References entry:
   ```bash
   python3 -c "import re; t=open('$REPORT').read(); body,refs=t.split('## References',1) if '## References' in t else (t,''); a=set(re.findall(r'\\[(S\\d+)\\]',body)); b=set(re.findall(r'\\[(S\\d+)\\]',refs)); print('missing in refs:', sorted(a-b))"
   ```
4. No fabricated DOIs:
   ```bash
   sqlite3 "${ULTRASEARCH_HOME:-${XDG_DATA_HOME:-$HOME/.local/share}/agentpipe/ultrasearch}/data/corpus.db" <<SQL
   .mode list
   SELECT count(*) FROM papers WHERE doi IS NOT NULL;
   SQL
   # Cross-check with: grep -oE '10\\.\\d{4,}/[^ )\\]]+' "$REPORT" | sort -u
   # Every DOI in the report must appear in papers.doi.
   ```
5. `corpus.db` grew by ≥1 paper this invocation (`schema_meta.last_indexed_at` advanced, or row counts increased).

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [zevtos](https://github.com/zevtos)
- **Source:** [zevtos/agentpipe](https://github.com/zevtos/agentpipe)
- **License:** MIT

Install and usage instructions live in the source repository linked above.