Research

Deep Research — Multi-Source Research Pipeline

Automates the full research workflow: classify sources, extract content in parallel, research the topic broadly via web search, synthesize everything into a comprehensive Obsidian research note, validate syntax, and write to the vault.

When to Use

• Researching a YouTube video, arXiv paper, SSRN paper, PDF, or blog post
• Combining multiple sources into a single comprehensive research note
• Creating research notes that match the quality bar of 05 Research/Ralph Wiggum Loop/
• Any "I watched/read this, now I want a deep note on it" workflow

When NOT to Use

• Quick notes or daily log entries → use /obsidian
• Literature reviews across many papers → use manual research workflow
• Content that doesn't need web research → just use the extraction tools directly

Invocation

/research                                         # Single source
/research                            # Multiple sources
/research  --output "05 Research/Custom Name"     # Custom output path
/research  --no-broad-search                      # Skip Phase 3 web research
/research  --slides-max 10                        # Override max slide count
/research ~/Downloads/paper.pdf --title "Paper Title"  # Local PDF with title

Skill Directory

~/.claude/skills/deep-research/
├── SKILL.md                              # This file
├── scripts/
│   ├── mathpix_convert.py                # Mathpix API: submit, poll, download, postprocess
│   ├── ssrn_download.py                  # SSRN PDF download with Cloudflare bypass
│   └── source_classifier.py              # URL classification + metadata extraction
├── settings/
│   ├── extraction-defaults.json          # Default params per source type
│   ├── research-note-template.md         # Synthesis template + formatting rules
│   └── credentials-paths.json            # Credential file locations
└── references/
    ├── obsidian-syntax-rules.md          # P0-P4 validation rules
    └── source-type-guide.md              # Per-source extraction methods + known issues

Vault Details

• Path: Auto-detected from CWD (walks up looking for .obsidian/), or set VAULT_ROOT env var
• Credentials: $VAULT_ROOT/.credentials/dissertation-research/.env (Mathpix keys)
• SSRN cookies: $VAULT_ROOT/.credentials/cookies/ssrn-cookies.txt

---

Pipeline — Six Phases

Phase 0: Input Classification

For each input URL or path:

1. Run classification (inline or via the script):

``bash python3 ~/.claude/skills/deep-research/scripts/source_classifier.py "" --json ``

1. Returns: type (youtube | arxiv | ssrn | pdfurl | pdflocal | web_article),

id, url, slug

1. Determine topic slug from classification output or --title override
2. Determine output location (see Output Routing below)

Phase 1: Parallel Extraction

Extract content from each source. Run independent sources in parallel using background Bash tasks or concurrent tool calls.

| Source Type | Extraction Method | Output | |---|---|---| | YouTube | summarize "" --extract --json then summarize slides "" --slides-dir | raw-transcript.md + slides/*.png | | arXiv | python3 scripts/mathpix_convert.py --url --output .mathpix.md | .mathpix.md + .pdf | | SSRN | python3 scripts/ssrn_download.py --ssrn-id --output .pdf then python3 scripts/mathpix_convert.py --file .pdf --output .mathpix.md | .mathpix.md + .pdf | | PDF (URL) | curl -L -o temp.pdf "" then python3 scripts/mathpix_convert.py --file temp.pdf --output .mathpix.md | .mathpix.md | | PDF (local) | python3 scripts/mathpix_convert.py --file --output .mathpix.md | .mathpix.md | | Web article | WebFetch(url, "Extract the full article...") | -article.md |

YouTube extraction details:

1. Extract transcript: summarize "" --extract --json

• Parse JSON output for title, content, metadata
• Save as raw-transcript.md with frontmatter

1. Extract slides: summarize slides "" --slides-dir "/slides"

• Saves PNGs named slide_NNNN_Ts.png (number + timestamp)
• Default max: 30 slides (override with --slides-max)

SSRN fallback: If cookies are expired or TLS fingerprint blocked:

1. Inform user cookies need refreshing
2. Provide the SSRN abstract URL for manual download
3. User provides local PDF path → continue with mathpix_convert.py --file

Phase 2: Visual Analysis (YouTube only)

If the source includes YouTube slides:

1. Read each extracted slide image via Claude's Read tool (multimodal)
2. For each slide, produce a one-line annotation:

"Slide N (Ts): [description of what the slide shows]"

1. Map slides to transcript timestamps for contextual embedding in the final note
2. This builds the ![[slide_NNNN_Ts.png]] references used in synthesis

Skip this phase if --no-broad-search is set and there are no slides.

Phase 3: Broad Topic Research

Unless --no-broad-search is set:

1. From the extracted content, generate 3-5 targeted WebSearch queries:

• Community discussions (Reddit, HN, forums)
• GitHub repositories and implementations
• Blog posts, tutorials, practitioner guides
• Official documentation or specs
• Academic context (if not already an academic source)

1. Execute WebSearch queries (parallel when independent)

1. WebFetch the top 3-5 most relevant results from search

1. Compile findings with source attribution:

`` [Source Title](url) — one-line summary of what it adds ``

Phase 4: Synthesis

Read the research note template at settings/research-note-template.md, then compose the research note using ALL Phase 1-3 outputs.

Required structure:

1. Frontmatter: YAML with real values — source URLs, authors, tags, status.

Use the appropriate source fields from the template. No placeholders.

1. Title: # Research Note:
2. Question: One clear question the note answers
3. Key Idea: 2-4 paragraph executive summary. Include canonical code/formula/diagram
4. Numbered H2 sections (## N. Title): 4-10 sections with H3/H4 hierarchy.

Depth driven by content richness. Include:

• Embedded slides: ![[slide_NNNN_Ts.png]] where relevant
• Code blocks with language identifiers
• Blockquotes with attribution: > "quote" -- source
• Comparison tables where applicable
• Internal links: [[raw-transcript.md]], [[slug.mathpix.md]]

1. Evidence: Primary sources (input content) + secondary sources (web research)
2. Implications: How this connects to our projects and workflows
3. Links: Internal wiki-links to extraction artifacts + related vault notes

Quality bar: The output should match 05 Research/Ralph Wiggum Loop/Ralph Wiggum Loop.md in depth — typically 300-500 lines with embedded slides, hyperlinked sources, numbered sections, code examples, and comparison tables.

Phase 5: Validation

Before writing the final note, validate against references/obsidian-syntax-rules.md:

P0 — Math delimiters:

• No \[...\] → must be $$...$$
• No backtick-wrapped LaTeX → must be $...$
• Mathpix \(...\) already converted by mathpix_convert.py postprocessor

P1 — Mermaid syntax:

• Labels with parentheses must be double-quoted: F["text (with parens)"]
• No LaTeX inside Mermaid labels
• No \n literals in labels — use em-dash separators
• No `` tags

P3 — Frontmatter:

• No placeholder tags (topic-tags, [Author Name])
• All required fields present with real values

Fix any violations inline before proceeding to Phase 6.

Phase 6: Write Output

1. Create the topic folder at the routed vault location:

``bash mkdir -p "//" ``

1. Write the main research note: /.md
2. Write/move all supporting files:

• raw-transcript.md (YouTube)
• .mathpix.md (arXiv/SSRN/PDF)
• .pdf (arXiv/SSRN/PDF)
• slides/ directory (YouTube)
• -article.md (web articles)

1. Report completion:

`` Research complete: /// ├── .md (NNN lines) ├── raw-transcript.md (if YouTube) ├── .mathpix.md (if PDF source) ├── .pdf (if PDF source) └── slides/ (N slides, if YouTube) ``

---

Output Routing

Strategy: Use --output if provided. Otherwise, infer or ask.

| Context | Behavior | |---|---| | --output provided | Use exact path under vault root | | arXiv/SSRN paper on a dissertation topic | Route to 10 School/Dissertation Research/Papers// | | General research / tutorial / technique | Route to 05 Research// | | Ambiguous | Ask the user via AskUserQuestion |

When asking, suggest 2-3 locations based on content analysis:

AskUserQuestion:
  "Where should this research note be saved?"
  options:
    - "05 Research/" (general research)
    - "10 School/Dissertation Research/Papers/" (dissertation)
    - "03 Resources/" (reference material)

---

Configuration

extraction-defaults.json

| Key | Default | Purpose | |---|---|---| | youtube.slides.max_slides | 30 | Max slides to extract | | youtube.slides.threshold | 0.3 | Scene change detection threshold | | mathpix.poll_interval_seconds | 5 | Seconds between status polls | | mathpix.poll_timeout_seconds | 300 | Max wait for Mathpix processing | | web_search.queries_per_topic | 4 | Number of search queries in Phase 3 | | web_search.max_fetch_results | 5 | Max pages to WebFetch per search | | output.default_location | 05 Research | Default vault subfolder |

credentials-paths.json

| Credential | Path | Notes | |---|---|---| | Mathpix .env | .credentials/dissertation-research/.env | MATHPIX_APP_ID, MATHPIX_APP_KEY | | SSRN cookies | .credentials/cookies/ssrn-cookies.txt | Netscape format, expires fast |

All paths relative to vault root (set VAULT_ROOT env var or run from within your vault directory)

---

Prerequisites

• summarize CLI: npm i -g @steipete/summarize (for YouTube transcript + slides)
• yt-dlp: brew install yt-dlp (keep updated: brew upgrade yt-dlp)
• ffmpeg: brew install ffmpeg (for slide extraction)
• jq: brew install jq (for JSON parsing in shell)
• Mathpix API: Credentials in .credentials/dissertation-research/.env
• Python 3.10+: For scripts (system Python on macOS works)

Optional:

• tesseract: For OCR on extracted slides (--slides-ocr)
• SSRN cookies: Only needed for SSRN papers

---

Error Handling

| Error | Cause | Recovery | |---|---|---| | summarize fails | yt-dlp outdated or bot detection | brew upgrade yt-dlp, try --cookies-from-browser brave | | Mathpix 401 | Bad API credentials | Check .credentials/dissertation-research/.env | | Mathpix timeout | Large PDF (50+ pages) | Use --page-ranges for partial processing | | SSRN HTML instead of PDF | Cookies expired | Re-export from browser, or download PDF manually | | SSRN TLS blocked | Cloudflare fingerprint mismatch | Download manually, use mathpix_convert.py --file | | WebFetch 403 | Site blocks automated requests | Try Playwright MCP or note limitation | | WebFetch empty content | JavaScript-rendered SPA | Escalate to Playwright MCP |

---

Examples

Single YouTube Video

/research https://www.youtube.com/watch?v=I7azCAgoUHc

→ Extracts transcript + slides, researches Ralph Wiggum loops broadly, synthesizes into 05 Research/Ralph Wiggum Loop/Ralph Wiggum Loop.md

arXiv Paper

/research https://arxiv.org/abs/2502.07766

→ Downloads PDF, converts via Mathpix, researches the topic, synthesizes into research note with embedded formulas

Multiple Sources

/research https://www.youtube.com/watch?v=abc123 https://blog.example.com/post

→ Extracts both in parallel, combines into single research note

Custom Output

/research https://arxiv.org/abs/2301.12345 --output "10 School/Dissertation Research/Papers/my-paper"

→ Routes output to dissertation folder

Local PDF

/research ~/Downloads/paper.pdf --title "Market Microstructure Dynamics"

→ Converts local PDF, researches the topic, writes note

Skip Web Research

/research https://arxiv.org/abs/2301.12345 --no-broad-search

→ Extract and synthesize only from the source content, no web search

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ZealousEar
• Source: ZealousEar/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.