# Social Fetch

> Fetch content from social-media URLs (HackerNews, Reddit, GitHub, X/Twitter, LinkedIn, YouTube, Bluesky, arXiv, Medium, Substack, RSS, generic articles) and run web/social searches (DuckDuckGo, Brave, SerpAPI, Tavily, X, HN, YouTube, Bluesky, arXiv) — output as clean markdown or structured JSON. Use whenever the user asks to "pull", "fetch", "download", "summarise", or "search the web/Twitter/HN/…

- **Type:** Skill
- **Install:** `agentstack add skill-jedi4ever-social-skills-social-fetch`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [jedi4ever](https://agentstack.voostack.com/s/jedi4ever)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [jedi4ever](https://github.com/jedi4ever)
- **Source:** https://github.com/jedi4ever/social-skills/tree/main/skills/social-fetch

## Install

```sh
agentstack add skill-jedi4ever-social-skills-social-fetch
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# social-fetch skill

Wraps the `social-fetch` Go binary at `scripts/social-fetch` (relative to this skill).

**Trust the CLI.** It is the authority for every fetch and search supported by this skill. Always shell out to `scripts/social-fetch` — never reimplement fetching with WebFetch, curl, custom parsers, or hand-rolled API calls, even if the binary returns empty results or an error you find surprising. If a fetch comes back empty, surface that to the user and (if appropriate) re-run with `--log -` to see audit lines, but do not try to "fix it" by going around the CLI.

**Before invoking any provider, run `scripts/social-fetch list` to see which platforms are usable in this environment.** Each provider is tagged with one of three states:

- `[ok]` — fully configured, fire away
- `[!auth]` — required env var not set; the row's suffix names which one (e.g. `→ missing BRAVE_API_KEY`). **Do not use these providers** — pick a different one in the same category. Suggesting the user configure the missing key is fine *only if they explicitly ask how*; don't proactively nag them about every unset key.
- `[bridge]` — needs the local browser bridge (LinkedIn / Medium / Substack / linkedin search & timeline). Only use after `scripts/social-fetch bridge status` reports connected.

The `auto` provider chains (`-p auto`) already skip unconfigured providers, so they're safe — but when an explicit provider name is needed (e.g. the user asks "search Twitter"), check `list` first to confirm `[ok]` status before invoking. For the MCP shape see `social_fetch_list_providers` — same data, structured as `{name, status, missing}` per category.

**For platform-specific quirks, run `scripts/social-fetch hints`** (no argument — dumps every platform's hints in one shot) before a search/fetch you haven't done recently. Captures things like "X recent search caps at 7 days strictly", "LinkedIn temp-bans accounts that scrape too fast", "Reddit anonymous search has worse relevance than `tavily site:reddit.com`". Pass a specific platform name to scope the output (e.g. `hints x`).

## Subcommands

```
scripts/social-fetch fetch     [...]    [flags]
scripts/social-fetch search   ""           [flags]
scripts/social-fetch timeline        [flags]   recent activity for a user (X / LinkedIn)
scripts/social-fetch ask      ""        [flags]   grounded answer engine (perplexity / grok / openai / anthropic / gemini / tavily / serpapi)
scripts/social-fetch research ""        [flags]   EXPERIMENTAL — multi-angle research (decompose → parallel fan-out → synthesize)
scripts/social-fetch bridge   {start|stop|status|run}
scripts/social-fetch bookmarks {list|profiles}              local browser bookmarks (chrome today; --platform NAME)
scripts/social-fetch hints    []                  per-platform quirks, rate limits, gotchas (x / linkedin / reddit / ...)
```

Run `scripts/social-fetch --help` for the full reference. Output defaults to **markdown**; pass `-f json` or `-f jsonl` for structured input to other tools.

## Credentials (.env support)

Provider keys (`X_API_KEY`, `X_API_SECRET`, `TAVILY_API_KEY`, `SERPAPI_KEY`, `BRAVE_API_KEY`, `PERPLEXITY_API_KEY`, `XAI_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`/`GOOGLE_API_KEY`/`GOOGLE_CSE_ID`, `YOUTUBE_API_KEY`, `BLUESKY_HANDLE`/`BLUESKY_APP_PASSWORD`, `GITHUB_TOKEN`) and routing hints (`HTML2MD_PROVIDER`, `HTML2MD_READER`, `YOUTUBE_TRANSCRIPT_PROVIDER`, `TAVILY_TOPIC`) can be set in the shell **or** placed in a `.env` file. At startup the binary loads, in order:

1. `./.env` (current working directory)
2. `/.env` (sits next to the installed binary — typically `~/.claude/skills/social-fetch/.env`)

Already-exported shell vars always win over file entries.

## Decision rules

- **One URL → fetch it.** `scripts/social-fetch fetch ` auto-detects the source from the host (HN, Reddit, GitHub, X, RSS, or generic article).
- **Images / diagrams in the post →** every fetched item carries a `Media` list (LinkedIn post photos, X/Twitter media, Medium/Substack figures, YouTube thumbnails, generic article images). The MCP `social_fetch_fetch` envelope surfaces this as a `media[]` array of `{url, type, alt}` entries. When the user asks "what's on the picture / diagram / screenshot in this post", the agent's vision-capable Read tool (Claude Code / Claude Desktop) can read each `media[].url` directly — no extra fetch call. Empty `alt` usually means the image is worth looking at; populated `alt` is the author's caption. Same data is in the `## Media` section of the markdown content.
- **A list of URLs → batch.** Pipe via stdin (`cat urls.txt | scripts/social-fetch fetch`) or use `-i FILE`. Add `-j 8` for parallel fetches; output stays in input order.
- **Save to disk →** `-o FILE` for one file, `-o DIR/` for one file per URL.
- **A user's recent posts → timeline.** `scripts/social-fetch timeline  [-p x|linkedin] [--kind ...] [-n N]`. Auto-detects the provider from URL; default for bare handles is X. See "Timeline subcommand" below.
- **A grounded question → ask.** `scripts/social-fetch ask "" -p perplexity|grok|openai|anthropic|gemini|tavily|serpapi`. Returns synthesized answer + sources. Use this only when the user explicitly wants a synthesized answer; for raw documents use `fetch` or `search`.
- **A multi-angle research question → research (EXPERIMENTAL).** `scripts/social-fetch research "" --max-angles 5 --jobs 4`. Decomposes into 3-8 angles, fans out parallel queries, synthesizes a final answer with citations. Use when you'd otherwise issue 4-8 manual queries. Costs roughly 2 LLM calls + N tool calls per question; use `ask` for simple lookups instead.
- **A query → search.** Pick the provider that matches the user's intent. `-p auto` walks `perplexity → tavily → brave → serpapi → duckduckgo`; comma-lists like `-p tavily,duckduckgo` define a custom fallback order. Each falls through on missing key / error / 0 results.
  - "search the web" / unspecified → `duckduckgo` (no auth)
  - "search Brave" / privacy-focused web → `brave` (needs `BRAVE_API_KEY`; native `--last 7d` via freshness)
  - "high-quality web search for AI agents" → `tavily` (needs `TAVILY_API_KEY`)
  - "Perplexity index without synthesis" → `perplexity` (needs `PERPLEXITY_API_KEY`; same key as `ask -p perplexity`, but cheaper since no LLM tokens)
  - "LinkedIn posts about " → `linkedin` (requires the browser bridge + a logged-in LinkedIn session; up to 50 results per query via scroll-to-bottom + wheel-event lazy-load). **Use sparingly.** LinkedIn aggressively rate-limits accounts that scrape — running this back-to-back will get the user temp-banned. Prefer `tavily` / `perplexity` / `serpapi` for general "who's writing about X" questions, and only reach for `-p linkedin` when LinkedIn-specific posts are explicitly the goal.
  - "search Bluesky" → `bluesky` (no auth, native date filter)
  - "search arXiv" / academic papers → `arxiv` (no auth, sorted newest-first)
  - "search HN" → `hackernews`
  - "search Reddit" → `reddit` (no auth, public search.json; rate-limited per IP)
  - "search Twitter/X" → `x` (needs `X_API_KEY` + `X_API_SECRET`)
  - "search via Google" → `serpapi` (needs `SERPAPI_KEY`)
  - "search YouTube" → `youtube` (needs `YOUTUBE_API_KEY`; supports `--last 7d` / `--after` natively, dates are strict)

## Flags worth remembering

| flag | when |
| -- | -- |
| `-f markdown\|json\|jsonl` | format (default markdown) |
| `-o PATH` | stdout / FILE / DIR/ |
| `-i FILE` | URLs file (`-` = stdin; auto-detected when piped) |
| `-j N` | parallel workers for batch fetch |
| `--no-comments` | skip comment trees on HN/Reddit/X |
| `--max-comments N` | cap comments per item |
| `--generic-extraction` | force the catch-all article extractor (debug) |
| `--log -` | print per-fetch audit lines to stderr |

Search-only:
| flag | when |
| -- | -- |
| `-p PROVIDER` | pick search provider |
| `-n N` | max results |
| `--after YYYY-MM-DD` / `--before YYYY-MM-DD` / `--last 7d` | date filters |
| `--site DOMAIN` / `--exclude-site DOMAIN` | domain filters (repeatable) |

## Examples

```bash
# Pull a HN story with comments → markdown to stdout
scripts/social-fetch fetch https://news.ycombinator.com/item?id=43000000

# Pull a Medium article → structured JSON
scripts/social-fetch fetch https://medium.com/@alice/some-post -f json

# Batch from a file → one .md file per URL in ./out/
scripts/social-fetch fetch -i bookmarks.txt -o out/ -j 8

# Pipe a list → JSONL stream
cat urls.txt | scripts/social-fetch fetch -f jsonl > all.jsonl

# Search the web, last 7 days, restrict to two domains
scripts/social-fetch search "vercel ai sdk" --last 7d --site vercel.com --site ai-sdk.dev

# HN search — top stories about a topic
scripts/social-fetch search "rust async" -p hackernews -n 20
```

## Timeline subcommand

```
scripts/social-fetch timeline  [flags]
  -p PROVIDER         x (default for bare handles) | linkedin
  --kind KIND         x:        all (default), tweets, replies, retweets
                      linkedin: all (default), posts, comments, reactions
  -n N                max items (default 30)
  --last DUR          sugar for --after (e.g. 7d, 24h)
                      x has a hard 7-day cap
  --after / --before  yyyy-mm-dd or RFC3339
  --expand            (LinkedIn) re-fetch each item via the post fetcher (slow)
  --no-reshares       (LinkedIn) drop reposts from the timeline
```

User identifier accepts:
- `swyx` (bare handle → x)
- `@swyx` (`@` implies x)
- `https://x.com/swyx` (auto-detected)
- `https://www.linkedin.com/in/patrickdebois/` (auto-detected)
- `patrickdebois` + `-p linkedin`

LinkedIn timelines drive the bridge through scroll/get_html cycles. Returns ~5–50 items depending on how active the user is. **Bridge required for LinkedIn — check `bridge status` first.** X timelines wrap recent-search; the 7-day cap applies and the binary pre-flights it with a clear error.

```bash
# Last 7d on X
scripts/social-fetch timeline swyx --last 7d

# LinkedIn posts only (no reshares), markdown
scripts/social-fetch timeline patrickdebois -p linkedin --kind posts --no-reshares

# LinkedIn full deep-fetch (each item gets its body + comments)
scripts/social-fetch timeline matthewskelton -p linkedin --expand -n 10
```

## Ask subcommand

```
scripts/social-fetch ask "" [flags]
  -p PROVIDER     perplexity (default), grok, openai, anthropic, google, tavily, serpapi
                  special values:
                    auto             try the built-in chain in order
                                     (perplexity → grok → openai → anthropic →
                                      google → tavily → serpapi)
                    name1,name2,…    comma-list to try in order
  -m MODEL        override the provider's default (empty = provider picks where supported)
  --last WINDOW   day | week | month | year (provider-dependent)
  --max-tokens N  cap response length
  --instructions  system-prompt-style preamble (alias: --system)
                  honored by perplexity / grok / openai / anthropic / google;
                  ignored by tavily / serpapi (no system-prompt support)
```

Returns a synthesized answer plus a numbered Sources list. Auth needed per provider — see Credentials above.

When `-p auto` or a comma-list is given, each provider in turn falls through on (a) missing API key, (b) upstream error, or (c) empty answer — the next provider gets a try, and the first non-empty response wins. The audit log records which provider answered.

```bash
scripts/social-fetch ask "what changed in the openai-microsoft revenue share clause" -p grok
scripts/social-fetch ask "best agent harness papers in the last month" -p perplexity --last month
scripts/social-fetch ask "what's the weather in NYC" -p auto                             # try the default chain
scripts/social-fetch ask "what's the weather in NYC" -p perplexity,anthropic,duckduckgo  # custom chain
```

## Listing supported sources/providers

```bash
scripts/social-fetch list
```

## Transports — bridge / headless / http / jina

The fetcher walks a **chain of transports** per platform; each fetched item carries `Extra.via` naming which one produced the body. Defaults live in the platform Go code; override per call via `SOCIAL_FETCH_CHAIN_`. Run `scripts/social-fetch hints ` for the per-platform recipe.

| transport | what it does | when needed |
| -- | -- | -- |
| **`bridge`** | drives your real, logged-in Chrome via the extension | auth-walled content (LinkedIn comments, Medium / Substack member-only posts) |
| **`headless`** | local stealth Chromium via chromedp; anonymous, JS-rendering | JS-rendered SPAs, soft anti-bot — also the article fetcher's preferred path |
| **`http`** | plain HTTP GET | static pages where JS isn't needed |
| **`jina`** | remote `r.jina.ai` service | last-resort catch-all when local methods fail |

Defaults (mirrored from code — `social-fetch hints ` is canonical):

| platform | default chain |
| -- | -- |
| article | `headless,http,bridge,jina` |
| linkedin | `headless,bridge,jina` (bridge for comments) |
| medium / substack | `bridge,http,headless,jina` (bridge for paywall) |
| twitter | `api,syndication,jina` |

### LinkedIn requires the bridge

**Setup once:** load `extensions/chrome/` (at repo root) as an unpacked Chrome extension.

**Bridge lifecycle:**
```
scripts/social-fetch bridge start          # daemonize, write PID file
scripts/social-fetch bridge status         # connected / not connected / not running
scripts/social-fetch bridge stop           # graceful SIGTERM
scripts/social-fetch bridge run            # foreground (good for `nohup` or terminals)
```

**Always check status before fetching authenticated URLs:**
```
$ scripts/social-fetch bridge status
connected           # → fetch will work
not connected       # → bridge up but extension hasn't attached (open the browser)
bridge not running on :5555   # → run `bridge start` first
```
Exit codes are `0` connected / `1` not connected / `2` bridge not running, so agents can branch on them.

**Then fetch:**
```
scripts/social-fetch fetch https://www.linkedin.com/posts/foo-activity-700…
```
The bridge tells the extension to navigate the URL in your real browser, scrapes the rendered DOM, and returns clean markdown.

URLs the LinkedIn fetcher claims: `linkedin.com/posts/…`, `linkedin.com/feed/update/urn:li:activity:…`, `linkedin.com/in/`, `linkedin.com/pulse/…`.

Errors you may see:
- `bridge unreachable` → start it (`bridge start`).
- `no extension connected` → open your browser; the extension reconnects every ~6s.

### Local browser bookmarks (`scripts/social-fetch bookmarks`)

Reads Chrome's local Bookmarks JSON and lists matching entries. Date-range filterable, multi-profile aware.

```
scripts/social-fetch bookmarks list                                # newest 100, default profile
scripts/social-fetch bookmarks list --since 2026-04-01             # bookmarked since April
scripts/social-fetch bookmarks list --folder-contains AI -n 20     # fuzzy folder match
scripts/social-fetch bookmarks list --folder "Bookmarks bar/AI"    # exact subtree (AI/, AI/papers/, …)
scripts/social-fetch bookmarks list --all-profiles -f json         # every profile, JSON
scripts/social-fetch bookmarks profiles                            # which profiles exist
```

`--platform chrome` is the default. Future platforms (Twitter bookmarks, Reddit saved posts — server-side, account-scoped) will plug in as additional values.

**Scope every call to one folder via env var**: set
`SOCIAL_FETCH_BOOKMARKS_ROOT_FOLDER="Bookmarks bar/AI"` once and the
agent's `bookmarks list` calls (CLI + MCP) only see bookmarks under
that folder + every nested subfolder. Override per-call with `--folder`.

### Ledger daemon (sandboxed / remote MCP)

`social-ledger daemon start` daemonises the SQLite ledger behind an HTTP API on port 5557. When it's running, every caller — CLI, social-fetch's auto-ingest, MCP read tools — routes through HTTP instead of openi

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [jedi4ever](https://github.com/jedi4ever)
- **Source:** [jedi4ever/social-skills](https://github.com/jedi4ever/social-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-jedi4ever-social-skills-social-fetch
- Seller: https://agentstack.voostack.com/s/jedi4ever
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.