Social Fetch

social-fetch skill

Wraps the social-fetch Go binary at scripts/social-fetch (relative to this skill).

Trust the CLI. It is the authority for every fetch and search supported by this skill. Always shell out to scripts/social-fetch — never reimplement fetching with WebFetch, curl, custom parsers, or hand-rolled API calls, even if the binary returns empty results or an error you find surprising. If a fetch comes back empty, surface that to the user and (if appropriate) re-run with --log - to see audit lines, but do not try to "fix it" by going around the CLI.

Before invoking any provider, run scripts/social-fetch list to see which platforms are usable in this environment. Each provider is tagged with one of three states:

• [ok] — fully configured, fire away
• [!auth] — required env var not set; the row's suffix names which one (e.g. → missing BRAVE_API_KEY). Do not use these providers — pick a different one in the same category. Suggesting the user configure the missing key is fine only if they explicitly ask how; don't proactively nag them about every unset key.
• [bridge] — needs the local browser bridge (LinkedIn / Medium / Substack / linkedin search & timeline). Only use after scripts/social-fetch bridge status reports connected.

The auto provider chains (-p auto) already skip unconfigured providers, so they're safe — but when an explicit provider name is needed (e.g. the user asks "search Twitter"), check list first to confirm [ok] status before invoking. For the MCP shape see social_fetch_list_providers — same data, structured as {name, status, missing} per category.

For platform-specific quirks, run scripts/social-fetch hints (no argument — dumps every platform's hints in one shot) before a search/fetch you haven't done recently. Captures things like "X recent search caps at 7 days strictly", "LinkedIn temp-bans accounts that scrape too fast", "Reddit anonymous search has worse relevance than tavily site:reddit.com". Pass a specific platform name to scope the output (e.g. hints x).

Subcommands

scripts/social-fetch fetch     [...]    [flags]
scripts/social-fetch search   ""           [flags]
scripts/social-fetch timeline        [flags]   recent activity for a user (X / LinkedIn)
scripts/social-fetch ask      ""        [flags]   grounded answer engine (perplexity / grok / openai / anthropic / gemini / tavily / serpapi)
scripts/social-fetch research ""        [flags]   EXPERIMENTAL — multi-angle research (decompose → parallel fan-out → synthesize)
scripts/social-fetch bridge   {start|stop|status|run}
scripts/social-fetch bookmarks {list|profiles}              local browser bookmarks (chrome today; --platform NAME)
scripts/social-fetch hints    []                  per-platform quirks, rate limits, gotchas (x / linkedin / reddit / ...)

Run scripts/social-fetch --help for the full reference. Output defaults to markdown; pass -f json or -f jsonl for structured input to other tools.

Credentials (.env support)

Provider keys (X_API_KEY, X_API_SECRET, TAVILY_API_KEY, SERPAPI_KEY, BRAVE_API_KEY, PERPLEXITY_API_KEY, XAI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY/GOOGLE_API_KEY/GOOGLE_CSE_ID, YOUTUBE_API_KEY, BLUESKY_HANDLE/BLUESKY_APP_PASSWORD, GITHUB_TOKEN) and routing hints (HTML2MD_PROVIDER, HTML2MD_READER, YOUTUBE_TRANSCRIPT_PROVIDER, TAVILY_TOPIC) can be set in the shell or placed in a .env file. At startup the binary loads, in order:

1. ./.env (current working directory)
2. /.env (sits next to the installed binary — typically ~/.claude/skills/social-fetch/.env)

Already-exported shell vars always win over file entries.

Decision rules

• One URL → fetch it. scripts/social-fetch fetch auto-detects the source from the host (HN, Reddit, GitHub, X, RSS, or generic article).
• Images / diagrams in the post → every fetched item carries a Media list (LinkedIn post photos, X/Twitter media, Medium/Substack figures, YouTube thumbnails, generic article images). The MCP social_fetch_fetch envelope surfaces this as a media[] array of {url, type, alt} entries. When the user asks "what's on the picture / diagram / screenshot in this post", the agent's vision-capable Read tool (Claude Code / Claude Desktop) can read each media[].url directly — no extra fetch call. Empty alt usually means the image is worth looking at; populated alt is the author's caption. Same data is in the ## Media section of the markdown content.
• A list of URLs → batch. Pipe via stdin (cat urls.txt | scripts/social-fetch fetch) or use -i FILE. Add -j 8 for parallel fetches; output stays in input order.
• Save to disk → -o FILE for one file, -o DIR/ for one file per URL.
• A user's recent posts → timeline. scripts/social-fetch timeline [-p x|linkedin] [--kind ...] [-n N]. Auto-detects the provider from URL; default for bare handles is X. See "Timeline subcommand" below.
• A grounded question → ask. scripts/social-fetch ask "" -p perplexity|grok|openai|anthropic|gemini|tavily|serpapi. Returns synthesized answer + sources. Use this only when the user explicitly wants a synthesized answer; for raw documents use fetch or search.
• A multi-angle research question → research (EXPERIMENTAL). scripts/social-fetch research "" --max-angles 5 --jobs 4. Decomposes into 3-8 angles, fans out parallel queries, synthesizes a final answer with citations. Use when you'd otherwise issue 4-8 manual queries. Costs roughly 2 LLM calls + N tool calls per question; use ask for simple lookups instead.
• A query → search. Pick the provider that matches the user's intent. -p auto walks perplexity → tavily → brave → serpapi → duckduckgo; comma-lists like -p tavily,duckduckgo define a custom fallback order. Each falls through on missing key / error / 0 results.
• "search the web" / unspecified → duckduckgo (no auth)
• "search Brave" / privacy-focused web → brave (needs BRAVE_API_KEY; native --last 7d via freshness)
• "high-quality web search for AI agents" → tavily (needs TAVILY_API_KEY)
• "Perplexity index without synthesis" → perplexity (needs PERPLEXITY_API_KEY; same key as ask -p perplexity, but cheaper since no LLM tokens)
• "LinkedIn posts about " → linkedin (requires the browser bridge + a logged-in LinkedIn session; up to 50 results per query via scroll-to-bottom + wheel-event lazy-load). Use sparingly. LinkedIn aggressively rate-limits accounts that scrape — running this back-to-back will get the user temp-banned. Prefer tavily / perplexity / serpapi for general "who's writing about X" questions, and only reach for -p linkedin when LinkedIn-specific posts are explicitly the goal.
• "search Bluesky" → bluesky (no auth, native date filter)
• "search arXiv" / academic papers → arxiv (no auth, sorted newest-first)
• "search HN" → hackernews
• "search Reddit" → reddit (no auth, public search.json; rate-limited per IP)
• "search Twitter/X" → x (needs X_API_KEY + X_API_SECRET)
• "search via Google" → serpapi (needs SERPAPI_KEY)
• "search YouTube" → youtube (needs YOUTUBE_API_KEY; supports --last 7d / --after natively, dates are strict)

Flags worth remembering

| flag | when | | -- | -- | | -f markdown\|json\|jsonl | format (default markdown) | | -o PATH | stdout / FILE / DIR/ | | -i FILE | URLs file (- = stdin; auto-detected when piped) | | -j N | parallel workers for batch fetch | | --no-comments | skip comment trees on HN/Reddit/X | | --max-comments N | cap comments per item | | --generic-extraction | force the catch-all article extractor (debug) | | --log - | print per-fetch audit lines to stderr |

Search-only: | flag | when | | -- | -- | | -p PROVIDER | pick search provider | | -n N | max results | | --after YYYY-MM-DD / --before YYYY-MM-DD / --last 7d | date filters | | --site DOMAIN / --exclude-site DOMAIN | domain filters (repeatable) |

Examples

# Pull a HN story with comments → markdown to stdout
scripts/social-fetch fetch https://news.ycombinator.com/item?id=43000000

# Pull a Medium article → structured JSON
scripts/social-fetch fetch https://medium.com/@alice/some-post -f json

# Batch from a file → one .md file per URL in ./out/
scripts/social-fetch fetch -i bookmarks.txt -o out/ -j 8

# Pipe a list → JSONL stream
cat urls.txt | scripts/social-fetch fetch -f jsonl > all.jsonl

# Search the web, last 7 days, restrict to two domains
scripts/social-fetch search "vercel ai sdk" --last 7d --site vercel.com --site ai-sdk.dev

# HN search — top stories about a topic
scripts/social-fetch search "rust async" -p hackernews -n 20

Timeline subcommand

scripts/social-fetch timeline  [flags]
  -p PROVIDER         x (default for bare handles) | linkedin
  --kind KIND         x:        all (default), tweets, replies, retweets
                      linkedin: all (default), posts, comments, reactions
  -n N                max items (default 30)
  --last DUR          sugar for --after (e.g. 7d, 24h)
                      x has a hard 7-day cap
  --after / --before  yyyy-mm-dd or RFC3339
  --expand            (LinkedIn) re-fetch each item via the post fetcher (slow)
  --no-reshares       (LinkedIn) drop reposts from the timeline

User identifier accepts:

• swyx (bare handle → x)
• @swyx (@ implies x)
• https://x.com/swyx (auto-detected)
• https://www.linkedin.com/in/patrickdebois/ (auto-detected)
• patrickdebois + -p linkedin

LinkedIn timelines drive the bridge through scroll/get_html cycles. Returns ~5–50 items depending on how active the user is. Bridge required for LinkedIn — check bridge status first. X timelines wrap recent-search; the 7-day cap applies and the binary pre-flights it with a clear error.

# Last 7d on X
scripts/social-fetch timeline swyx --last 7d

# LinkedIn posts only (no reshares), markdown
scripts/social-fetch timeline patrickdebois -p linkedin --kind posts --no-reshares

# LinkedIn full deep-fetch (each item gets its body + comments)
scripts/social-fetch timeline matthewskelton -p linkedin --expand -n 10

Ask subcommand

scripts/social-fetch ask "" [flags]
  -p PROVIDER     perplexity (default), grok, openai, anthropic, google, tavily, serpapi
                  special values:
                    auto             try the built-in chain in order
                                     (perplexity → grok → openai → anthropic →
                                      google → tavily → serpapi)
                    name1,name2,…    comma-list to try in order
  -m MODEL        override the provider's default (empty = provider picks where supported)
  --last WINDOW   day | week | month | year (provider-dependent)
  --max-tokens N  cap response length
  --instructions  system-prompt-style preamble (alias: --system)
                  honored by perplexity / grok / openai / anthropic / google;
                  ignored by tavily / serpapi (no system-prompt support)

Returns a synthesized answer plus a numbered Sources list. Auth needed per provider — see Credentials above.

When -p auto or a comma-list is given, each provider in turn falls through on (a) missing API key, (b) upstream error, or (c) empty answer — the next provider gets a try, and the first non-empty response wins. The audit log records which provider answered.

scripts/social-fetch ask "what changed in the openai-microsoft revenue share clause" -p grok
scripts/social-fetch ask "best agent harness papers in the last month" -p perplexity --last month
scripts/social-fetch ask "what's the weather in NYC" -p auto                             # try the default chain
scripts/social-fetch ask "what's the weather in NYC" -p perplexity,anthropic,duckduckgo  # custom chain

Listing supported sources/providers

scripts/social-fetch list

Transports — bridge / headless / http / jina

The fetcher walks a chain of transports per platform; each fetched item carries Extra.via naming which one produced the body. Defaults live in the platform Go code; override per call via SOCIAL_FETCH_CHAIN_. Run scripts/social-fetch hints for the per-platform recipe.

| transport | what it does | when needed | | -- | -- | -- | | bridge | drives your real, logged-in Chrome via the extension | auth-walled content (LinkedIn comments, Medium / Substack member-only posts) | | headless | local stealth Chromium via chromedp; anonymous, JS-rendering | JS-rendered SPAs, soft anti-bot — also the article fetcher's preferred path | | http | plain HTTP GET | static pages where JS isn't needed | | jina | remote r.jina.ai service | last-resort catch-all when local methods fail |

Defaults (mirrored from code — social-fetch hints is canonical):

| platform | default chain | | -- | -- | | article | headless,http,bridge,jina | | linkedin | headless,bridge,jina (bridge for comments) | | medium / substack | bridge,http,headless,jina (bridge for paywall) | | twitter | api,syndication,jina |

LinkedIn requires the bridge

Setup once: load extensions/chrome/ (at repo root) as an unpacked Chrome extension.

Bridge lifecycle:

scripts/social-fetch bridge start          # daemonize, write PID file
scripts/social-fetch bridge status         # connected / not connected / not running
scripts/social-fetch bridge stop           # graceful SIGTERM
scripts/social-fetch bridge run            # foreground (good for `nohup` or terminals)

Always check status before fetching authenticated URLs:

$ scripts/social-fetch bridge status
connected           # → fetch will work
not connected       # → bridge up but extension hasn't attached (open the browser)
bridge not running on :5555   # → run `bridge start` first

Exit codes are 0 connected / 1 not connected / 2 bridge not running, so agents can branch on them.

Then fetch:

scripts/social-fetch fetch https://www.linkedin.com/posts/foo-activity-700…

The bridge tells the extension to navigate the URL in your real browser, scrapes the rendered DOM, and returns clean markdown.

URLs the LinkedIn fetcher claims: linkedin.com/posts/…, linkedin.com/feed/update/urn:li:activity:…, linkedin.com/in/, linkedin.com/pulse/….

Errors you may see:

• bridge unreachable → start it (bridge start).
• no extension connected → open your browser; the extension reconnects every ~6s.

Local browser bookmarks (scripts/social-fetch bookmarks)

Reads Chrome's local Bookmarks JSON and lists matching entries. Date-range filterable, multi-profile aware.

scripts/social-fetch bookmarks list                                # newest 100, default profile
scripts/social-fetch bookmarks list --since 2026-04-01             # bookmarked since April
scripts/social-fetch bookmarks list --folder-contains AI -n 20     # fuzzy folder match
scripts/social-fetch bookmarks list --folder "Bookmarks bar/AI"    # exact subtree (AI/, AI/papers/, …)
scripts/social-fetch bookmarks list --all-profiles -f json         # every profile, JSON
scripts/social-fetch bookmarks profiles                            # which profiles exist

--platform chrome is the default. Future platforms (Twitter bookmarks, Reddit saved posts — server-side, account-scoped) will plug in as additional values.

Scope every call to one folder via env var: set SOCIAL_FETCH_BOOKMARKS_ROOT_FOLDER="Bookmarks bar/AI" once and the agent's bookmarks list calls (CLI + MCP) only see bookmarks under that folder + every nested subfolder. Override per-call with --folder.

Ledger daemon (sandboxed / remote MCP)

social-ledger daemon start daemonises the SQLite ledger behind an HTTP API on port 5557. When it's running, every caller — CLI, social-fetch's auto-ingest, MCP read tools — routes through HTTP instead of openi

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jedi4ever
• Source: jedi4ever/social-skills
• License: MIT

Install and usage instructions live in the source repository linked above.