# Run Pulse > Watchlist-driven intelligence pipeline, crawl LinkedIn/X/RSS/email via composed leaves, classify and domain-tag in parallel sub-agents, then synthesize one neutral signals brief. The brief is the public core; your relevance + delivery layer attaches at the seam. - **Type:** Skill - **Install:** `agentstack add skill-thangnguyenworkspace-industry-pulse-run-pulse` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [thangnguyenworkspace](https://agentstack.voostack.com/s/thangnguyenworkspace) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [thangnguyenworkspace](https://github.com/thangnguyenworkspace) - **Source:** https://github.com/thangnguyenworkspace/industry-pulse/tree/main/.claude/skills/run-pulse ## Install ```sh agentstack add skill-thangnguyenworkspace-industry-pulse-run-pulse ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Run Pulse **Argument:** $ARGUMENTS (optional, all flags optional with defaults; see Runtime Inputs) Crawl four source-types, classify and domain-tag each in parallel classifier sub-agents, then synthesize one neutral **signals brief**: the main thread authors a signals-only, third-person brief that states what happened across your watchlist and why it matters. The brief is the reusable core's final artifact and the handoff point, your own relevance layer (and any delivery) attaches downstream at the seam (§3.0 Step 5). --- ## Preamble ### Runtime Inputs Parse `$ARGUMENTS` for these optional flags; apply defaults when absent: ``` --days={N} [positive integer; default 1; time window passed to every crawl leaf] --linkedin-max-posts={N} [positive integer; default 3; per-URL cap for the LinkedIn leaf] --email-max-per-sender={N} [positive integer; default 10; per-sender cap for the email leaf] --x-max-per-handle={N} [positive integer; default 5; per-handle cap for the X leaf] --rss-max-items-per-feed={N} [positive integer; default 15; per-feed cap for the RSS leaf] ``` All flags optional. Invalid values (non-positive integers, malformed) → halt with the matching §6.0 row. ### Global References All paths are repo-relative (the skill runs with the repo root as the working directory) and are the configuration surface, repoint any of them to your own layout. The `config/` inputs are the working copies you create from the shipped `.example` templates (copy `config/profile.example.md` → `config/profile.md`, each `config/watchlists/{lane}-watchlist.example.md` → `config/watchlists/{lane}-watchlist.md`); `output/` is gitignored and regenerable per run. ``` PROFILE_PATH = config/profile.md PROFILE_DOMAIN_ENUM = config/profile.md §3.0 (the domain enum + scope/boundary classifier contract, the tagging vocabulary) LINKEDIN_WATCHLIST_PATH = config/watchlists/linkedin-watchlist.md EMAIL_WATCHLIST_PATH = config/watchlists/email-watchlist.md X_WATCHLIST_PATH = config/watchlists/x-watchlist.md RSS_WATCHLIST_PATH = config/watchlists/rss-watchlist.md RAW_DIR = output/raw/ (gitignored; per-source-type subdirs) TAGGED_DIR = output/tagged/ (gitignored; per-run tagged-output files) REPORTS_DIR = output/reports/ (the briefs; gitignored by default, repoint or un-ignore to keep them) TAGGED_SCHEMA = schemas/schema-pulse-tagged-output.md REPORT_SCHEMA = schemas/schema-pulse-report.md RUN_DIR = output/reports/pulse-report-{report_date}/ (per-run folder) BRIEF_PATH = {RUN_DIR}pulse-report-{report_date}.md (the neutral signals brief, the public artifact and the handoff to your relevance layer) L1_TOKEN_BUDGET = 25000 (per-RSS-worker raw-token budget; drives the RSS volume auto-scaler bin count) L1_CONCURRENCY_CAP = 14 (max parallel classifier workers; tune to your harness's concurrency limit) SOURCES_PER_WORKER = 30 (per-bin source cap for the count-sharded X + LinkedIn lanes; a dial, not a wall, dial down for wide --days windows, a raised per-source post cap, or the heavier LinkedIn lane. Daily defaults assumed: 5 posts/X-handle, 3/LinkedIn-URL, 1-day window.) ``` This skill composes four pure-primitive leaves (`/research-crawl-linkedin-posts`, `/research-crawl-email`, `/research-crawl-x-posts`, `/research-crawl-rss`); it never calls `mcp__apify__*` or `mcp__claude_ai_Gmail__*` directly. Each leaf owns its API mechanics and pitfalls; context isolation is caller-owned. Architecture: L0 leaves crawl → classifier sub-agents classify + domain-tag (email = one worker; X + LinkedIn count-sharded; RSS volume-auto-scaled) → the main thread synthesizes the neutral signals brief. The heavy, parallelizable reading lives in the sub-agents; the main thread holds only the tagged signal files and the synthesis. The public core ends at the brief, your relevance and delivery layer attaches at the seam documented in §3.0 Step 5. --- ## 1.0 Context Capture ### Step 1: Read the profile Read `PROFILE_PATH`. Capture `version:` for provenance. Capture §1.0 North Star + Current Bets, §2.0 Skill Targets, §3.0 Domain Interests (`PROFILE_DOMAIN_ENUM`, the tagging vocabulary the classifiers run against), §4.0 Lens Filters, the full lens carried into classification and the synthesis. The profile is your lens; replace its contents to retarget the whole pipeline at a different reader. ### Step 2: Read all four watchlists Read each watchlist; extract the per-source-type source list and capture its entry count. Flag a source-type empty when its watchlist has no entries (or frontmatter `total_entries: 0`): - **LinkedIn** (`LINKEDIN_WATCHLIST_PATH`): the canonical-URL column of every §3.0 entry → ordered list of LinkedIn URLs. - **Email** (`EMAIL_WATCHLIST_PATH`): the URL column (the literal email address) of every §3.0 entry → ordered list of sender addresses. - **X** (`X_WATCHLIST_PATH`): the URL column (`https://x.com/{handle}`) of every §3.0 entry → extract the **bare handle** from each URL (strip scheme/host + any leading `@`) → ordered list of bare handles. The X leaf takes bare handles, not URLs. - **RSS** (`RSS_WATCHLIST_PATH`): the `URL` + `Entity Type` columns of every §3.0 Watchlist Entries row → ordered list of `{feed_url, class}` pairs, mapping `URL`→`feed_url` and `Entity Type`→`class` (`Deep-Content Feed`→`deep-content`, `News-Firehose Feed`→`news-firehose`). If all four watchlists are empty, halt with §6.0 (`all-watchlists-empty`). ### Step 3: RSS crawl-first (volume probe) Skip when the RSS watchlist is empty. The RSS leaf writes feed bodies to disk via a Python subprocess and returns only a manifest, running it here lands only per-feed counts + token estimates in context, letting the auto-scaler size dispatch from real volume. Set `rss_raw_dir = output/raw/rss/rss-raw-{report_date}/`. If it already exists, remove it first (`rm -rf {rss_raw_dir}`): it can only be residue from an earlier same-day run, a completed same-day run is caught separately by the `report-already-exists` halt at Step 5, and the dir is gitignored and regenerable, so clearing it prevents stale feed files from a halted run bleeding into this run's binning. Invoke `/research-crawl-rss` via the Skill tool with: ``` --feed-urls=[] --raw-output-dir={rss_raw_dir} --max-items-per-feed={--rss-max-items-per-feed} --days={--days} ``` Read the returned summary + the `manifest.json` at `manifest_path`. Capture per-feed `{feed_url, class, items_emitted, est_tokens, crawl_status, raw_output_path}`. Do **not** read the per-feed body files, they stay on disk for the RSS classifier workers. Aggregate `crawl_status: failed` → halt with §6.0 (`rss-leaf-failed`). ### Step 4: Bin the sharded lanes → classifier worker counts (auto-scaler) Size every lane's classifier dispatch here, never hard-coded. RSS bins by measured token volume (crawl-first probe at Step 3); X and LinkedIn bin by source count (the paid Apify lanes cannot free-probe volume, so they shard on the known watchlist count before crawling); email is always one worker. **RSS: volume bins** (skip when RSS empty, `N_rss = 0`): 1. Keep feeds with per-feed `crawl_status` in `{clean, empty}`; a per-feed `failed` drops from downstream (surfaced in §5.0). 2. Partition kept feeds by `class`: a `deep-content` group and a `news-firehose` group. The two classes load on different axes (deep = low-count/high-token, news = high-count/low-token), so they never share a bin. 3. Within each group, greedily pack feeds into bins so each bin's summed `est_tokens` stays ≤ `L1_TOKEN_BUDGET`. A single feed whose `est_tokens` alone exceeds the budget gets its own bin (the per-feed `--rss-max-items-per-feed` cap already bounds it). News feeds packed together keep overlapping coverage in one worker's context for consistent tagging. 4. `N_rss` = total RSS bin count. Record each bin's feed list (the bin's `raw_output_path`s) for §3.0 dispatch. **X + LinkedIn: count bins** (per lane, skip when that lane is empty): 5. For X: `N_x_bins = ceil(x_handle_count / SOURCES_PER_WORKER)`; split the ordered bare-handle list into `N_x_bins` contiguous near-even slices (e.g., 51 handles at 30 → 2 bins of 26 + 25). For LinkedIn: `N_linkedin_bins = ceil(linkedin_source_count / SOURCES_PER_WORKER)`; split the ordered URL list the same way. Record each bin's source slice for §3.0 dispatch. A lane at or under `SOURCES_PER_WORKER` yields exactly one bin, the unsharded case falls out for free. **Email + concurrency cap:** 6. `N_email` = 1 when email is non-empty, else 0 (email is never sharded, small sender count). 7. `total_workers = N_x_bins + N_linkedin_bins + N_email + N_rss`. If `total_workers > L1_CONCURRENCY_CAP`, cap-merge, merge the smallest bins (start with the lane carrying the most bins, then RSS news bins) until `total_workers ≤ L1_CONCURRENCY_CAP`, and log the cap-hit in §5.0 (a signal to raise the cap or split a watchlist). ### Step 5: Compute paths + cost estimate - `report_date` = today (ISO). - `RUN_DIR` = `output/reports/pulse-report-{report_date}/`. If it exists, halt with §6.0 (`report-already-exists`). - `BRIEF_PATH` = `{RUN_DIR}pulse-report-{report_date}.md`. - `tagged_dir` = `output/tagged/tagged-{report_date}/`. - Per-source-type raw dirs: `linkedin_raw_dir`, `email_raw_dir`, `x_raw_dir` = `output/raw/{type}/{type}-raw-{report_date}/` (the RSS dir was created at Step 3). - Cost estimate (paid crawls only; email + RSS are free): LinkedIn `linkedin_source_count × --linkedin-max-posts × $1.50 / 1000`; X `x_handle_count × --x-max-per-handle × $0.0004`. Realistic ≈ 30-60% of worst case for short `--days` windows. No scaffold-output-dirs step, each leaf, each classifier worker, and the brief writer `mkdir -p`s its own dated output subdir. --- ## 2.0 Plan & Confirm ### Pre-flight: assert crawl-tool availability Before presenting the plan (and before any paid dispatch at §3.0), assert the external crawl tools are connected. This is a cheap tool-namespace presence check, not an API call. It fails fast here rather than deep inside the classifier workers, where a disconnected Apify silently ships a LinkedIn + X-less brief with no warning, discovered only after the paid dispatch returns: - `mcp__apify__*`, required by the LinkedIn + X leaves. Assert only when LinkedIn or X is non-empty. Absent → halt with §6.0 (`apify-unavailable`). Paid sources are load-bearing; this halt is unconditional. - `mcp__claude_ai_Gmail__*`, required by the email leaf. Assert only when email is non-empty. Absent → the email source-type can drop, but treat the drop as a decision, not an FYI: a lapsed connector silently ships a brief missing the entire email layer. Surface the absence as an explicit reconnect-or-proceed gate folded into the §2.0 plan-confirm pause (see Plan below). Reconnecting is a quick action and no money has been spent at this point, so a silent default-drop forfeits a cheap recovery. ### Plan Present the planned scope: ``` | Field | Value | |---|---| | Days window | {--days} | | LinkedIn | {linkedin_source_count} URLs × max {--linkedin-max-posts}/URL → {N_linkedin_bins} worker(s) (or "skipped, empty") | | Email | {email_sender_count} senders × max {--email-max-per-sender}/sender (or "skipped, empty", or "dropped, Gmail not connected") | | X | {x_handle_count} handles × max {--x-max-per-handle}/handle → {N_x_bins} worker(s) (or "skipped, empty") | | RSS | {rss_feed_count} feeds → {rss_in_window_items} items in window → {N_rss} worker(s) (or "skipped, empty") | | Classifier workers | {total_workers} total ({N_x_bins} X + {N_linkedin_bins} LinkedIn + {N_email} email + {N_rss} RSS); cap {L1_CONCURRENCY_CAP}, cap-merged when a cap-hit fired | | Cost estimate (paid) | LinkedIn ${linkedin_cost} + X ${x_cost} (~30-60% realistic); email + RSS free | | Profile version | v{profile_version_consumed} | | Brief destination | {BRIEF_PATH} | ``` RSS is already crawled (free) at §1.0 Step 3, so the plan shows its real volume + computed worker count. The paid LinkedIn + X crawls run only after approval, at §3.0. **Gmail reconnect-or-proceed gate.** When the pre-flight found `mcp__claude_ai_Gmail__*` absent and the email watchlist is non-empty, the plan's Email row reads `dropped, Gmail not connected` and the approval pause carries two options: (1) reconnect the Gmail MCP, then re-run the namespace check and proceed with email included; (2) proceed without email this run. Resolve the choice at this same pause, do not dispatch until it is answered. Wait for user approval before §3.0. --- ## 3.0 Execute ### Step 1: Dispatch all classifier sub-agents in one message Dispatch every classifier worker in a **single message** via parallel `Agent({...})` calls, one per LinkedIn bin (`j = 1..N_linkedin_bins`), one email worker, one per X bin (`i = 1..N_x_bins`), plus one per RSS bin (`k = 1..N_rss`). Sequential dispatch loses the parallelism; author all calls in the same response. Skip any empty source-type. ``` Agent({subagent_type: "general-purpose", prompt: }) # one per LinkedIn bin j Agent({subagent_type: "general-purpose", prompt: }) # if email non-empty Agent({subagent_type: "general-purpose", prompt: }) # one per X bin i Agent({subagent_type: "general-purpose", prompt: }) # one per RSS bin k ``` Two worker shapes share one classify-tag-write contract, differing only in how they obtain raw: - **Crawl-and-classify** (LinkedIn / email / X): invoke the source-type's pure-primitive leaf in-worker on the worker's source slice (isolating the Apify / Gmail raw dataset out of the main thread), read the per-source raw files, then classify + domain-tag. X and LinkedIn shard into one worker per count-bin (§1.0 Step 4); email is a single worker over all senders. - **RSS classifier** (one per bin): the RSS raw is already on disk from §1.0 Step 3; read only the assigned bin's per-feed files, then classify + domain-tag. **SHARED CLASSIFIER CONTRACT**: embed verbatim in every worker prompt (interpolate `{source_type}`, `{tagged_path}`, `{report_date}`, `{--days}`, `{profile_version}`, and the profile §3.0 enum + §4.0 drop criteria): ``` You are the {source_type} signals classifier-tagger. Classify and domain-tag every item from one source-type, preserve each source's framing, and write ONE tagged-output file. You do NOT assemble the brief and you do NOT dedupe. # Classify each item: bucket = Authored / Reposted / Mentioned / Drop - Authored, the watchlist source originated the item. - Reposted, the source reshared another author's item verbatim. A quote-post with substantive commentary counts as Authored, not Reposted. - Mentioned, the source is referenced or quoted inside another's item; surface only when the mention adds signal. - Drop, none of the above, OR matches a profile drop criterion: . Drop items are excluded from the file. # Domain-tag each survivor: keys from the profile's domain enum Tag each survivor with one or more domain keys (an item may carry a primary + secondary key). Apply the scope + boundary cross-routing rules verbatim: . # Preserve framing: do NOT flatten, do NOT dedupe Write each survivor's compact_summary to capture THIS source's own framing, the angle it took + notable wording, in 1-3 sentences, not a neutral flattened fact. Use no em dashes in compact_summary prose (commas, parentheses, or sentence breaks instead) so none flow into the brief. When multiple of your sources cover the same story, keep each as its own item (repetition is signal; differing framings are insight). Never collapse duplicates. # Write the tagged-output file (conform to schema-pulse-tagged-output) mkdir -p the tagged dir, then write to: {tagged_path} - Frontmatter: type "Pulse Tagged Output"; schema pulse-tagged-output; created/updated/tagged_date {report_date}; tags [pu … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [thangnguyenworkspace](https://github.com/thangnguyenworkspace) - **Source:** [thangnguyenworkspace/industry-pulse](https://github.com/thangnguyenworkspace/industry-pulse) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** yes - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-thangnguyenworkspace-industry-pulse-run-pulse - Seller: https://agentstack.voostack.com/s/thangnguyenworkspace - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.