Run Pulse

Run Pulse

Argument: $ARGUMENTS (optional, all flags optional with defaults; see Runtime Inputs)

Crawl four source-types, classify and domain-tag each in parallel classifier sub-agents, then synthesize one neutral signals brief: the main thread authors a signals-only, third-person brief that states what happened across your watchlist and why it matters. The brief is the reusable core's final artifact and the handoff point, your own relevance layer (and any delivery) attaches downstream at the seam (§3.0 Step 5).

---

Preamble

Runtime Inputs

Parse $ARGUMENTS for these optional flags; apply defaults when absent:

--days={N}                     [positive integer; default 1; time window passed to every crawl leaf]
--linkedin-max-posts={N}       [positive integer; default 3; per-URL cap for the LinkedIn leaf]
--email-max-per-sender={N}     [positive integer; default 10; per-sender cap for the email leaf]
--x-max-per-handle={N}         [positive integer; default 5; per-handle cap for the X leaf]
--rss-max-items-per-feed={N}   [positive integer; default 15; per-feed cap for the RSS leaf]

All flags optional. Invalid values (non-positive integers, malformed) → halt with the matching §6.0 row.

Global References

All paths are repo-relative (the skill runs with the repo root as the working directory) and are the configuration surface, repoint any of them to your own layout. The config/ inputs are the working copies you create from the shipped .example templates (copy config/profile.example.md → config/profile.md, each config/watchlists/{lane}-watchlist.example.md → config/watchlists/{lane}-watchlist.md); output/ is gitignored and regenerable per run.

PROFILE_PATH               = config/profile.md
PROFILE_DOMAIN_ENUM        = config/profile.md §3.0   (the domain enum + scope/boundary classifier contract, the tagging vocabulary)
LINKEDIN_WATCHLIST_PATH    = config/watchlists/linkedin-watchlist.md
EMAIL_WATCHLIST_PATH       = config/watchlists/email-watchlist.md
X_WATCHLIST_PATH           = config/watchlists/x-watchlist.md
RSS_WATCHLIST_PATH         = config/watchlists/rss-watchlist.md
RAW_DIR                    = output/raw/      (gitignored; per-source-type subdirs)
TAGGED_DIR                 = output/tagged/   (gitignored; per-run tagged-output files)
REPORTS_DIR                = output/reports/  (the briefs; gitignored by default, repoint or un-ignore to keep them)
TAGGED_SCHEMA              = schemas/schema-pulse-tagged-output.md
REPORT_SCHEMA              = schemas/schema-pulse-report.md
RUN_DIR                    = output/reports/pulse-report-{report_date}/   (per-run folder)
BRIEF_PATH                 = {RUN_DIR}pulse-report-{report_date}.md   (the neutral signals brief, the public artifact and the handoff to your relevance layer)
L1_TOKEN_BUDGET            = 25000   (per-RSS-worker raw-token budget; drives the RSS volume auto-scaler bin count)
L1_CONCURRENCY_CAP         = 14      (max parallel classifier workers; tune to your harness's concurrency limit)
SOURCES_PER_WORKER         = 30      (per-bin source cap for the count-sharded X + LinkedIn lanes; a dial, not a wall, dial down for wide --days windows, a raised per-source post cap, or the heavier LinkedIn lane. Daily defaults assumed: 5 posts/X-handle, 3/LinkedIn-URL, 1-day window.)

This skill composes four pure-primitive leaves (/research-crawl-linkedin-posts, /research-crawl-email, /research-crawl-x-posts, /research-crawl-rss); it never calls mcp__apify__* or mcp__claude_ai_Gmail__* directly. Each leaf owns its API mechanics and pitfalls; context isolation is caller-owned. Architecture: L0 leaves crawl → classifier sub-agents classify + domain-tag (email = one worker; X + LinkedIn count-sharded; RSS volume-auto-scaled) → the main thread synthesizes the neutral signals brief. The heavy, parallelizable reading lives in the sub-agents; the main thread holds only the tagged signal files and the synthesis. The public core ends at the brief, your relevance and delivery layer attaches at the seam documented in §3.0 Step 5.

---

1.0 Context Capture

Step 1: Read the profile

Read PROFILE_PATH. Capture version: for provenance. Capture §1.0 North Star + Current Bets, §2.0 Skill Targets, §3.0 Domain Interests (PROFILE_DOMAIN_ENUM, the tagging vocabulary the classifiers run against), §4.0 Lens Filters, the full lens carried into classification and the synthesis. The profile is your lens; replace its contents to retarget the whole pipeline at a different reader.

Step 2: Read all four watchlists

Read each watchlist; extract the per-source-type source list and capture its entry count. Flag a source-type empty when its watchlist has no entries (or frontmatter total_entries: 0):

• LinkedIn (LINKEDIN_WATCHLIST_PATH): the canonical-URL column of every §3.0 entry → ordered list of LinkedIn URLs.
• Email (EMAIL_WATCHLIST_PATH): the URL column (the literal email address) of every §3.0 entry → ordered list of sender addresses.
• X (X_WATCHLIST_PATH): the URL column (https://x.com/{handle}) of every §3.0 entry → extract the bare handle from each URL (strip scheme/host + any leading @) → ordered list of bare handles. The X leaf takes bare handles, not URLs.
• RSS (RSS_WATCHLIST_PATH): the URL + Entity Type columns of every §3.0 Watchlist Entries row → ordered list of {feed_url, class} pairs, mapping URL→feed_url and Entity Type→class (Deep-Content Feed→deep-content, News-Firehose Feed→news-firehose).

If all four watchlists are empty, halt with §6.0 (all-watchlists-empty).

Step 3: RSS crawl-first (volume probe)

Skip when the RSS watchlist is empty. The RSS leaf writes feed bodies to disk via a Python subprocess and returns only a manifest, running it here lands only per-feed counts + token estimates in context, letting the auto-scaler size dispatch from real volume.

Set rss_raw_dir = output/raw/rss/rss-raw-{report_date}/. If it already exists, remove it first (rm -rf {rss_raw_dir}): it can only be residue from an earlier same-day run, a completed same-day run is caught separately by the report-already-exists halt at Step 5, and the dir is gitignored and regenerable, so clearing it prevents stale feed files from a halted run bleeding into this run's binning. Invoke /research-crawl-rss via the Skill tool with:

--feed-urls=[]
--raw-output-dir={rss_raw_dir}
--max-items-per-feed={--rss-max-items-per-feed}
--days={--days}

Read the returned summary + the manifest.json at manifest_path. Capture per-feed {feed_url, class, items_emitted, est_tokens, crawl_status, raw_output_path}. Do not read the per-feed body files, they stay on disk for the RSS classifier workers. Aggregate crawl_status: failed → halt with §6.0 (rss-leaf-failed).

Step 4: Bin the sharded lanes → classifier worker counts (auto-scaler)

Size every lane's classifier dispatch here, never hard-coded. RSS bins by measured token volume (crawl-first probe at Step 3); X and LinkedIn bin by source count (the paid Apify lanes cannot free-probe volume, so they shard on the known watchlist count before crawling); email is always one worker.

RSS: volume bins (skip when RSS empty, N_rss = 0):

1. Keep feeds with per-feed crawl_status in {clean, empty}; a per-feed failed drops from downstream (surfaced in §5.0).
2. Partition kept feeds by class: a deep-content group and a news-firehose group. The two classes load on different axes (deep = low-count/high-token, news = high-count/low-token), so they never share a bin.
3. Within each group, greedily pack feeds into bins so each bin's summed est_tokens stays ≤ L1_TOKEN_BUDGET. A single feed whose est_tokens alone exceeds the budget gets its own bin (the per-feed --rss-max-items-per-feed cap already bounds it). News feeds packed together keep overlapping coverage in one worker's context for consistent tagging.
4. N_rss = total RSS bin count. Record each bin's feed list (the bin's raw_output_paths) for §3.0 dispatch.

X + LinkedIn: count bins (per lane, skip when that lane is empty):

1. For X: N_x_bins = ceil(x_handle_count / SOURCES_PER_WORKER); split the ordered bare-handle list into N_x_bins contiguous near-even slices (e.g., 51 handles at 30 → 2 bins of 26 + 25). For LinkedIn: N_linkedin_bins = ceil(linkedin_source_count / SOURCES_PER_WORKER); split the ordered URL list the same way. Record each bin's source slice for §3.0 dispatch. A lane at or under SOURCES_PER_WORKER yields exactly one bin, the unsharded case falls out for free.

Email + concurrency cap:

1. N_email = 1 when email is non-empty, else 0 (email is never sharded, small sender count).
2. total_workers = N_x_bins + N_linkedin_bins + N_email + N_rss. If total_workers > L1_CONCURRENCY_CAP, cap-merge, merge the smallest bins (start with the lane carrying the most bins, then RSS news bins) until total_workers ≤ L1_CONCURRENCY_CAP, and log the cap-hit in §5.0 (a signal to raise the cap or split a watchlist).

Step 5: Compute paths + cost estimate

• report_date = today (ISO).
• RUN_DIR = output/reports/pulse-report-{report_date}/. If it exists, halt with §6.0 (report-already-exists).
• BRIEF_PATH = {RUN_DIR}pulse-report-{report_date}.md.
• tagged_dir = output/tagged/tagged-{report_date}/.
• Per-source-type raw dirs: linkedin_raw_dir, email_raw_dir, x_raw_dir = output/raw/{type}/{type}-raw-{report_date}/ (the RSS dir was created at Step 3).
• Cost estimate (paid crawls only; email + RSS are free): LinkedIn linkedin_source_count × --linkedin-max-posts × $1.50 / 1000; X x_handle_count × --x-max-per-handle × $0.0004. Realistic ≈ 30-60% of worst case for short --days windows.

No scaffold-output-dirs step, each leaf, each classifier worker, and the brief writer mkdir -ps its own dated output subdir.

---

2.0 Plan & Confirm

Pre-flight: assert crawl-tool availability

Before presenting the plan (and before any paid dispatch at §3.0), assert the external crawl tools are connected. This is a cheap tool-namespace presence check, not an API call. It fails fast here rather than deep inside the classifier workers, where a disconnected Apify silently ships a LinkedIn + X-less brief with no warning, discovered only after the paid dispatch returns:

• mcp__apify__*, required by the LinkedIn + X leaves. Assert only when LinkedIn or X is non-empty. Absent → halt with §6.0 (apify-unavailable). Paid sources are load-bearing; this halt is unconditional.
• mcp__claude_ai_Gmail__*, required by the email leaf. Assert only when email is non-empty. Absent → the email source-type can drop, but treat the drop as a decision, not an FYI: a lapsed connector silently ships a brief missing the entire email layer. Surface the absence as an explicit reconnect-or-proceed gate folded into the §2.0 plan-confirm pause (see Plan below). Reconnecting is a quick action and no money has been spent at this point, so a silent default-drop forfeits a cheap recovery.

Plan

Present the planned scope:

| Field | Value |
|---|---|
| Days window | {--days} |
| LinkedIn | {linkedin_source_count} URLs × max {--linkedin-max-posts}/URL → {N_linkedin_bins} worker(s)  (or "skipped, empty") |
| Email | {email_sender_count} senders × max {--email-max-per-sender}/sender  (or "skipped, empty", or "dropped, Gmail not connected") |
| X | {x_handle_count} handles × max {--x-max-per-handle}/handle → {N_x_bins} worker(s)  (or "skipped, empty") |
| RSS | {rss_feed_count} feeds → {rss_in_window_items} items in window → {N_rss} worker(s)  (or "skipped, empty") |
| Classifier workers | {total_workers} total ({N_x_bins} X + {N_linkedin_bins} LinkedIn + {N_email} email + {N_rss} RSS); cap {L1_CONCURRENCY_CAP}, cap-merged when a cap-hit fired |
| Cost estimate (paid) | LinkedIn ${linkedin_cost} + X ${x_cost}  (~30-60% realistic); email + RSS free |
| Profile version | v{profile_version_consumed} |
| Brief destination | {BRIEF_PATH} |

RSS is already crawled (free) at §1.0 Step 3, so the plan shows its real volume + computed worker count. The paid LinkedIn + X crawls run only after approval, at §3.0.

Gmail reconnect-or-proceed gate. When the pre-flight found mcp__claude_ai_Gmail__* absent and the email watchlist is non-empty, the plan's Email row reads dropped, Gmail not connected and the approval pause carries two options: (1) reconnect the Gmail MCP, then re-run the namespace check and proceed with email included; (2) proceed without email this run. Resolve the choice at this same pause, do not dispatch until it is answered.

Wait for user approval before §3.0.

---

3.0 Execute

Step 1: Dispatch all classifier sub-agents in one message

Dispatch every classifier worker in a single message via parallel Agent({...}) calls, one per LinkedIn bin (j = 1..N_linkedin_bins), one email worker, one per X bin (i = 1..N_x_bins), plus one per RSS bin (k = 1..N_rss). Sequential dispatch loses the parallelism; author all calls in the same response. Skip any empty source-type.

Agent({subagent_type: "general-purpose", prompt: })  # one per LinkedIn bin j
Agent({subagent_type: "general-purpose", prompt: })         # if email non-empty
Agent({subagent_type: "general-purpose", prompt: })          # one per X bin i
Agent({subagent_type: "general-purpose", prompt: })        # one per RSS bin k

Two worker shapes share one classify-tag-write contract, differing only in how they obtain raw:

• Crawl-and-classify (LinkedIn / email / X): invoke the source-type's pure-primitive leaf in-worker on the worker's source slice (isolating the Apify / Gmail raw dataset out of the main thread), read the per-source raw files, then classify + domain-tag. X and LinkedIn shard into one worker per count-bin (§1.0 Step 4); email is a single worker over all senders.
• RSS classifier (one per bin): the RSS raw is already on disk from §1.0 Step 3; read only the assigned bin's per-feed files, then classify + domain-tag.

SHARED CLASSIFIER CONTRACT: embed verbatim in every worker prompt (interpolate {source_type}, {tagged_path}, {report_date}, {--days}, {profile_version}, and the profile §3.0 enum + §4.0 drop criteria):

You are the {source_type} signals classifier-tagger. Classify and domain-tag every item from one source-type, preserve each source's framing, and write ONE tagged-output file. You do NOT assemble the brief and you do NOT dedupe.

# Classify each item: bucket = Authored / Reposted / Mentioned / Drop
- Authored, the watchlist source originated the item.
- Reposted, the source reshared another author's item verbatim. A quote-post with substantive commentary counts as Authored, not Reposted.
- Mentioned, the source is referenced or quoted inside another's item; surface only when the mention adds signal.
- Drop, none of the above, OR matches a profile drop criterion: . Drop items are excluded from the file.

# Domain-tag each survivor: keys from the profile's domain enum
Tag each survivor with one or more domain keys (an item may carry a primary + secondary key). Apply the scope + boundary cross-routing rules verbatim: .

# Preserve framing: do NOT flatten, do NOT dedupe
Write each survivor's compact_summary to capture THIS source's own framing, the angle it took + notable wording, in 1-3 sentences, not a neutral flattened fact. Use no em dashes in compact_summary prose (commas, parentheses, or sentence breaks instead) so none flow into the brief. When multiple of your sources cover the same story, keep each as its own item (repetition is signal; differing framings are insight). Never collapse duplicates.

# Write the tagged-output file (conform to schema-pulse-tagged-output)
mkdir -p the tagged dir, then write to: {tagged_path}
- Frontmatter: type "Pulse Tagged Output"; schema pulse-tagged-output; created/updated/tagged_date {report_date}; tags [pu

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [thangnguyenworkspace](https://github.com/thangnguyenworkspace)
- **Source:** [thangnguyenworkspace/industry-pulse](https://github.com/thangnguyenworkspace/industry-pulse)
- **License:** MIT

Install and usage instructions live in the source repository linked above.