Research Crawl X Posts

Research Crawl X Posts

Argument: $ARGUMENTS (required, see Runtime Inputs)

If $ARGUMENTS is empty or missing required fields, STOP and report which fields are absent.

---

Preamble

Runtime Inputs

Parse from $ARGUMENTS:

--handles=[h1,h2,...]            [JSON-style array of bare X/Twitter handles, non-empty; no leading @, no URL. A leading @ is stripped during normalization. Singular invocation expressed as a single-element array.]
--raw-output-dir={PATH}          [absolute path to a directory where per-handle rendered markdown files will be written]
--max-per-handle={N}             [positive integer per-handle cost-cap; MUST be ≥ 1, see §1.0 Step 3 + §6.0 row max-per-handle-zero]

# Optional power params (omit when not needed):
--days={N}                       [positive integer time window in days; default 1]

Validation rules (before §1.0):

• --handles, --raw-output-dir, and --max-per-handle mandatory. If any missing, STOP and report which.
• --handles must parse as a non-empty JSON array of strings. Else reject (handles-malformed).
• Normalize each handle by stripping a single leading @; the remainder must match the X handle shape (1-15 characters, [A-Za-z0-9_] only). First failing entry → STOP and report (handles-malformed).
• --raw-output-dir must be absolute. Reject relative paths (caller owns path resolution).
• --max-per-handle must parse as integer ≥ 1. Zero rejected per §6.0 max-per-handle-zero.
• --days must parse as a positive integer when supplied; default 1.

If any validation fails, STOP and report which field failed.

Global References

APIFY_ACTOR_ID    = apidojo/tweet-scraper           # chosen X/Twitter scraper actor (no-cookie, date-windowed search mode)
APIFY_CALL_ACTOR  = mcp__apify__call-actor           # Actor invocation, waitSecs 0-45 controls inline wait; no `async` flag exists
APIFY_GET_RUN     = mcp__apify__get-actor-run        # run-status polling endpoint (waitSecs blocks server-side until terminal)
APIFY_GET_ITEMS   = mcp__apify__get-dataset-items    # dataset retrieval, projects nested dot-notation fields via fields= (auto-flatten)
APIFY_ARTICLE_ACTOR = fastcrawler/x-twitter-article-to-markdown   # §3.0 Step 3 second-hop, recovers full X Article body that tweet-scraper gutted to a bare t.co (BRONZE PAY_PER_EVENT; tweetIds ≤10/run)

Leaf does not read tool guides at runtime, parameter choices in §1.0 Step 4 and the execution procedure in §3.0 already encode the pitfalls (cost-safety cap via per-handle maxItems, search-mode date targeting via from:/since:/until: since handle-mode ignores the date window, one Apify run per handle since maxItems is per-run-total, call-actor with waitSecs:45 inline + get-actor-run poll fallback (no async flag), {noResults} sentinel filtering, fetch-time narrow-field projection via the get-dataset-items fields= param, group by author.userName for the per-source split).

Article-enrichment pitfalls (§3.0 Step 3, APIFY_ARTICLE_ACTOR), all live-confirmed 2026-06-16: pass the HOST tweet id, the tweet's own id for a standalone article, quote.id for a quoted article, NEVER article.id or the /i/article/{id} id (both return empty md); the actor pads its dataset to a fixed 10-row minimum with mock filler ({id:0, text:"mock data"}, no tweet_id, no md), so keep only rows carrying a tweet_id AND a non-empty md and discard the rest; cost is a flat per-call floor (output always pads to 10, input caps at 10 → ≤$0.05/call at BRONZE regardless of real-id count), so batch all candidate ids for the whole invocation into one ≤10-id call (chunk into successive ≤10 calls only past 10 candidates); non-article ids return md:"" (clean negative, zero false positives); Apify is the only path, Exa/Firecrawl cannot reach the login-gated /i/article/ body.

Caller Isolation

Pure primitive, no agent spawn, no pipeline agents, no telemetry. The skill runs the Apify calls + dataset render directly in whatever context invokes it; the caller absorbs the MCP dataset context cost (raw tweet items land in the caller's window). Callers crawling many handles, or composing this leaf alongside other work, wrap the invocation in their own isolation:

Agent({subagent_type: "general-purpose", model: "sonnet", prompt: "Invoke /research-crawl-x-posts with "})

Consumer skills document their own isolation choice. The run-pulse composer wraps each source-type leaf in one classifier sub-agent per source-type.

---

1.0 Context Capture

Step 1: Verify raw output directory parent exists

Check the parent directory of --raw-output-dir exists. If not, route to §6.0 (raw-output-parent-missing). Do not create parents, caller owns path resolution. The --raw-output-dir itself MAY or MAY NOT exist; §3.0 Step 1 creates it via mkdir -p before writing the first per-handle file.

Step 2: Normalize and validate handles

For each entry in --handles: strip a single leading @, then confirm the remainder matches the X handle shape (1-15 characters, [A-Za-z0-9_] only). Build handles[] of normalized bare handles preserving caller order. First entry that fails the shape → route to §6.0 (handles-malformed) with the offending value named.

Step 3: Validate --max-per-handle lower bound

Confirm --max-per-handle ≥ 1. If --max-per-handle == 0, route to §6.0 (max-per-handle-zero): pass a positive integer. --max-per-handle is the per-handle maxItems cap. Total estimated cost ≈ N_handles × max-per-handle × $0.0004 at the BRONZE/paid Apify tier; the FREE tier is 100× ($0.04/tweet) and only safe on a paid plan.

Step 4: Compute date window and construct per-handle Actor inputs

Compute the window:

since = (today - days)  formatted YYYY-MM-DD     # lower bound, inclusive
until = (today + 1 day)  formatted YYYY-MM-DD     # upper bound, EXCLUSIVE, +1 day so today is included

For each handle h in handles[], build a per-handle Actor input (one run per handle, maxItems is per-run-total, so a shared batched run starves trailing handles):

ACTOR_INPUT[h] = {
  searchTerms: ["from:{h} since:{since} until:{until}"],
  maxItems: {--max-per-handle},
  sort: "Latest",
  includeSearchTerms: true,
}

Cost-safety note: maxItems is the only verified-functional cost cap on this Actor. Do NOT add maxTotalChargeUsd (untested / non-functional on the sibling Actor; caps silently ignored). Do NOT set customMapFunction, the Actor README warns that using it for filtering triggers an automatic ban; filtering happens downstream, not in the input. Targeting MUST use searchTerms with from:/since:/until:, the twitterHandles + start/end path silently ignores the date window.

---

2.0 Plan & Confirm

Skill runs autonomously, no user pause. Pure primitive; executes directly in the caller's context. Composed leaves never gate parallelization; direct invocations log the execution config and proceed.

State the planned execution config in one structured log statement, then proceed directly to §3.0:

## /research-crawl-x-posts: executing crawl

handles:               {handles[]}                      # N normalized handles in caller-supplied order
raw_output_dir:        {--raw-output-dir}
max_per_handle:        {--max-per-handle}                # per-handle maxItems cap
days:                  {--days}                          # default 1
window:                {since} .. {until}                # until exclusive
actor:                 {APIFY_ACTOR_ID}
article_enrichment:    {APIFY_ARTICLE_ACTOR}             # §3.0 Step 3 second-hop; one batched ≤10-id call across all handles, fires only when article candidates exist
execution:             direct (pure primitive, no sub-agent spawn); one Apify run per handle
invocation:            call-actor waitSecs:45 inline → get-actor-run poll if not yet terminal (no `async` param)

---

3.0 Crawl & Enrich

Execute the crawl directly, no sub-agent spawn. Run the Apify apidojo/tweet-scraper Actor once per handle (search-mode, date-windowed), enrich any X Article posts with their full body (Step 3), then write per-handle rendered markdown files into --raw-output-dir. This step writes raw bytes only, it does NOT classify (Authored/Reposted/Mentioned), summarize, template-fill, or filter beyond the {noResults} sentinel and the per-handle cap; the composing router or domain orchestrator handles all downstream transformation. Article enrichment is raw-recovery, not transformation, it restores the verbatim article body that apidojo/tweet-scraper collapsed to a bare t.co link; it never classifies or summarizes.

Inputs (from §1.0)

Handles:               {handles[]}                       # N normalized handles in caller-supplied order
Raw output dir:        {--raw-output-dir}
Max per handle:        {--max-per-handle}                 # per-handle maxItems cap
Window:                {since} .. {until}                 # until exclusive
Actor:                 apidojo/tweet-scraper
Actor input (JSON):    {ACTOR_INPUT[h]}                   # one per handle, searchTerms from:{h} since:{since} until:{until}
invocation:            call-actor waitSecs:45 → poll get-actor-run if not terminal

The Actor inputs were auto-constructed in §1.0 Step 4. Do NOT substitute different params. Do NOT add maxTotalChargeUsd. Invoke via call-actor with waitSecs (0-45), there is NO async flag on the MCP tool. waitSecs:45 waits inline (this Actor finishes in ~5s, so the run is usually terminal on return); fall back to get-actor-run polling only when a run is still running.

Procedure

Execute these steps in order:

1. Ensure {--raw-output-dir} exists. Use Bash `mkdir -p {--raw-output-dir}`.

2. For EACH handle h in handles[] (one Apify run per handle):

   a. Call mcp__apify__call-actor with { actor: apidojo/tweet-scraper,
      input: ACTOR_INPUT[h], waitSecs: 45 }. The MCP tool has NO `async`
      flag, `waitSecs` (0-45) sets the inline wait. Capture `runId`, the
      dataset id at `storages.datasets.default.id`, and `status`. If
      `status === "SUCCEEDED"` already (typical, Actor finishes in ~5s),
      skip step 2b and go straight to step 2c.

   b. If step 2a's run was not yet terminal, poll
      mcp__apify__get-actor-run({ runId, waitSecs: 30 }), the param blocks
      server-side up to 30s per call, re-calling until `status` reaches one of:
      - `SUCCEEDED`, proceed to step 2c.
      - `FAILED` / `ABORTED` / `TIMED-OUT`, this handle's run failed. Skip
        steps 2c-2e; write an empty placeholder file for h (so callers can
        rely on one file per requested handle), set this handle's
        crawl_status: failed with diagnostic = run failure reason, continue
        to the next handle.
      Maximum poll duration per handle: 600 seconds. If still not terminal
      after 600s, abandon polling for h, treat as failed, write placeholder,
      continue with diagnostic = "Actor run exceeded 600s poll budget".

   c. Fetch this run's dataset via mcp__apify__get-dataset-items({
        datasetId: {storages.datasets.default.id}, limit: 1000,
        fields: "id,fullText,text,url,twitterUrl,createdAt,author.userName,isRetweet,isQuote,isReply,isPinned,inReplyToUsername,likeCount,retweetCount,replyCount,quoteCount,viewCount,quote.author.userName,quote.text,quote.id,article.id,article.title,article.previewText,retweet.author.userName,retweet.text,noResults,entities.media.media_url_https,extendedEntities.media.media_url_https,entities.media.type" }).
      - The `fields=` param projects AT FETCH TIME and supports nested
        dot-notation (server auto-flattens parent prefixes), so the raw
        ~198-field payload never enters caller context. This is the
        dataset-read tool's `fields` param, NOT the Actor INPUT `fields[]`
        (which is top-level-only; see step 2e note).
      - `quote.id` + the top-level `article.*` fields are the article-detection
        signals consumed by Step 3 (see step 2e); the rest carry the tweet body
        and structural flags.
      - Paginate via `offset` only when the dataset exceeds 1000 items
        (rare, bounded by max-per-handle).

   d. Filter out sentinel records where `noResults === true`. The remaining
      items are h's tweets. (A handle with no tweets in the window returns
      only the sentinel → tweets array is empty → crawl_status: empty.)

   e. The narrow field set is already projected by step 2c's `fields=` param
      (the raw record otherwise carries ~198 fields). The kept fields:
        id, fullText (fallback `text`), url (fallback `twitterUrl`),
        createdAt, author.userName, isRetweet, isQuote, isReply, isPinned,
        quote.author.userName, quote.text, quote.id, article.id, article.title,
        article.previewText, retweet.author.userName, retweet.text,
        inReplyToUsername, and media URLs
        (entities.media + extendedEntities.media `media_url_https`).
      Note, nested objects use `.text`, not `.fullText`: the nested quote/
      retweet payloads carry `quote.text` / `retweet.text`; `.fullText`
      exists only at the top level (verified via live probe). The
      get-dataset-items `fields=` param resolves all these nested keys; the
      top-level-only limitation belongs to the Apify Actor INPUT `fields[]`,
      not this dataset-read tool.
      Article-detection fields (live-confirmed 2026-06-16): a STANDALONE X
      Article populates a top-level `article` object (`article.id` /
      `article.title` / truncated `article.previewText`) while its `fullText`
      collapses to a bare t.co; a QUOTED article exposes NO `quote.article.*`
      object, only `quote.id` plus a link-collapsed `quote.text`. Both are
      consumed by Step 3.

3. Article-enrichment pass (cross-handle, single batched second-hop). After all
   handles are crawled and projected (Step 2), recover the full body of any X
   Article posts that `apidojo/tweet-scraper` collapsed to a bare t.co link.
   Detection + enrichment run ONCE across all handles' tweets, not per-handle,
   so the fixed cost floor (see Global References) is paid at most once per
   invocation.

   a. Detect article candidates across every projected tweet in this
      invocation. Two shapes, both live-confirmed:
      - STANDALONE authored article: top-level `article.id` present on the
        tweet. Candidate host id = the tweet's OWN `id`. (Free detection via the
        `article` object; the actor is still needed because `article.previewText`
        is truncated.)
      - QUOTED article: `isQuote === true` AND the quoted layer is
        link-collapsed. Mechanical test: strip every t.co/x.com URL and all
        whitespace from `quote.text`; the layer is link-collapsed when the
        residual is empty (no prose remains). Candidate host id = `quote.id`.
        (No `quote.article.*` object exists, so this is the only signal. A
        `quote.text` with residual prose is NOT a candidate, a missed real
        article is cheaper than burning the ≤10-id batch budget, and a false
        candidate that slips through returns `md:""` harmlessly.)
      Build a deduped set CANDIDATE_IDS, and a back-reference recording for each
      candidate id every tweet it belongs to plus whether each occurrence is
      that tweet's OWN body (standalone) or its QUOTED layer, needed for Step 4
      render injection. The same host id can occur in BOTH roles within one
      handle (a self-quote of one's own article); record every occurrence.

   b. If CANDIDATE_IDS is empty, skip enrichment entirely (no actor call, no
      cost); proceed to Step 4 with an empty ARTICLE_MD map.

   c. Enrich in batches of ≤10 ids (APIFY_ARTICLE_ACTOR caps tweetIds at 10).
      For each batch, call mcp__apify__call-actor with { actor:
      APIFY_ARTICLE_ACTOR, input: { tweetIds: [] }, waitSecs: 45 }. The
      actor takes ~30s; if not terminal on return, poll

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [thangnguyenworkspace](https://github.com/thangnguyenworkspace)
- **Source:** [thangnguyenworkspace/industry-pulse](https://github.com/thangnguyenworkspace/industry-pulse)
- **License:** MIT

Install and usage instructions live in the source repository linked above.