# 03 Firecrawl Research

> >

- **Type:** Skill
- **Install:** `agentstack add skill-zevenue-headless-gtm-03-firecrawl-research`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Zevenue](https://agentstack.voostack.com/s/zevenue)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Zevenue](https://github.com/Zevenue)
- **Source:** https://github.com/Zevenue/headless-gtm/tree/main/skills/03-firecrawl-research
- **Website:** https://zevenue.com

## Install

```sh
agentstack add skill-zevenue-headless-gtm-03-firecrawl-research
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Firecrawl Research

Given a company domain, scrape its website and return clean markdown organized
by page type. Downstream skills (signal-builder, email-writer) consume this
output. This skill fetches only - scoring and interpretation belong to
signal-builder.

## Quick start

The user provides domain(s) and optionally a mode. Default to standard.

| Mode | Credits | Pages |
|------|---------|-------|
| standard | 5-8 | Homepage, About, Careers, Blog, Pricing, Customers, Integrations, Product |
| deep | 5-11 | Standard + Changelog, Leadership |
| minimal | 3 | Homepage, About only |
| extract | ~20+ | Structured JSON via LLM extraction |

If the user doesn't specify a mode, use standard. Confirm mode before running
only when the choice is ambiguous or the batch is large (>50 domains).

## Running the scraper

```bash
# Single domain
python3 scripts/firecrawl_scrape.py --domain "acme.com" --mode standard

# Batch (one domain per line in file)
python3 scripts/firecrawl_scrape.py --batch domains.txt --mode standard

# Resume interrupted batch
python3 scripts/firecrawl_scrape.py --resume runs/
```

All paths are relative to the skill folder (`firecrawl-research/`).

The script creates a timestamped run folder under `runs/` with a `tracker.json`
for progress and per-domain JSON scan files under `runs//scans/`.

After a scrape completes, read the tracker and show the user a summary
(completed/failed count, total credits).

## Writing to Google Sheet

Use `scripts/sheets_writer.py` - the Google Sheets MCP fails on large content.

```bash
python3 scripts/sheets_writer.py \
  --run-dir runs/ \
  --spreadsheet-id 
```

Options: `--summary` (char counts instead of full content), `--tab-name "name"`.

### Output columns

| Column | Content |
|--------|---------|
| Domain | acme.com |
| Status | success / partial / blocked |
| Mode | standard / deep / minimal / extract |
| Date | 2026-06-18 |
| URLs Found | 47 |
| Pages Scraped | 6 |
| Credits Used | 7 |
| Homepage…Product | Page content (or empty if not found) |

## Page types

See `references/page-types.md` for full classification and multilingual patterns.

| Tier | Pages | Modes |
|------|-------|-------|
| 1 | Homepage, About, Careers, Blog | All |
| 2 | Customers, Pricing, Integrations, Product | Standard + Deep |
| 3 | Changelog, Leadership | Deep only |

Careers scrapes the main `/careers` page only (1 credit).

## Extract mode

When the user picks extract without a custom schema, the script uses:

```json
{
  "founder": "string", "headcount_clues": "string",
  "tech_mentions": "array", "funding_clues": "string",
  "product_category": "string", "customers_mentioned": "array",
  "partners": "array", "investors": "array",
  "year_founded": "string", "locations": "array"
}
```

Extract mode is the best way to get customer/partner/investor data - it uses
LLM-powered extraction that understands context better than HTML parsing.
The user can override with a custom schema.

## Cost and credit rules

| Trigger | Action |
|---------|--------|
| Single domain > 10 credits | Warn before proceeding |
| Batch > 50 domains | Show estimated total, ask confirmation |
| Batch > 500 domains | Suggest minimal mode, require confirmation |
| Estimated batch > $10 | Hard stop, require user approval |

Credit math: credits x $0.001 (Standard plan) or credits x $0.0004 (Growth).

Credit tracking rules - these prevent silent cost overruns:
- Read actual credits from `response.metadata.credits_used`, not hardcoded counts. Firecrawl's stealth proxy charges 5 credits instead of 1 and activates automatically on blocked sites.
- Always map before scraping - blind scraping wastes credits on wrong URLs.
- Check the Firecrawl dashboard credit balance before large batches. Stealth proxy can silently 5x expected cost.
- Running scrape + extract together on the same domain has no benefit over running them separately and costs more.

## Shared output (records.jsonl)

After each run (single or batch), the script writes `records.jsonl` and `meta.json`
to the run folder alongside `tracker.json` and `scans/`. Each record carries the
stage-03 fields per `_shared/CONVENTIONS.md` - `scraped_markdown` keyed by page
type (capped at 15K chars/page; `scans/*.json` keep the full text),
`pages_scraped`, and `scrape_status` - plus `has_` labels in
`filters_matched`:

```jsonl
{"company": "Acme Corp", "domain": "acme.com", "person": null, "scraped_markdown": {"homepage": "# Acme...", "about": "## Our story..."}, "pages_scraped": 2, "scrape_status": "success", "filters_matched": ["has_homepage", "has_about"]}
```

`--batch` also accepts an upstream `records.jsonl` (from 01/02/04) directly: it
reads the domains from it and carries every upstream field through into the
output records, so the chain record keeps evolving instead of restarting here.

## Scope boundaries

| Not this skill | Use instead |
|----------------|-------------|
| Score or rank signals | signal-builder |
| Find email addresses | prospeo-resolve |
| Discover domains | prospeo-discover |

## Troubleshooting

| Issue | Fix |
|-------|-----|
| `FIRECRAWL_API_KEY not found` | Add to `.env` file |
| Map returns 0 URLs | Script falls back to homepage-only scrape |
| All pages thin_content | Site is JS-heavy or blocked |
| Credits running low | Switch to minimal mode |
| Sheet writer auth fails | Re-auth: `rm ~/.google/token.json` then re-run |
| Interrupted batch | Resume with `--resume ` |
| Script import error | `pip install -r ../_shared/requirements.txt` |

## Auth

- Firecrawl: `FIRECRAWL_API_KEY` in `.env`
- Google Sheets: OAuth2 token at `~/.google/token.json`

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Zevenue](https://github.com/Zevenue)
- **Source:** [Zevenue/headless-gtm](https://github.com/Zevenue/headless-gtm)
- **License:** MIT
- **Homepage:** https://zevenue.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-zevenue-headless-gtm-03-firecrawl-research
- Seller: https://agentstack.voostack.com/s/zevenue
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.