AgentStack
Browse Sign in
Browse Why AgentStack Sell Docs
Sign in
SKILL verified MIT Self-run

SiteCrawler

skill-cg-shmoop-claude-skills-sitecrawler · by cg-shmoop

Browser-based site crawler for URL discovery. USE WHEN crawl site, discover pages, find URLs, enumerate pages, site map, list all pages, URL discovery, crawl for accessibility, find all links on site.

↗ Repository
— No reviews yet
0 installs
34 views
0.0% view→install

Install

$ agentstack add skill-cg-shmoop-claude-skills-sitecrawler

✓ scanned · ✓ verified, works with Claude Code, Cursor, and more.

Security review

✓ Passed

No issues found. Passed automated security review. · v0.1.0 How review works →

  • ✓ Prompt-injection patterns
  • ✓ Secret / credential exfiltration
  • ✓ Dangerous shell & filesystem operations
  • ✓ Untrusted network calls
  • ✓ Known-malicious package signatures

What it can access

  • ✓ Network access No
  • ✓ Filesystem access No
  • ✓ Shell / process execution No
  • ✓ Environment & secrets No
  • ✓ Dynamic code execution No

From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.

View the full security report →

Verified badge

Passed review? Show it. Paste this badge into your README, it links to the public security report.

AgentStack Verified badge Links to your public security report.
[![AgentStack Verified](https://agentstack.voostack.com/badges/verified.svg)](https://agentstack.voostack.com/security/report/skill-cg-shmoop-claude-skills-sitecrawler)

Reliability & compatibility

✓ Security review passed
0 installs to date
— no reviews yet
● 2mo ago

Declared compatibility

Claude CodeClaude Desktop

Compatibility is declared by the source manifest. End-to-end runtime verification is coming, see below.

Preview Execution monitoring

We're building live execution health for every listing: tool-call success rate, median latency, uptime, and last-checked timestamps, measured, not self-reported. It isn't live yet, so we don't show numbers we can't stand behind.

How agent discovery & health will work →
Are you the author of SiteCrawler? Claim this listing to set pricing, connect Stripe payouts, and keep 70% of every sale.
Sign up to claim

About

SiteCrawler

Discovers user-facing pages on a website by following internal links with a real browser (Playwright). Outputs a deduplicated URL list ready for AccessibilityScan's --batch mode.

Customization

Check ~/.claude/skills/PAI/USER/SKILLCUSTOMIZATIONS/SiteCrawler/ for user overrides.

Voice Notification

When executing a workflow, do BOTH:

  1. Send voice notification:
  1. Output text notification:

`` Running the **CrawlSite** workflow in the **SiteCrawler** skill to discover pages... ``

Workflow Routing

| Workflow | Trigger | File | |----------|---------|------| | CrawlSite | "crawl site", "discover pages", "find all URLs", "enumerate pages", "list pages for scanning" | Workflows/CrawlSite.md |

Examples

Example 1: Crawl for accessibility batch scan

User: "Crawl app.example.com and find all pages for an accessibility scan"
-> Invokes CrawlSite workflow
-> Launches Playwright, follows internal links BFS from homepage
-> Outputs urls.txt with all discovered pages
-> User runs AccessibilityScan --batch urls.txt

Example 2: Authenticated crawl with sitemap

User: "Crawl the logged-in site, use the sitemap too"
-> Invokes CrawlSite workflow with --auth and --sitemap flags
-> Loads saved auth state, parses sitemap.xml for seed URLs
-> Follows links from both sitemap and on-page anchors
-> Outputs comprehensive URL list including protected pages

Example 3: Targeted shallow crawl

User: "Quick crawl of example.com, just the top 50 pages"
-> Invokes CrawlSite workflow with --depth 2 --max-pages 50
-> Fast shallow crawl, outputs first 50 unique pages found

Quick Reference

| Feature | Detail | |---------|--------| | Runtime | npx tsx (NOT bun) | | Browser | Playwright Chromium (headless) | | Algorithm | BFS (breadth-first, closest pages first) | | Auth | Playwright storage state JSON (same as AccessibilityScan) | | Sitemap | Optional: --sitemap parses sitemap.xml as seed URLs | | Output | One URL per line, sorted, deduplicated | | Defaults | depth=5, max-pages=500, parallel=3 | | Filters | Strips fragments, query params; skips images/PDFs/scripts | | Exclusions | --exclude flag (repeatable) for URL patterns |

Pipeline Integration

SiteCrawler (discover URLs) -> urls.txt -> AccessibilityScan --batch (scan all)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

Install and usage instructions live in the source repository linked above.

Reviews

No reviews yet, be the first.

Write a review

Versions

  • v0.1.0 Imported from the upstream source.

More in Web & Browser