AgentStack
Browse Sign in
Browse Why AgentStack Sell Docs
Sign in
SKILL verified MIT Self-run

Taoguba Crawler

skill-lisniuse-taoguba-crawler-skill-taoguba-crawler-skill · by lisniuse

This skill should be used when the user asks to "crawl taoguba", "crawl tgb", "scrape taoguba articles", "run the crawler", "crawl bbs", "crawl home page", "generate article HTML", or needs to run the Taoguba (tgb.cn) web crawlers.

↗ Repository
— No reviews yet
0 installs
32 views
0.0% view→install

Install

$ agentstack add skill-lisniuse-taoguba-crawler-skill-taoguba-crawler-skill

✓ scanned · ✓ verified, works with Claude Code, Cursor, and more.

Verified badge

Passed review? Show it. Paste this badge into your README, it links to the public security report.

AgentStack Verified badge Links to your public security report.
[![AgentStack Verified](https://agentstack.voostack.com/badges/verified.svg)](https://agentstack.voostack.com/security/report/skill-lisniuse-taoguba-crawler-skill-taoguba-crawler-skill)

Reliability & compatibility

✓ Security review passed
0 installs to date
— no reviews yet
○ 5mo ago

Declared compatibility

Claude CodeClaude Desktop

Compatibility is declared by the source manifest. End-to-end runtime verification is coming, see below.

Preview Execution monitoring

We're building live execution health for every listing: tool-call success rate, median latency, uptime, and last-checked timestamps, measured, not self-reported. It isn't live yet, so we don't show numbers we can't stand behind.

How agent discovery & health will work →
Are you the author of Taoguba Crawler? Claim this listing to set pricing, connect Stripe payouts, and keep 70% of every sale.
Sign up to claim

About

Taoguba Crawler

This skill runs the Taoguba (tgb.cn) article crawlers located in the project root.

Prerequisites

  • Python 3 with requests, beautifulsoup4, python-dotenv installed
  • A .env file in the project root containing COOKIE and optionally USER_AGENT

Available Crawlers

1. BBS Crawler (crawler_bbs.py)

Crawl the forum board at tgb.cn/bbs/1/1 using HTML scraping.

python crawler_bbs.py
  • Extracts article list by parsing a.overhide.mw300 elements
  • Gets each article's main post and author replies
  • Downloads images and embeds them as base64 in HTML
  • Outputs: output/bbs_YYYY-MM-DD.json and output/bbs_YYYY-MM-DD_HHMMSS.html

2. Home Crawler (crawler_home.py)

Crawl the homepage recommendations via JSON API (/newIndex/getZh).

python crawler_home.py
  • Fetches articles from the JSON API (default 2 pages)
  • Same content extraction and HTML generation as BBS crawler
  • Outputs: output/home_YYYY-MM-DD.json and output/home_YYYY-MM-DD_HHMMSS.html

Common Workflow

To run both crawlers:

python crawler_bbs.py && python crawler_home.py

Key Implementation Details

  • Authentication: Both scripts read COOKIE from .env via python-dotenv
  • Rate limiting: 0.5-1s delay between requests to avoid being blocked
  • Image handling: Images are downloaded and embedded as base64 in the HTML output
  • Article content: Extracts main post (#first) and author replies (.comment-data with author badge)
  • Output directory: All results saved to output/ folder

Scripts

The crawler scripts are bundled in scripts/:

  • scripts/crawler_bbs.py - BBS forum crawler (HTML scraping)
  • scripts/crawler_home.py - Homepage crawler (JSON API)

To run the bundled scripts directly:

python scripts/crawler_bbs.py
python scripts/crawler_home.py

Troubleshooting

  • If no articles are returned, check that .env contains a valid COOKIE value
  • If image downloads fail, the HTML will show error messages inline
  • Network timeouts default to 10-15 seconds per request

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

Install and usage instructions live in the source repository linked above.

Reviews

No reviews yet, be the first.

Write a review

Versions

  • v0.1.0 Imported from the upstream source.

More in Web & Browser