# Web2Skill > Convert one public website URL or an explicit batch of public URLs into a reusable skill zip backed by rendered HTML snapshots and a bounded JSONL retrieval index. Use to discover a documentation directory from one URL, crawl a supplied URL set sequentially, generate a source profile, or package indexed web content as a specialized skill. - **Type:** Skill - **Install:** `agentstack add skill-zhimaai-webtoskill-webtoskill` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [zhimaAi](https://agentstack.voostack.com/s/zhimaai) - **Installs:** 0 - **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [zhimaAi](https://github.com/zhimaAi) - **Source:** https://github.com/zhimaAi/WebToSkill - **Website:** https://chatwiki.com ## Install ```sh agentstack add skill-zhimaai-webtoskill-webtoskill ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Web2Skill Use three deterministic stages. Keep every agent-created intermediate file and the final zip under the writable task directory provided by the system prompt. Run all scripts with Python 3. Do not install or upgrade packages or browser binaries at runtime. Let `` be the skill base directory supplied by the skill loader immediately above these instructions. Use that directory exactly as supplied. Execute bundled scripts in place with `python3 /scripts/...`; never copy them into the task directory or a temporary directory. Let `` be the writable task directory supplied by the system prompt. Use it exactly as supplied for every intermediate artifact, model-authored JSON file, and final zip. ## Choose create or update mode Use create mode when no existing skill zip is supplied. Use update mode only when the request explicitly supplies an existing generated skill zip path and the required unchanged skill name. Do not extract or inspect the zip with ad hoc commands. The update-mode Stage 2 command validates the archive and stages its old index and HTML under `/existing/`. Update mode is a current-scope rebuild, not an append-only merge. Stage 1 must rebuild the URL list using the original input mode: rediscover the current directory from one entry URL, or validate, normalize, and deduplicate only the current explicit batch when two or more URLs are supplied. For each current URL, Stage 2 may reuse a valid matching page from the staged cache. Pages found only in the old zip are not copied into the new crawl, so pages outside the rebuilt scope disappear from the updated skill. Cached Yuque error pages and advertisement-only pages are rejected and fetched again. ## Stage 1: Prepare the URL list Always run `/scripts/prepare_urls.py` first. - When the user supplies exactly one URL, the script opens that page with Playwright, detects its directory or navigation tree, and writes the start page plus all discovered in-scope page URLs. - When the user supplies two or more URLs, the script skips all network discovery. It only validates, normalizes, deduplicates, and writes the supplied URLs. - The output is always a UTF-8 text file with one URL per line. ```bash python3 /scripts/prepare_urls.py \ --out /crawl/url-list.txt \ "https://example.com/docs" ``` ```bash python3 /scripts/prepare_urls.py \ --out /crawl/url-list.txt \ "https://example.com/page-a" \ "https://example.com/page-b" ``` Do not manually create the URL-list file and do not run directory discovery for a batch supplied by the user. Directory navigation and each directory-collection pass use fixed 10-minute limits. A failed navigation is retried once. Directory scroll passes, convergence, and any truncation reason are appended to `/crawl/crawl.log`; they are not added to the URL-list file or the retrieval index. For ChatWiki Docs (`help.chatwiki.com`), use the Docusaurus sitemap rather than the currently visible sidebar. Rebuild sitemap paths on the supplied origin because the sitemap publishes a placeholder host, and preserve the language selected by the supplied start URL. For KanCloud books, read the complete summary tree from `application/payload+json` before using rendered catalog links. Treat a result containing only the supplied page as a preparation failure; do not continue to build a one-page skill. ## Stage 2: Crawl the prepared URLs Run `/scripts/crawl_urls.py` with the URL-list file from Stage 1. ```bash python3 /scripts/crawl_urls.py \ --url-list /crawl/url-list.txt \ --out-dir /crawl ``` In update mode, supply the existing skill zip and required unchanged name: ```bash python3 /scripts/crawl_urls.py \ --url-list /crawl/url-list.txt \ --out-dir /crawl \ --existing-skill \ --expected-name ``` The crawler has intentionally fixed behavior: - Crawl URLs sequentially with no concurrency. - Use Playwright and a fixed 60-second page timeout. - Retry one time after a timeout, browser network error, HTTP 429 or 5xx response, empty rendered body, Yuque error page, or advertisement-only page. - Stop after four consecutive final timeouts and skip the remaining URLs. A success or a non-timeout failure resets the timeout streak. - Apply built-in body selectors for ChatWiki Docs, Yuque, Feishu, OpenClaw Docs, Alibaba Cloud Help, KanCloud, and WeChat Official Account articles; use the rendered page body as the fallback. - For Feishu, retain the longest stable body snapshot when the final body is empty or shorter. - Save cleaned rendered HTML under `/crawl/html/`, removing recognized advertisement nodes. - In update mode, check each current normalized URL against the staged existing index. Reuse only a complete readable record whose HTML is neither a Yuque error page nor an advertisement-only page. Remove recognized advertisement nodes from an otherwise reusable cached snapshot. Write reused records and HTML into the new crawl output one page at a time; never prefill the new crawl with the complete old index or HTML directory. - If every current URL is reused, skip launching Chromium and still write a complete crawl log and fresh index. - For `www.yuque.com`, retry a rendered error page once, then record it as `yuque_error_page` and omit it from the HTML snapshots and retrieval index. - Retry a page whose `#main` is empty and whose remaining content contains only recognized advertisements. If it remains advertisement-only, record it as a crawl failure and omit it from the HTML snapshots and retrieval index. - When different prepared URLs redirect to the same final URL, capture and index that page once. Record the other prepared URLs as redirect duplicates in crawl coverage; they are not crawl failures. - Extract keywords with jieba from the title, description, and selected body, then merge them with the page's original metadata keywords. - When the final index contains at least four pages, including reused pages, remove cross-page high-frequency noise terms unless they occur in the page title or description, then keep at most 12 keywords per page. - Append one successful page object per line to `/crawl/index.jsonl`. - Append immediate structured progress, retry, timeout-stop, and error events to the same `/crawl/crawl.log` created during Stage 1. Use debug mode only when validating the workflow. It processes at most the first five URLs: ```bash python3 /scripts/crawl_urls.py \ --url-list /crawl/url-list.txt \ --out-dir /crawl-debug \ --debug ``` Do not pass browser, concurrency, wait, retry, depth, link-scope, or timeout options. Those controls are not part of this workflow. Never edit or delete the generated URL list, JSONL index, crawl log, or rendered HTML snapshots by hand. After crawl validation succeeds, never rerun URL preparation or crawling because a downstream stage fails. If crawl validation fails, rerun the crawler at most once without deleting artifacts; if the same error repeats, return that error instead of restarting the workflow. Validate the completed crawl with the bundled helper: ```bash python3 /scripts/validate_crawl.py \ --index /crawl/index.jsonl ``` The helper resolves every crawl-index `html_path` relative to the directory containing `index.jsonl`, never relative to the process working directory. It returns a bounded completion result containing the latest `crawl_urls run.done` counts, a bounded failure summary, a bounded redirect-duplicate summary, and a bounded Yuque error-page-skip summary. Do not open, list, print, `cat`, or otherwise load `crawl.log` through model file tools or ad hoc commands. Use only the helper output when evaluating crawl completion, failures, redirect duplicates, or skipped Yuque error pages. ## Stage 3: Build the specialized skill Run the deterministic metadata-outline helper. Do not open, list, or read `index.jsonl` through model file tools. ```bash python3 /scripts/metadata_outline.py \ --index /crawl/index.jsonl ``` The helper returns at most 60 compact page records. It allocates slots in proportion to each source site's successful page count, preserves source-site coverage when the limit permits, and samples evenly within each site instead of taking only its first pages. Use only the returned `outline` and exact URLs to create one UTF-8 JSON metadata file at `/skill-metadata.json`. Read [references/metadata.md](references/metadata.md) for the schema and limits. The model must provide the skill identity and source profile; the build script does not infer them. If `outline_truncated` is true, describe represented themes without claiming exhaustive index coverage. Do not infer negative coverage from topics missing in the outline. In update mode, set the top-level `name` in `/skill-metadata.json` to the exact required unchanged skill name supplied by the system prompt. Copy that value verbatim; do not infer, normalize, translate, or regenerate it from the current outline. Regenerate all other metadata fields from the current final index using the normal schema. If build validation rejects the metadata file, correct only `/skill-metadata.json` and rerun the build stage. For an update-mode `metadata.name` mismatch, replace only that field with the exact required unchanged name before rerunning the build. Do not repeat URL preparation or crawling when their validated outputs are already present. The build stage also reads the last `crawl_urls run.done` event from `/crawl/crawl.log`, validates its counts against `index.jsonl`, and writes a deterministic reused, succeeded, failed, redirect-duplicate, Yuque-error-page-skip, and timeout-skipped coverage note into the generated Skill. Model-authored `coverage_notes` supplement this deterministic crawl boundary and cannot replace or hide it. Use the metadata `name` value as `` and build the zip: ```bash python3 /scripts/build_skill.py \ --index /crawl/index.jsonl \ --metadata /skill-metadata.json \ --zip-out /generate_skill/.zip ``` In update mode, pass the required unchanged name: ```bash python3 /scripts/build_skill.py \ --index /crawl/index.jsonl \ --metadata /skill-metadata.json \ --expected-name \ --zip-out /generate_skill/.zip ``` The generated skill contains: ```text / |-- SKILL.md |-- agents/ | `-- openai.yaml |-- references/ | |-- web-index.jsonl | `-- html/ | `-- *.html `-- scripts/ |-- search_index.py `-- fetch_rendered_html.py ``` `search_index.py` is the bounded local retrieval helper. `fetch_rendered_html.py` fetches one current rendered page only when the saved snapshots are insufficient or the user explicitly requests current content. ## Validation and completion Before returning success: 1. Confirm the crawl-validation helper reports `status: complete` with nonzero `url_count` and `index_rows`. Require `run_done.requested == url_count`, `run_done.reused + run_done.succeeded == index_rows`, `html_paths_checked == index_rows`, `failure_summary.count == run_done.failed`, `duplicate_summary.count == run_done.duplicate_final_urls`, and `error_page_skip_summary.count == run_done.skipped_error_pages`. 2. Confirm the metadata-outline helper reports `status: complete`, at least one outline item, and no more than 60. 3. Confirm the build command succeeds. Its deterministic input validation and packaging create the required Skill contents at `/generate_skill/.zip`. 4. Immediately return only the final zip path required by the system prompt. Do not call `ls` or reopen the zip, index, HTML snapshots, or `crawl.log` after a successful build. Inspect them only to diagnose a reported script error. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [zhimaAi](https://github.com/zhimaAi) - **Source:** [zhimaAi/WebToSkill](https://github.com/zhimaAi/WebToSkill) - **License:** Apache-2.0 - **Homepage:** https://chatwiki.com Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-zhimaai-webtoskill-webtoskill - Seller: https://agentstack.voostack.com/s/zhimaai - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.