Seo Robots Ai
AI bot crawler budget and permissions manager. Configures robots.txt to grant access to citation-capable AI search crawlers (GPTBot, ClaudeBot, Google-Extended) while blocking aggressive scrapers.
Install
$ agentstack add skill-seoskillsai-seo-skills-ai-seo-robots-ai ✓ scanned · ✓ verified, works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
Verified badge
Passed review? Show it. Paste this badge into your README, it links to the public security report.
Links to your public security report.[](https://agentstack.voostack.com/security/report/skill-seoskillsai-seo-skills-ai-seo-robots-ai) Reliability & compatibility
Declared compatibility
Compatibility is declared by the source manifest. End-to-end runtime verification is coming, see below.
We're building live execution health for every listing: tool-call success rate, median latency, uptime, and last-checked timestamps, measured, not self-reported. It isn't live yet, so we don't show numbers we can't stand behind.
How agent discovery & health will work →About
AI Bot Crawler Permissions & Budget Manager
Configures robots.txt and HTTP response headers to optimize AI engine citations while protecting proprietary infrastructure from scraping attacks.
🤖 AI Crawler Policy Matrix
| Crawler Name | Operating Entity | Policy | Rationale | | :--- | :--- | :--- | :--- | | GPTBot | OpenAI (ChatGPT Search) | ✅ Allow | Powers live ChatGPT web citations and search grounding. | | ClaudeBot / Anthropic-ai | Anthropic (Claude) | ✅ Allow | Powers Claude search grounding and web browsing. | | Google-Extended | Google (Gemini / Overviews) | ✅ Allow | Enables inclusion in Google AI Overviews and Gemini citations. | | Applebot-Extended | Apple (Apple Intelligence) | ✅ Allow | Enables citations in Siri and Apple Search features. | | PerplexityBot | Perplexity AI | ✅ Allow | Powers real-time search citations and AI Overviews. | | CCBot | Common Crawl | ❌ Disallow / Rate-Limit | Bulk unlicensed scraping without search citation benefits. | | Bytespider | ByteDance (TikTok) | ❌ Disallow | Aggressive, high-frequency crawl patterns that exhaust server resources. | | PetalBot | Huawei | ❌ Disallow | High crawl frequency with minimal search referral volume. |
📋 Recommended robots.txt Snippet
User-agent: *
Allow: /
# Sitemaps
Sitemap: https://example.com/sitemap-index.xml
# AI Citations Permitted
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
# Scrapers Blocked
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: seoskillsai
- Source: seoskillsai/seo-skills-ai
- License: MIT
- Homepage: https://seoskillsai.com
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet, be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.