# Seo Robots Ai

> AI bot crawler budget and permissions manager. Configures robots.txt to grant access to citation-capable AI search crawlers (GPTBot, ClaudeBot, Google-Extended) while blocking aggressive scrapers.

- **Type:** Skill
- **Install:** `agentstack add skill-seoskillsai-seo-skills-ai-seo-robots-ai`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [seoskillsai](https://agentstack.voostack.com/s/seoskillsai)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [seoskillsai](https://github.com/seoskillsai)
- **Source:** https://github.com/seoskillsai/seo-skills-ai/tree/main/skills/seo-robots-ai
- **Website:** https://seoskillsai.com

## Install

```sh
agentstack add skill-seoskillsai-seo-skills-ai-seo-robots-ai
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# AI Bot Crawler Permissions & Budget Manager

Configures `robots.txt` and HTTP response headers to optimize AI engine citations while protecting proprietary infrastructure from scraping attacks.

---

## 🤖 AI Crawler Policy Matrix

| Crawler Name | Operating Entity | Policy | Rationale |
| :--- | :--- | :--- | :--- |
| **`GPTBot`** | OpenAI (ChatGPT Search) | ✅ **Allow** | Powers live ChatGPT web citations and search grounding. |
| **`ClaudeBot` / `Anthropic-ai`** | Anthropic (Claude) | ✅ **Allow** | Powers Claude search grounding and web browsing. |
| **`Google-Extended`** | Google (Gemini / Overviews) | ✅ **Allow** | Enables inclusion in Google AI Overviews and Gemini citations. |
| **`Applebot-Extended`** | Apple (Apple Intelligence) | ✅ **Allow** | Enables citations in Siri and Apple Search features. |
| **`PerplexityBot`** | Perplexity AI | ✅ **Allow** | Powers real-time search citations and AI Overviews. |
| **`CCBot`** | Common Crawl | ❌ **Disallow / Rate-Limit** | Bulk unlicensed scraping without search citation benefits. |
| **`Bytespider`** | ByteDance (TikTok) | ❌ **Disallow** | Aggressive, high-frequency crawl patterns that exhaust server resources. |
| **`PetalBot`** | Huawei | ❌ **Disallow** | High crawl frequency with minimal search referral volume. |

---

## 📋 Recommended `robots.txt` Snippet

```
User-agent: *
Allow: /

# Sitemaps
Sitemap: https://example.com/sitemap-index.xml

# AI Citations Permitted
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

# Scrapers Blocked
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [seoskillsai](https://github.com/seoskillsai)
- **Source:** [seoskillsai/seo-skills-ai](https://github.com/seoskillsai/seo-skills-ai)
- **License:** MIT
- **Homepage:** https://seoskillsai.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-seoskillsai-seo-skills-ai-seo-robots-ai
- Seller: https://agentstack.voostack.com/s/seoskillsai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.