Robots Txt
A Claude skill from fabricioctelles/skills.
Install
$ agentstack add skill-fabricioctelles-skills-robots-txt ✓ scanned · ✓ verified — works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
About
Implement robots.txt
Publish a valid robots.txt at your site root per RFC 9309.
Requirements
- Serve
/robots.txtastext/plainwith HTTP 200 - Include
User-agentdirectives withAllow/Disallowrules - Reference your sitemap if one exists:
Sitemap: https://example.com/sitemap.xml
Cloudflare
AI Crawl Control can manage your robots.txt from the dashboard, including AI-specific bot rules.
Validate
POST https://isitagentready.com/api/scan
Content-Type: application/json
{"url": "https://YOUR-SITE.com"}Check that checks.discoverability.robotsTxt.status is "pass".
Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: fabricioctelles
- Source: fabricioctelles/skills
- License: Apache-2.0
- Homepage: https://skill.dev.br
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet — be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.