Your files stay on your device

# AI robots policy

Per-bot allow and block rules for AI crawlers, merged with your robots.txt, with RFC 9309 path checks.

robots.txt is a cooperative signal, not a wall: well-behaved crawlers honor it and anything else can ignore it. Blocking a training crawler does not recall data already collected, and Google-Extended does not keep pages out of AI Overviews. Existing files up to 256 K are merged in place.
Options

## More web & auth tools

- [robots.txt tester](/web/robots)
- [CSP evaluator & builder](/web/csp)
- [HTTP status reference](/web/http-status)
- [HTTP header parser](/web/headers)
[All web & auth tools](/web)

Was this tool useful?
YesNo

---

Canonical HTML: https://nutter.tools/web/ai-bot-robots-txt-policy-builder
Markdown version: https://nutter.tools/web/ai-bot-robots-txt-policy-builder.md
Plain-text version: https://nutter.tools/web/ai-bot-robots-txt-policy-builder.txt
Agent index: https://nutter.tools/llms.txt