Your files stay on your device

AI robots policy

Per-bot allow and block rules for AI crawlers, merged with your robots.txt, with RFC 9309 path checks.

robots.txt is a cooperative signal, not a wall: well-behaved crawlers honor it and anything else can ignore it. Blocking a training crawler does not recall data already collected, and Google-Extended does not keep pages out of AI Overviews. Existing files up to 256 K are merged in place.
Options

More web & auth tools

- robots.txt tester (/web/robots)
- CSP evaluator & builder (/web/csp)
- HTTP status reference (/web/http-status)
- HTTP header parser (/web/headers)
All web & auth tools (/web)

Was this tool useful?
YesNo

Canonical HTML: https://nutter.tools/web/ai-bot-robots-txt-policy-builder
Markdown version: https://nutter.tools/web/ai-bot-robots-txt-policy-builder.md
Plain-text version: https://nutter.tools/web/ai-bot-robots-txt-policy-builder.txt
Agent index: https://nutter.tools/llms.txt