Your files stay on your device

AI crawler access checker

Paste a robots.txt and see per-crawler access verdicts for search indexers, AI training crawlers and assistant fetchers.

Paste a robots.txt file to see each crawler's verdict.

Verdicts come from matching your pasted rules against a catalogue of crawler tokens each operator documents, grouped by purpose: search indexers, AI training crawlers, and assistant fetchers. Matching follows RFC 9309: tokens compare case-insensitively, all groups naming the same token combine, the longest rule wins, and an equal-length Allow beats a Disallow. This is what your file says, not what any crawler does: a bot can ignore robots.txt or send a token you do not expect, and blocking Google-Extended or Applebot-Extended does not stop Googlebot or Applebot from crawling. The catalogue only contains tokens the operators publish, so a crawler you block by an undocumented token will not appear here.
Options

More web & auth tools

- robots.txt tester (/web/robots)
- AI robots policy (/web/ai-bot-robots-txt-policy-builder)
- Open graph preview (/web/og-preview)
All web & auth tools (/web)

Was this tool useful?
YesNo

Canonical HTML: https://nutter.tools/web/ai-crawler-access-checker
Markdown version: https://nutter.tools/web/ai-crawler-access-checker.md
Plain-text version: https://nutter.tools/web/ai-crawler-access-checker.txt
Agent index: https://nutter.tools/llms.txt