Your files stay on your device

Robots.txt generator

Pick a default crawl policy, per-bot rules, crawl-delay, sitemap lines, and blocked directories from a form, and get a ready robots.txt to copy or download.

User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Crawl-delay: 10 User-agent: GPTBot Disallow: / Crawl-delay: 10 User-agent: CCBot Disallow: / Crawl-delay: 10 Sitemap: https://example.com/sitemap.xml

Groups3Crawl delay10 secondsOverridden botsGPTBot, CCBotRestricted directories2

robots.txt is a request, not a lock: well-behaved crawlers read it, anything malicious ignores it. Under RFC 9309 a bot reads the one group that names it and falls back to the star group only when none does, which is why every overridden bot gets its own complete group here. Googlebot ignores Crawl-delay no matter where it appears; rate limits for Google live in Search Console, not in this file. The AI crawler list covers the crawlers site owners ask about today; new ones appear often, so check the big operators' documentation when one matters to you.
Options

More web & auth tools

- robots.txt tester (/web/robots)
- AI robots policy (/web/ai-bot-robots-txt-policy-builder)
- HTTP status reference (/web/http-status)
- HTTP header parser (/web/headers)
All web & auth tools (/web)

Was this tool useful?
YesNo

Canonical HTML: https://nutter.tools/web/robots-txt-generator
Markdown version: https://nutter.tools/web/robots-txt-generator.md
Plain-text version: https://nutter.tools/web/robots-txt-generator.txt
Agent index: https://nutter.tools/llms.txt