Your files stay on your device

Robots.txt generator

Pick a default crawl policy, per-bot rules, crawl-delay, sitemap lines, and blocked directories from a form, and get a ready robots.txt to copy or download.

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Crawl-delay: 10

User-agent: GPTBot
Disallow: /
Crawl-delay: 10

User-agent: CCBot
Disallow: /
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Groups: 3
Crawl delay: 10 seconds
Overridden bots: GPTBot, CCBot
Restricted directories: 2

robots.txt is a request, not a lock: well-behaved crawlers read it, anything malicious ignores it. Under RFC 9309 a bot reads the one group that names it and falls back to the star group only when none does, which is why every overridden bot gets its own complete group here. Googlebot ignores Crawl-delay no matter where it appears; rate limits for Google live in Search Console, not in this file. The AI crawler list covers the crawlers site owners ask about today; new ones appear often, so check the big operators' documentation when one matters to you.

Crawl policy

Default, all robots

Crawl delayA line every group carries; Googlebot ignores it entirely

Sitemap URLOptional, absolute http(s), up to 2048 characters

Restricted directories

Each path is relative to the site root and must start and end with a slash, like /admin/.

Search crawlers

Googlebot

Googlebot-Image

Bingbot

DuckDuckBot

YandexBot

Baiduspider

AI crawlers

GPTBot

OAI-SearchBot

ChatGPT-User

ClaudeBot

Claude-User

PerplexityBot

Perplexity-User

Google-Extended

CCBot

Amazonbot

Applebot-Extended

Bytespider

meta-externalagent

Diffbot

All web & auth tools

Was this tool useful?

Robots.txt generator

More web & auth tools