AI Robots.txt Generator – Block AI training, keep AI search

1. Start from your current robots.txt

If your site already has a robots.txt file, load it so your current rules are kept. Without it, the new file replaces everything.

Or paste it here:

2. Choose which AI crawlers to block

Block AI training, allow AI search AI companies are asked not to train on your pages, but ChatGPT search, Perplexity and other AI tools can still find and cite them. Block all AI crawlers Blocks AI training, AI search and user-triggered visits. Your pages won't be cited in most AI answers. Allow all AI crawlers Removes AI-specific rules, so AI crawlers follow your rules for all bots. Custom Tick the crawlers you want to block below.

AI training – Collect pages to train AI models.

GPTBot OpenAI ClaudeBot Anthropic Google-Extended Google Applebot-Extended Apple meta-externalagent Meta CCBot Common Crawl Bytespider ByteDance Amazonbot Amazon MistralAI-Training Mistral AI

AI search – Index pages so AI assistants can find, cite and link to them.

OAI-SearchBot OpenAI Claude-SearchBot Anthropic PerplexityBot Perplexity meta-webindexer Meta Amzn-SearchBot Amazon MistralAI-Index Mistral AI DuckAssistBot DuckDuckGo

User-triggered – Fetch a page when someone asks an AI assistant about it.

ChatGPT-User OpenAI Claude-User Anthropic Perplexity-User Perplexity meta-externalfetcher Meta Amzn-User Amazon MistralAI-User Mistral AI

Not sure what a crawler does? See the AI crawler list.

3. Choose what to block

The whole site Only these folders or pages

4. Your new robots.txt

Upload it to the root of your site so it's at https://yourdomain.com/robots.txt, then check it with the robots.txt checker. Each subdomain needs its own file. Crawlers usually pick up changes within a day.