robots.txt tester

Check whether a crawler may fetch a URL, and see the exact rule and line that decides it. Test the live file, or a draft before you upload it.

URLs to test, one per line

Up to 20 URLs. Paths are case-sensitive, just as crawlers treat them.

Crawlers to test (6 selected)

AI training

GPTBot OpenAI ClaudeBot Anthropic Google-Extended Google Applebot-Extended Apple meta-externalagent Meta CCBot Common Crawl Bytespider ByteDance Amazonbot Amazon MistralAI-Training Mistral AI

AI search

OAI-SearchBot OpenAI Claude-SearchBot Anthropic PerplexityBot Perplexity meta-webindexer Meta Amzn-SearchBot Amazon MistralAI-Index Mistral AI DuckAssistBot DuckDuckGo

User-triggered

ChatGPT-User OpenAI Claude-User Anthropic Perplexity-User Perplexity meta-externalfetcher Meta Amzn-User Amazon MistralAI-User Mistral AI

Search engines

Googlebot Google Bingbot Microsoft

Other crawler names, separated by commas

Test a draft robots.txt instead of the live file

With a draft, you can test paths like /blog/post as well as full URLs.