One wrong line in robots.txt can hide a site from Google or open it to every AI crawler. We check the file every day and email you when its rules change, with the lines that changed and what that means for AI crawlers and search engines. Free for up to 3 sites.
What an alert tells you
The lines that changed, with their line numbers.
What changed for AI crawlers, such as "GPTBot (OpenAI): Allowed → Blocked", for every AI crawler we track.
Red flags: Googlebot or Bingbot newly blocked, or the file deleted or replaced by a web page.
Outages: robots.txt failing to load twice in a row, a few hours apart. While it returns a server error, Google treats the whole site as blocked.
How it works
Nothing is watched until you click the link in the email we send you.
Our crawler, RobotsTesterBot, fetches the site's robots.txt once a day. It doesn't read any other page.
Changes to comments and blank lines are ignored, so you only hear about changes to the rules.
Every email has a link to manage your sites or unsubscribe. Unsubscribing deletes your address at once. What we store
Already watching a site? Get a link to manage your alerts