AI crawler checker: can ChatGPT and Claude read your site?
We read your robots.txt and homepage and tell you which of 9 AI crawlers you block, whether you have an llms.txt, and whether your text is visible without JavaScript.
- No email, no signup
- 9 AI crawlers checked
- Results in seconds
What the AI crawler checker looks at
Three things decide whether an AI assistant can use your site in its answers. This tool checks all three.
AI crawlers in robots.txt
Whether each of these is allowed or blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot.
Training versus answering
Which blocked crawlers only collect training data, and which fetch pages to answer a question someone asked. The second kind matters most.
llms.txt
Whether you have an llms.txt at the root of your site, and whether it starts the way the format expects.
Content without JavaScript
How much text is in your homepage HTML before scripts run, since many AI crawlers do not run JavaScript.
How to use it
- 1Enter your website address and run the check.
- 2Look at the crawlers marked blocked, and whether they are training or answering crawlers.
- 3Edit robots.txt to allow the ones you want, then run the check again.
Blocking AI crawlers by accident
Many robots.txt files were copied from a template or changed by a plugin, and some block AI crawlers without anyone deciding to. Blocking training crawlers can be a reasonable choice. Blocking the crawlers that fetch pages to answer questions means AI assistants cannot read you when someone asks about your industry.
robots.txt is only one layer. Some hosts and CDNs, Cloudflare included, can block AI crawlers in their own settings, which robots.txt does not show. The checker tells you what robots.txt says, so check your host as well.
AI crawler checker: questions and answers
Which AI crawlers does it check?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot. These are the user-agent names OpenAI, Anthropic, Perplexity, Google and Common Crawl publish for their crawlers.
Should I block GPTBot and ClaudeBot?
That is a business decision. GPTBot and ClaudeBot collect training data; blocking them does not remove you from ChatGPT or Claude search, which use OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User. If you want to appear in AI answers, keep the search and browsing crawlers allowed.
Does an llms.txt help me rank in AI answers?
No AI company has confirmed llms.txt as a ranking signal. It is a proposed standard (llmstxt.org) and cheap to add, so treat it as housekeeping rather than a growth lever.
Does robots.txt actually stop AI crawlers?
Only crawlers that respect it. The major AI companies say their named crawlers follow robots.txt, but it is an instruction, not a lock.
More free tools
See all toolsWant your site ready for AI search?
We fix crawler access, structured data and content that only appears with JavaScript. Tell us about your site on a 30-minute call.