Is your website invisible to AI search?
One line in one file can quietly turn AI assistants away from your site. Enter your address and see for yourself – free, instant, nothing to sign up for.
Free AI search visibility checker: instantly test whether your website's robots.txt file blocks AI search crawlers, AI assistants, or AI training crawlers such as GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot.
What this check actually reads
Every website can publish a small file called robots.txt at its root. It is the standard way a site tells crawlers which parts they may visit. Search engines have followed it for decades, and the crawlers behind AI search tools use the same file. One misplaced line in it can quietly tell every AI crawler to stay away, and most site owners have no idea it is there.
This check fetches your live robots.txt and evaluates it against the published rules of the robots standard (RFC 9309), the same way major crawlers read it: longest rule wins, wildcards and end anchors respected, and a missing file means everything is allowed. It then tests the file against three groups of AI crawlers, because they do different jobs.
AI search crawlers build the indexes behind tools like ChatGPT search and Perplexity. If these are blocked, AI search simply cannot recommend your business. AI assistants fetch a page live when someone asks about your business directly. AI training crawlers collect content for model training. Blocking those last ones is a legitimate policy choice, and it does not remove you from AI search, which is exactly why a simple pass or fail answer is not good enough.
One honest limit: robots.txt is your declared policy. If your firewall or CDN blocks AI crawlers at the network level, that never shows up in this file. Our free Website Intelligence Audit goes further: it scans your live site across SEO, AI search readiness, accessibility, and performance, and shows the score your site could realistically achieve. About 30 seconds, no email required.
The crawlers this check covers
Fifteen AI crawlers and policy tokens across three categories, checked against your robots.txt exactly as each vendor documents them.
AI Search Crawlers
Build the indexes behind ChatGPT search, Perplexity, and other AI answers. Blocked here means AI search can’t recommend you.
- OAI-SearchBotOpenAI
Builds the index behind ChatGPT search results and link citations.
- PerplexityBotPerplexity
Indexes sites for Perplexity answers and source citations.
- Claude-SearchBotAnthropic
Indexes sites to improve Claude search result quality and citations.
- DuckAssistBotDuckDuckGo
Fetches sources for DuckAssist AI-generated answers.
AI Assistants
Fetch your pages live when someone asks an assistant about your business directly.
- ChatGPT-UserOpenAI
Fetches a page when a ChatGPT user asks about it directly.
- Claude-UserAnthropic
Fetches a page when a Claude user asks about it directly.
- Perplexity-UserPerplexity
Fetches a page when a Perplexity user asks about it directly. Perplexity documents that user-initiated fetches may not honor robots.txt; the directive still declares your policy.
- Meta-ExternalFetcherMeta
Fetches links shared or requested inside Meta AI experiences.
AI Training Crawlers
Collect content for model training. Blocking these is a legitimate policy choice – it doesn’t remove you from AI search.
- GPTBotOpenAI
Collects content that can be used to train OpenAI models.
- ClaudeBotAnthropic
Collects content that can be used to improve Anthropic models.
- Google-ExtendedGoogle · policy control token
Control token governing use of your content for Gemini training and grounding. Not a crawler – Googlebot still crawls normally. Blocking this only opts content out of Gemini training/grounding.
- Applebot-ExtendedApple · policy control token
Control token governing use of your content to train Apple foundation models. Not a crawler – Applebot still crawls for Siri and Spotlight. This token only controls AI training use.
- CCBotCommon Crawl
Builds the Common Crawl corpus, a primary training source for many AI models.
- Meta-ExternalAgentMeta
Collects content that can be used to train Meta AI models.
- BytespiderByteDance
Collects content associated with ByteDance/TikTok AI model training.
Registry v2026.07.07 – updated as vendors publish new crawlers.
Common questions
If I block GPTBot, does my site disappear from ChatGPT?
No. GPTBot is OpenAI’s training crawler – blocking it only opts your content out of model training. ChatGPT’s search results are built by a different crawler, OAI-SearchBot, and live user requests arrive as ChatGPT-User. That is exactly why this check grades the three groups separately instead of giving one pass-or-fail answer.
Should I block AI training crawlers?
It is a legitimate policy choice, and it does not remove you from AI search. Some businesses block training crawlers to control how their content is used; others leave them open because appearing in training data is one more way AI systems learn your business exists. What matters is deciding deliberately – most sites we check are blocking or allowing these crawlers by accident, not by policy.
Why can’t I block Google’s AI separately from Google Search?
Google’s AI features gather content through regular Googlebot, so blocking it would remove you from Google Search entirely. The separate Google-Extended token only controls whether your content can be used for Gemini training and grounding – it is a policy switch, not a crawler.
My robots.txt passes – why doesn’t AI mention my business?
Being crawlable is the door; being cited is a quality decision. AI answers favour sites with clear structure, proper schema markup, direct answers to real questions, and visible expertise signals. That is what our free Website Intelligence Audit measures – it scans your live site across SEO, AI search readiness, accessibility, and performance, and shows the score your site could realistically achieve.
Ready to see where your site actually stands? Run the free Website Intelligence Audit – about 30 seconds, no email required.