Is your website invisible to AI search?
One line in one file can quietly turn AI assistants away from your site. Enter your address and see for yourself: free, instant, nothing to sign up for.
Free AI search visibility checker: instantly test whether your website's robots.txt file blocks AI search crawlers, AI assistants, or AI training crawlers such as GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot.
What this check actually reads
Every website can publish a small file called robots.txt at its root. It is the standard way a site tells crawlers which parts they may visit. Search engines have followed it for decades, and the crawlers behind AI search tools use the same file. One misplaced line in it can quietly tell every AI crawler to stay away, and most site owners have no idea it is there.
This check fetches your live robots.txt and evaluates it against the published rules of the robots standard (RFC 9309), the same way major crawlers read it: longest rule wins, wildcards and end anchors respected, and a missing file means everything is allowed. It then tests the file against three groups of AI crawlers, because they do different jobs.
AI search crawlers build the indexes behind tools like ChatGPT search and Perplexity. If these are blocked, AI search simply cannot recommend your business. AI assistants fetch a page live when someone asks about your business directly. AI training crawlers collect content for model training. Blocking those last ones is a legitimate policy choice, and it does not remove you from AI search, which is exactly why a simple pass or fail answer is not good enough.
One honest limit: robots.txt is your declared policy. If your firewall or CDN blocks AI crawlers at the network level, that never shows up in this file. Our free Website Intelligence Audit goes further: it scans your live site across SEO, AI search readiness, accessibility, and performance, and shows the score your site could realistically achieve. About 30 seconds, no email required.
The crawlers this check covers
15 crawlers and 2 policy tokens across three categories, checked against your robots.txt exactly as each vendor documents them.
AI Search Crawlers
5 crawlers
Build the indexes behind ChatGPT search, Perplexity, and other AI answers. Blocked here means AI search can’t recommend you.
AI Assistants
5 crawlers
Fetch your pages live when someone asks an assistant about your business directly.
AI Training Crawlers
5 crawlers and 2 policy tokens
Collect content for model training. Blocking these is a legitimate policy choice. It doesn’t remove you from AI search.
Registry v2026.09.22, updated as vendors publish new crawlers.
Common questions
If I block GPTBot, does my site disappear from ChatGPT?
Not necessarily, and that is the problem. OpenAI, Anthropic, Perplexity and Meta each run more than one crawler, and each one does a different job: some build the AI search index, some fetch your page live for an assistant, some collect training data. A single robots.txt line can block the wrong one, or all of them, and the file gives no warning. This check grades the three groups separately. What it does not do is tell you which line did it; that is a read of the whole file against every vendor’s current documentation, which is part of what the audit is for.
Should I block AI training crawlers?
It is a legitimate policy choice, and it does not remove you from AI search. Some businesses block training crawlers to control how their content is used; others leave them open because appearing in training data is one more way AI systems learn your business exists. What matters is deciding deliberately. Most sites we check are blocking or allowing these crawlers by accident, not by policy.
Why can’t I block Google’s AI separately from Google Search?
Google’s AI features gather content through regular Googlebot, so blocking it would remove you from Google Search entirely. The separate Google-Extended token only controls whether your content can be used for Gemini training and grounding. It is a policy switch, not a crawler.
My robots.txt passes, so why doesn’t AI mention my business?
Being crawlable is the door; being cited is a quality decision. AI answers favour sites with clear structure, proper schema markup, direct answers to real questions, and visible expertise signals. That is what our free Website Intelligence Audit measures. It scans your live site across SEO, AI search readiness, accessibility, and performance, and shows the score your site could realistically achieve.
Ready to see where your site actually stands? Run the free Website Intelligence Audit: about 30 seconds, no email required.
More free tools from Xavarro
See all free toolsWebsite Intelligence Audit
Free, about 30 seconds
Score any website across four pillars: SEO, AI search visibility, accessibility, and performance. Category breakdowns, an overall grade, and mobile versus desktop performance.
Lighthouse Score Calculator
Free, interactive
See exactly how Google scores your site. Move the five performance metrics and watch the published curves, weights, and mobile versus desktop models decide the number. Then put your own traffic in and see what your load time costs you each month.
WCAG 2.1 AA Contrast Checker
Free, interactive
Check any text and background pair against WCAG 2.1 at your actual text size and weight. One straight pass or fail with the required ratio, the way an accessibility review reads it.
Schema Markup Checker
Free, instant
See every schema.org entity your page declares in the HTML your server actually sends, broken markup included, and how many of the local business signals search engines and AI assistants look for the page carries.
SERP Snippet Preview
Free, instant
See how your page reads in a Google search result. Title and meta description measured against real scoring bands, canonical and robots checks, and how many of your social share tags are there.
NAP Consistency Checker
Free, two steps
Your website next to your Google Business Profile listing: name, address, phone and website as factual match or differs rows. Pick your listing, and the check runs against that one only.
JavaScript Rendering Check
Free, about 5 seconds
Your page loaded twice on our browser, with JavaScript off and on. See how much of the text a crawler that never runs JavaScript can read, measured against the same bands the audit scores with, and read exactly what it sees.
WordPress Exposure Check
Free, about 5 seconds
Every WordPress site has the same handful of doors. See which of yours are open, from XML-RPC and username listing to public logs, leftover files, headers and HTTPS, graded on the same scale as the audit. Passive: ordinary requests only, never a login attempt.