One Line in Your robots.txt Can Hide You From AI Search

Somewhere on your web server there is a plain-text file called robots.txt. It was probably written once, by a plugin, a developer or a hosting default, and never opened again. It is a few dozen lines at most. In 2026 it decides whether ChatGPT, Claude and Perplexity are permitted to read your site at all, and a large share of the files we check were written before any of those crawlers existed. The previous article in this series covered what an AI crawler can read once it is inside a page. This one is about whether it gets in, what the file that decides that actually says, and why the file is the smallest part of the answer.

One Vendor Is Now Several Crawlers

Two years ago "block the AI bot" meant one line naming one crawler. That is no longer what the vendors run, and the difference is the whole story. OpenAI documents four agents: GPTBot, which collects content for training its models; OAI-SearchBot, which builds the index behind ChatGPT search; ChatGPT-User, which fetches a page when a person asks ChatGPT about it; and OAI-AdsBot, which only visits pages submitted as ads. OpenAI’s own documentation states that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, that a change to robots.txt takes about 24 hours to register, and that when both crawlers are allowed one crawl may serve both purposes.

Anthropic documents three, and updated the page in February 2026 to say what each block costs. Blocking ClaudeBot excludes future content from training. Blocking Claude-User means Anthropic cannot fetch the page when a Claude user asks about it, which the company says may reduce visibility for user-directed search. Blocking Claude-SearchBot keeps the site out of the index behind Claude’s search answers. Perplexity documents two: PerplexityBot for its search index and Perplexity-User for a user’s own request, and states that it builds no foundation models, so nothing it crawls trains one.

Google is the exception again. Its AI Overviews and AI Mode read the same index that ordinary Googlebot builds, so there is no separate door to close without closing Google Search. Google-Extended, the token most people add to "block Google’s AI", is not a crawler; it is a policy switch that governs whether content already crawled may be used for Gemini training and grounding, and Google states it has no effect on Search inclusion or ranking. In March 2026 Google added a fourth identity, Google-Agent, for agents on its own infrastructure browsing on a user’s behalf.

Fifteen named agents from nine vendors, in three groups that do three different jobs. A rule aimed at the training crawler does not touch the search crawler. A rule aimed at the search crawler is the one that removes you from the answer.

What the File Is, and What It Is Not

robots.txt is a declaration. A crawler reads it and agrees, or does not, to stay out of what it names. It is not a lock, and the standard that defines it says so in plain words: it is not a form of access control. Two of its failure modes are silent. If the file is missing, every crawler is permitted everywhere, which is fine for visibility and is also the state of a site nobody has thought about. If the server returns an error for the file rather than the file itself, the standard tells crawlers to assume the entire site is off limits, search engines included, and nothing on the screen changes. A site can be closed to Google and to every AI engine at once, with the pages loading perfectly for every human visitor, because of a response nobody has ever looked at.

There is a third thing the file cannot do, and it comes from the vendors themselves. OpenAI states that because ChatGPT-User acts on a person’s request, robots.txt rules may not apply to it. Perplexity states that Perplexity-User generally ignores robots.txt. Google states that its user-triggered fetchers, Google-Agent among them, generally ignore robots.txt because a user asked for the fetch. The assistants are the fastest-growing category of AI traffic, and the file was never written to govern them.

Where the Web Has Landed

The largest measurement of the question is Hostinger’s read of 66.7 billion bot requests across more than five million sites through late 2025. Between August and November, the share of sites GPTBot could reach fell from 84% to 12%, while OAI-SearchBot, the crawler that decides whether a site appears in ChatGPT’s answers, reached 55.67%. Sites did not close the door on AI. They closed it on training and left it open for search, a pattern Cloudflare’s own robots.txt data has shown since early 2025, when GPTBot, CCBot and Google-Extended were the three most-disallowed agents on the top ten thousand domains.

The reason is in the ratio. Cloudflare publishes how many HTML pages each vendor crawls for every visitor it refers back. In June 2025 OpenAI’s figure was 1,700 to 1 and Anthropic’s 73,000 to 1. On the same measurement in June 2026, as reported by Nobori from Cloudflare Radar, Anthropic had narrowed to about 4,580 to 1, OpenAI to 848 to 1, Perplexity to 186 to 1, and Google stood at 5 to 1. Those are training-crawler economics: enormous reading, almost no visiting. The search crawlers and the assistants are the ones that send a person to a page, and they are the ones a blanket rule takes out with the rest.

What This Looks Like From the Crawler’s Side

The free AI Search Visibility Check on this site reads a robots.txt the way the crawlers do, under the same standard, and returns three verdicts rather than one: whether the AI search crawlers are open, partly blocked or blocked; the same for the assistants; the same for the training crawlers. It names which of the fifteen agents each verdict rests on, and it prints two caveats in small type that matter more than the verdicts. The first is that it reads declared policy only: what a firewall, a content delivery network or a security plugin does to the same crawler is invisible in the file. The second is that Google’s AI cannot be separated from Googlebot, whatever the file says about Google-Extended.

When we ran it on our own site it read all three groups open, every one of the fifteen agents allowed, because this site’s robots.txt was written on purpose, and the deliberate part is the point. Open is a policy here. On most of the sites we check it is an accident, and so is closed.

The deep audit takes the file further than a verdict. It scores robots.txt inside the SEO pillar on the things that cost search visibility outright: a blanket block, a block on the scripts and styles a renderer needs, a missing file. It then measures what a crawler that is allowed in can actually read, the JavaScript-off retention the previous article covered, and scores the rest of the GEO/AEO pillar on whether the page, once read, is worth citing: structured data, a parseable description of the business, trust and expertise signals, the kind of statements an answer engine can quote. Allowed, readable and citable are three different questions, and the file answers only the first.

What Is True, and What Is Overstated

• •True: every agent named above is documented by its vendor, with the vendor’s own statement of what blocking it does. Nothing in this article about a crawler comes from anywhere but the company that runs it.
• •True: blocking the training crawlers is a legitimate choice and does not remove a site from AI search. The web at large has made that choice; whether it is the right one for a business whose content is its product is a different question.
• •Overstated: "robots.txt blocks AI." It declares; it does not enforce. In August 2025 Cloudflare created unlisted domains whose robots.txt refused every crawler, then asked Perplexity about them and got detailed answers; the fetches arrived under a generic Chrome identity from rotating addresses, and Cloudflare removed Perplexity from its verified-bot list. In the same test ChatGPT-User read the file and stopped.
• •Overstated: "AI crawlers are taking your traffic, block them all." The ratios are a training-crawler number. The crawlers with a referral to give are the search crawlers and the assistants, and a blanket rule is the one instrument guaranteed to catch them.
• •Overstated: "allow everything and you are visible." Allowed is not read, which the previous article measured, and read is not cited, which is most of the GEO/AEO pillar.
• •Often missed: the file governs one host. A subdomain, a staging copy and a store on a different platform each carry their own, and the one that shipped by accident is the one nobody checks.

Why the Fix Is a Policy Decision and an Engineering Decision, Not a Text Edit

Once you know which crawler is blocked, the line that blocks it is trivial to change, and that is exactly the trap. The line is not the policy. Which doors should be open is a decision with a cost on each side: a professional-services firm whose expertise is the content it publishes may have a real interest in staying out of training sets, and a real interest in being in every search index, and the two are not the same decision. And whatever the file says, what actually happens to a crawler is decided in layers the file does not touch. Since July 2025 a new domain on Cloudflare blocks AI crawlers unless the owner chooses otherwise. Hosts ship bot rules of their own. Security plugins ship theirs. A content management system update can rewrite the file, and a staging site’s file can go live with the site. Perplexity now publishes firewall configuration instructions for its own crawler, which is a vendor conceding where the blocks really happen. The four assistants read none of it either way.

So the honest read of "is my site open to AI search" is not a read of the file. It is a read of the whole path a crawler travels on this particular site, the declared policy and the enforced one, what it finds when it arrives, and whether what it finds is worth repeating, by someone who can see every layer at once and knows what each change does to the others. That is what the deep audit does in one pass, and it is why the file, on its own, was never going to tell you.