Which AI crawlers to allow in robots.txt

The four OpenAI agents, and why the distinction matters

OpenAI documents four separate user agents, and they do genuinely different jobs. OAI-SearchBot is used to surface websites in ChatGPT's search features, which is the one that puts your business in front of a person asking a question. GPTBot crawls content that may be used to train generative foundation models, which is a different transaction entirely.

ChatGPT-User visits a page when someone asks ChatGPT a question that requires it, and OpenAI notes that robots.txt rules may not apply, because the fetch is user initiated rather than an automatic crawl. OAI-AdsBot validates the safety of pages submitted as ads. Each setting operates independently, so you can be present in search while staying out of training.

The Google switch that costs nothing

Google-Extended is the token publishers use to control whether crawled content may train future Gemini models and support grounding for AI applications. The documentation is unusually direct about the consequence of blocking it: Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal.

That sentence resolves the question most business owners are actually asking. You can refuse Gemini training without risking your Search visibility. What you give up is grounding, meaning Gemini answering about you from your own pages. For a local business that wants to be found, giving that up is usually the wrong trade.

What each agent does, and what blocking it costs you

User agent	Purpose	Cost of blocking
OAI-SearchBot	Surfacing sites in ChatGPT search	You disappear from ChatGPT search results
GPTBot	Crawling content for model training	Nothing visible; content is not used for training
ChatGPT-User	Visits a page when a user asks	robots.txt may not apply to it at all
Google-Extended	Gemini training and grounding	No effect on Google Search, by Google's own statement
GoogleOther	General research and development fetches	Does not affect any specific product

Purposes and effects are taken from the OpenAI and Google documentation linked below. The wording there is the authority, not this table.

Which of these agents actually sends visitors

The decision gets easier once you know where the traffic comes from. Measured across 101,574 websites over sixteen months, ChatGPT carried 74.78 percent of AI referrals in 2026, Gemini 11.56 percent, Perplexity 7.23 percent, Copilot 3.51 percent and Claude 2.62 percent.

So the single most valuable line in your robots.txt is the one that leaves OAI-SearchBot free, because that is the agent behind three quarters of the visits. Copilot looks negligible at 3.51 percent until you remember it draws on the Bing index, which is far easier to enter than Google's. Meanwhile 45 percent of consumers used AI tools to find a local business in 2026, against 6 percent in 2025, so the share of a small pie is growing quickly.

A robots.txt for a business that wants to be found

The default position for a local business is straightforward. Allow everything that leads to a customer, which means OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot and Google's normal crawler. Then make a separate, deliberate decision about the training crawlers, GPTBot and Google-Extended.

Our own July 2026 audit of 187 live hosts found the policy debate largely beside the point. Zero hosts blocked any AI crawler, so GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended all had free access everywhere. The network robots.txt now names 10 agents explicitly, which is documentation rather than restriction.

The most common real fault is not a wrong directive at all. In our own audit of 187 live hosts, 25 served no robots.txt whatsoever, which means every question about crawler policy on those sites was moot. Write the file first, then argue about its contents.

Questions and answers

Should I block GPTBot?

It is a values question, not a visibility one. Blocking it keeps your content out of training and costs you nothing in ChatGPT search, because that runs on a different agent. If you have no strong feeling about training, leaving it open is harmless.

Will blocking Google-Extended hurt my rankings?

No. Google states that Google-Extended does not impact inclusion in Google Search and is not used as a ranking signal. It only governs Gemini training and grounding.

What if I have no robots.txt at all?

Then everything is allowed by default, which for most local businesses is the outcome they wanted anyway. It is still worth writing one, because it is the file where your sitemap is announced.

Does allowing AI crawlers mean my content gets stolen?

Allowing OAI-SearchBot means your pages can be shown and cited as a source, which is the opposite of being copied silently. Training crawlers are the separate question, and that is exactly why the agents are separate.

Sources

SE Ranking, AI traffic research study, 18 June 2026 Platform shares across 101,574 websites
BrightLocal, Local Consumer Review Survey 2026 1,002 consumers; the share using AI to find local businesses
OpenAI, Bots and crawlers documentation The four user agents and what each is used for
Google Search Central, Google crawlers and user agents Google-Extended, GoogleOther, and the statement on Search impact