Who are these crawlers?
Each AI engine sends its own robots (crawlers) to read the web. They identify themselves by their user-agent name, and your robots.txt file can accept or refuse them individually. The main ones:
| Crawler | Company | Used for |
|---|---|---|
GPTBot | OpenAI | Feeding ChatGPT and its answers |
OAI-SearchBot | OpenAI | ChatGPT's web search (live citations) |
ClaudeBot | Anthropic | Feeding Claude |
PerplexityBot | Perplexity | The Perplexity answer engine, which cites its sources |
Google-Extended | Gemini and Google's AI features (distinct from Googlebot) | |
CCBot | Common Crawl | A public archive of the web used to train many models |
Important point: Google-Extended does not touch your classic SEO. Blocking it does not remove you from Google, but it does remove you from the answers its AI generates.
What does blocking cost you?
The debate "should I let AI read my content?" is legitimate for a publisher that sells its articles. For a business whose website exists to bring in customers, the math is different: your website is a storefront, and a blocked crawler is a closed recommendation channel.
- A prospect asks ChatGPT for a recommendation: if GPTBot has never been able to read your website, your accessible competitors get cited, not you.
- Perplexity cites its sources with clickable links: blocking it removes you from that traffic.
- AI answers at the top of Google (AI Overviews) capture a growing share of clicks: blocking Google-Extended shuts you out of them.
The most common trap: unintentional blocking. Some firewalls, CDNs and hosting settings block AI crawlers by default, and a robots.txt containing User-agent: * followed by a misplaced Disallow: / does the same thing. The website looks normal to visitors, but it has been invisible to AI for months.
The lines to put in your robots.txt
To explicitly allow AI crawlers (recommended for most businesses):
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /To block a specific crawler while keeping the others, replace its Allow: / with Disallow: /. The file lives at the root of your website: yoursite.com/robots.txt.
When does blocking make sense?
Allowing is not a dogma. Three cases where blocking (all or part) is a rational choice:
- Paid content: if your value is the content itself (courses, studies, premium articles), handing it to AI can cannibalize your sales.
- Private areas: intranet, customer portal, internal search results pages. Block those specific directories rather than the whole website.
- A principled stance on the use of your content for training: in that case, know that you are trading visibility for control, and do it with your eyes open.
How do you check your situation in 30 seconds?
Open yoursite.com/robots.txt and look for the crawler names above. No mention = they inherit the general User-agent: * rule. A Disallow: / under their name = blocked. And robots.txt does not tell the whole story: a block at the firewall or CDN level does not show up there.
Our audit tests exactly this point, crawler by crawler, as part of its 11 AI visibility checks. Once access is open, the next step is helping AI understand you: that is the job of llms.txt and, more broadly, of the 4 conditions of GEO.