Controlling AI crawlers: how to configure llms.txt, robots.txt and GPTBot

Short answer: You control AI crawlers primarily through robots.txt – and you do it by purpose. Training bots such as GPTBot, ClaudeBot and Google-Extended collect for model weights. Search and retrieval bots such as OAI-SearchBot or PerplexityBot feed answers and citations. llms.txt is not an opt-out and not a ranking lever: it helps models navigate, but it does not control access. Block everything and you give away AI visibility. Allow everything and you hand content into training data uncritically. The defensible 2026 default: steer training deliberately, allow retrieval, stop aggressive scrapers at the server edge.

The most common bad decision I see in logs: User-agent: * plus a blanket Disallow: / for “all AI”. That is about as precise as wiring the fire alarm to the light switch. You need a matrix, not panic.

Why “AI bot” is the wrong umbrella term

Under “AI crawler” sit at least three completely different jobs:

OpenAI documents this explicitly: GPTBot and OAI-SearchBot are independent. You can forbid training and still appear in ChatGPT search – or the reverse. That separation is missing from most robots.txt files I review.

Google-Extended is not a classic HTTP user-agent for search, but a control token for Gemini/AI training. Disallowing Google-Extended does not, according to Google, affect classic Search or rankings. Confuse Googlebot with Google-Extended and you either block too much or too little.

Demand picture (DataForSEO, Germany, July 2026)

llms.txt is the clear demand magnet – and the SERP is already packed with German agency and tool pieces. At the same time, People-also-ask questions (“How do I generate llms.txt?”, “How do I verify it?”) and the AI Overview sources (including SISTRIX, llmstxt.org, traffic3.net) show that many readers still confuse the file with access control. That is the content lever: clarify instead of joining the hype.

robots.txt: the only standard that counts

robots.txt has been practice since 1994 and was formalised as RFC 9309 in 2022. Every major AI vendor publicly says they read the file. That makes it the right place for opt-outs – with two limits:

1. It is a request, not a wall. Reputable providers honour it. Spoofing and aggressive scrapers do not.
2. It works forward. Content already inside training data does not come back out.

OpenAI: three bots, three decisions

Per the current OpenAI bot docs:

Pragmatic default for most content and services sites:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Search-eligibility changes take about 24 hours to apply, according to OpenAI. IP ranges for verification: openai.com/gptbot.json and openai.com/searchbot.json.

Anthropic, Perplexity, Google, the rest

Exact user-agent strings change with version numbers. In logs and filters, always match the stable token (GPTBot, OAI-SearchBot, …), not the full versioned string.

llms.txt: what it is – and what it is not

llms.txt is a proposal by Jeremy Howard (Answer.AI, September 2024): a Markdown file at /llms.txt that curatedly tells an AI which pages and Markdown versions matter. Rough format:

# Company name

> One-sentence description

## Docs

- [Services overview](https://example.com/services.md): What we offer
- [AI Search & GEO](https://example.com/services/ai-search-geo.md): GEO service

## Optional

- [Blog](https://example.com/blog.md): In-depth articles

Expectations that matter:

• Not an IETF standard. Community proposal, not an RFC.
• Not a training opt-out. Access still goes through robots.txt.
• No proven ranking/citation lever in 2026. Large-domain analyses find no solid correlation between llms.txt and AI citation rate. Google has made clear that classic SEO signals count – not this file.
• Most useful for docs, APIs, knowledge systems. There a curated Markdown map helps agents and coding assistants avoid filling the context window with HTML noise.

My practical rule: ship llms.txt when you already have clean Markdown versions or strong docs – and when the effort stays under an hour. Sell it as a “GEO silver bullet”: no. Treat it as a navigation aid and forward investment: yes.

The 2026 decision matrix

For most of my clients, row two is the sweet spot: show up in answers without giving training data away for free. That is not a moral question, it is an economic one: retrieval delivers referrals and mentions; training usually delivers no click back.

Similarweb and Semrush still show AI referrals as low-volume but high-converting – see also my article Measuring AI visibility. Blocking OAI-SearchBot and PerplexityBot while selling “GEO” is a contradiction.

Cloudflare, WAF and the invisible block

Two common traps in practice:

1. robots.txt allows OAI-SearchBot – Cloudflare blocks “AI bots”. Edge settings override your intent. Check AI Crawl Control / Bot Management and make sure the crawlers you want actually get through.
2. User-agent spoofing. Anyone can put GPTBot in a header. Real OpenAI traffic comes from published IP ranges. Hard enforcement matches IPs – not just strings.

Server logs are the reality check. Without logs you debate policy; with logs you see who actually arrives.

What actually shows up in the logs

A policy without a log sample is a statement of intent. In the access logs of a typical Next/Nginx site you often see:

• a few regular GPTBot / OAI-SearchBot hits on robots.txt, then on allowed URLs
• sporadic ChatGPT-User fetches after user questions (short burst, not a crawl session)
• PerplexityBot on source pages that later appear in answers
• “GPTBot” strings from IPs outside OpenAI’s ranges → spoofing or scanners

If you block training, GPTBot should stop fetching content URLs after the robots.txt check. If it still does from a foreign IP: WAF. If it does from a real OpenAI IP: support ticket and re-check the config – that is the exception, not the rule.

Useful filter order: user-agent first, then IP range, then path. User-agent alone is too thin.

Three scenarios, three policies

1. Consulting / agency site (lead gen)
Goal: get named in AI answers without giving content away for free training.
→ GPTBot / Google-Extended / CCBot / Bytespider: Disallow. OAI-SearchBot / PerplexityBot: Allow. llms.txt: optional if service pages are easy to curate.

2. Docs / SaaS / developer product
Goal: agents and coding assistants find the right documentation.
→ Retrieval Allow. Training Allow case by case (many docs benefit from landing in model knowledge). llms.txt: strongly recommended, plus .md mirrors of the key pages.

3. Media / paywall / premium content
Goal: control over reuse.
→ Training Disallow. Retrieval only for free teasers; paywalled URLs behind login/noindex anyway. Prioritise WAF against scraping. llms.txt usually skippable.

The matrix is intent – not religion. If you lean hard on brand mentions from model knowledge (without live search), you may allow training on purpose. If you are legally or commercially tight, you close more.

Practical checklist (30 minutes)

1. Open https://yourdomain.com/robots.txt.
2. Search for GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider.
3. Decide per bot: Allow / Disallow – training vs retrieval separated.
4. Check Cloudflare/WAF: no silent AI blocks against your intent.
5. Optional: /llms.txt with 5–15 curated links and short descriptions.
6. Verify with curl -A "GPTBot" https://yourdomain.com/ and a log sample.
7. Document the policy (date, rationale) – otherwise someone “quickly” changes it in three months.

Example: balanced robots.txt (excerpt)

# Training: opt-out
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Retrieval: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

That is a starting point, not a universal formula. Paid content, member areas and legally sensitive pages need login or noindex anyway – robots.txt alone will not keep them out of an index.

Common mistakes

Mistake: Block all AI and still expect GEO. Better: Allow retrieval, steer training on purpose.

Mistake: Treat llms.txt as a robots.txt replacement. Better: Access in robots.txt, navigation optionally in llms.txt.

Mistake: Set only GPTBot and believe you are “done”. Better: Think OpenAI, Anthropic, Google-Extended, Perplexity and scrapers one by one.

Mistake: Change robots.txt and ignore Cloudflare. Better: Check edge and origin together.

Mistake: Celebrate policy without logs. Better: Spot-check user-agents and IPs.

Conclusion: steer instead of locking or opening everything

AI crawlers in 2026 are not a yes/no question. They are a matrix of training, retrieval and enforcement. robots.txt remains the tool with a standard and real effect. llms.txt remains an optional navigation aid without a proven citation boost. The default for most sites: training opt-out where it makes sense, search bots open, aggressive scrapers stopped at the WAF.

If you want, I’ll review your robots.txt and edge settings in a free intro call. Dig deeper: AI Search & GEO, the foundation piece AI Search & GEO 2026 and Measuring AI visibility. Technically this often lands in web development & tools.

FAQ on AI crawlers and llms.txt

Does blocking GPTBot kill my chance of being named in ChatGPT? No. GPTBot controls training. ChatGPT search is governed by OAI-SearchBot. Both can be configured separately.

Do I absolutely need an llms.txt? No. Useful for docs and clear content maps, optional otherwise. It does not work as training protection or a ranking hack.

Is robots.txt enough against all AI bots? Against cooperative vendors, yes as policy. Against spoofing and aggressive scrapers you need IP/WAF rules.

Does Google-Extended affect my Google rankings? According to Google, no. Google-Extended concerns AI/Gemini training, not classic Search.