Training Bots vs. Answer Bots: What to Block in robots.txt | Insights | Mecverse

Most teams treat "AI crawlers" as one thing: either you let them in or you block them. That is how sites end up missing from ChatGPT, Claude and Perplexity answers while believing they only opted out of model training. Each AI company runs separate crawlers for separate jobs, and a robots.txt rule only covers the user-agent it names.

Three Jobs, Three Kinds of User-Agent

Training crawlers collect pages that may be used to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), Meta-ExternalAgent, Amazonbot, CCBot (Common Crawl) and Bytespider (ByteDance). Google-Extended and Applebot-Extended are robots.txt tokens rather than separate crawlers. Google and Apple fetch with their normal crawlers and use the token to decide whether your content can be used for their AI models. Blocking these tokens does not affect your normal search rankings.

Search crawlers build the index that AI answers cite: OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot. Block these and you stop appearing as a source in those products.

User-triggered fetchers visit a page because a person asked an assistant about it right now: ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User. Blocking them means the assistant tells your prospect it could not open your page.

Decide on Training Separately From Search

Opting out of training is a legitimate policy choice, and it costs you nothing in live AI search. If that is your position, name the training crawlers explicitly and leave everything else to your default rule:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

The mistake to avoid is a blanket User-agent: * disallow added to keep scrapers out. It removes you from every answer engine along with them.

Your CDN May Be Overruling Your robots.txt

robots.txt is a request, not a gate. The actual gate is your CDN or firewall. Settings such as Cloudflare's "Block AI bots" and Bot Fight Mode, Vercel Firewall rules, and WordPress security plugins can refuse AI user-agents even when robots.txt allows them. Sometimes the bot gets a 403. Often it gets a 200 that is actually a CAPTCHA or "Just a moment..." challenge page, which looks fine in your logs and is useless to the crawler.

The reliable test is to request the same URL twice, once as a normal browser and once with the bot's user-agent, then compare status codes and look for challenge markup. Some CDNs only admit the real bot from its published IP ranges, so confirm the result in your CDN's bot settings as well.

Most AI Crawlers Do Not Run JavaScript

GPTBot, ClaudeBot, PerplexityBot and most other AI crawlers read the HTML your server sends and stop there. A client-rendered single-page app gives them an empty <div id="root"> and a bundle of scripts. View your page source, not the inspector: if your headline and key copy are not in it, AI systems cannot see them.

Server-render or statically generate the content that matters, put a real H1 in the HTML, split long pages into H2 sections (AI retrieval chunks by heading), and add JSON-LD so the facts about your company are unambiguous. Our free AI Crawlability Checker runs all of these checks against 16 AI crawlers in one pass.