aeocrawlersecommerce

Which AI crawlers matter for your shop (and which are only policy tokens)

Separate answer crawlers, user-fetchers, training bots and policy tokens so your shop blocks intentionally without turning away buyers.

The short version

Do not manage every AI-related token as if it were the same bot. Allowing answer crawlers and user-fetchers can affect whether shoppers find or open your products through an assistant; training crawlers are a separate policy choice; tokens such as Google-Extended do not fetch pages at all.

Audit the role, response and verified network identity of each crawler before changing robots.txt or CDN rules. Use the free AI access scanner for the first response check.

Use four buckets, not one “AI bots” switch

Answer crawlers

Answer crawlers fetch pages so an assistant or answer engine can find, summarize and cite them. The current named examples are:

  • OAI-SearchBot from OpenAI;
  • Claude-SearchBot from Anthropic;
  • PerplexityBot from Perplexity.

Blocking these can remove your products and policies from AI answers. For an ecommerce store, test product, collection, shipping and returns pages before deciding that cost is acceptable.

User-fetchers

User-fetchers act because a person has already asked an assistant to open or inspect a page. Current examples are:

  • ChatGPT-User;
  • Claude-User;
  • Perplexity-User.

Treating these as background crawlers can turn away a shopper mid-question. Test them separately from search or training bots, because their purpose and operator controls differ.

Training crawlers

Training crawlers collect material for future model development. Confirmed named examples include GPTBot and ClaudeBot. Allowing or blocking them is a business and content-policy decision; it does not need to match your answer-search policy.

Other names commonly grouped here—such as CCBot, Bytespider, meta-externalagent and Amazonbot—need their own operator-documentation and network checks. Do not infer a role from a third-party list and call the audit finished.

Policy tokens

Some names appear in robots.txt controls but never send a request. Google-Extended and Applebot-Extended govern how content fetched through other systems may be reused. They are policy tokens, not crawler identities.

You cannot prove that one of these tokens is “allowed through the CDN” by sending a request with that name. No legitimate request under that identity exists to test.

Start with the shoppers you could lose

Audit in this order:

  1. User-fetchers: a live shopper may already be asking for the page.
  2. Answer crawlers: blocking can remove the store from discovery and citations.
  3. Training crawlers: make an explicit reuse-policy decision.
  4. Policy tokens: configure the policy, but do not put them in fetch tests.
  5. Unconfirmed names: verify operator documentation before assigning a role.

This order separates commercial access from model-training preferences. A single “block all AI” rule hides that distinction.

Verify identity, not just the User-Agent

A User-Agent header is easy to copy. For OpenAI, Anthropic and Perplexity, compare source IPs with the operator's published ranges. Use the distinctive crawler token in rules rather than matching a full example string whose browser boilerplate or version can change.

For every test, save:

  • request time and URL;
  • claimed crawler token;
  • source IP and range-verification result;
  • status code, final URL and response size;
  • a body sample showing real content or the block page;
  • the CDN, WAF or bot-management rule that made the decision.

A 200 alone is not a pass: challenge pages and empty application shells can also return 200.

Keep robots policy and edge enforcement aligned

robots.txt expresses crawl preferences. Your CDN or web application firewall decides whether a request actually reaches useful content.

Test a browser control beside each request-capable crawler on the same representative URLs. If robots.txt allows an answer crawler but the edge serves it a 403, challenge or empty shell, the operational policy is “blocked” regardless of the file.

Check response headers too. An X-Robots-Tag: noindex header can contradict a permissive crawler rule.

Check yourself

Run the AI access scanner, then confirm failures with direct requests and CDN logs. Your finished register should state, for every named token: operator, role, whether it issues requests, desired policy, observed response and evidence date.

Re-check operator documentation when crawler names, versions or published IP files change. Deprecated names can outlive the systems that once used them.

Sources