aeoecommerceai-access

The AI access audit: get fetched, get read, get understood

Audit AI access in dependency order: fetchability, edge blocks, rendering, product truth, permissions, then optional agent protocols.

The short version

Audit AI access in dependency order. First prove that answer engines and user-fetchers receive real product pages; then fix edge blocks, server rendering and product-data errors. Only after those pass should you spend time on llms.txt, permission signals or agent protocols.

Use the free AI access scanner for the first check, then save direct request and log evidence for every fix.

1. Test what crawlers actually receive

Choose a small, representative set:

  • the home page;
  • one collection page;
  • one in-stock product;
  • one out-of-stock product;
  • shipping and returns policies.

Fetch each URL as a desktop-browser control and as the answer and user-fetch crawler identities that matter to your store. Test from the same host at roughly the same time so geography, rate limits and outages do not confuse the result.

Record the final URL, status code, content type and response byte count. A 200 is not enough: a CDN challenge or empty application shell can also return 200.

Use current crawler identities from operator documentation. For example, OpenAI separates OAI-SearchBot from ChatGPT-User, while Anthropic separates Claude-SearchBot from Claude-User. A policy token such as Google-Extended does not issue requests and should not be treated as a crawler test identity.

Pass condition: each allowed crawler receives the same useful page content as the browser control, with no challenge, login wall or stub response.

2. Find the enforcement layer

robots.txt states crawl preferences; it does not prove that the CDN, web application firewall or bot manager serves the request.

Interpret the comparison before changing settings:

  • One crawler blocked while the browser and other crawlers pass: inspect a User-Agent or bot-category rule at the edge.
  • All crawler identities blocked while the browser passes: inspect bot verification, IP validation and broad AI-bot categories; a header-only test may not behave like a signed real crawler.
  • Everything blocked: check rate limits, geography, authentication and availability before blaming crawler policy.
  • Status matches but byte counts differ sharply: inspect the body for a challenge, consent wall or stripped response.

Also check response headers for X-Robots-Tag: noindex. That directive can contradict an apparently permissive robots.txt file.

Pass condition: the intended crawler policy and the response delivered at the edge agree.

3. Make the useful content exist without JavaScript

Inspect the raw response body, not only a rendered browser screenshot. The initial HTML should contain:

  • the final page title;
  • product name and description;
  • price, currency and availability;
  • variant context;
  • shipping and returns links;
  • Product structured data for the same offer.

If the browser builds all of that after hydration, a text-only reader may see an empty shelf. Server-render the commercial facts and the JSON-LD needed to interpret them.

Pass condition: a raw HTML fetch contains enough accurate information to identify the product, its current offer and the policies that govern the purchase.

4. Reconcile product truth

Compare the same sample across four surfaces:

  1. the product feed;
  2. visible product-page content;
  3. Product and Offer structured data;
  4. checkout.

Price, currency, availability and variant identity must agree. Add shipping and return-policy structured data only when it matches the operational policy shoppers receive.

Fix the highest-revenue or highest-margin mismatches first. Valid markup that states the wrong price confidently is worse than missing markup.

Pass condition: the sampled offer is the same offer everywhere, including checkout.

5. Add low-cost identity and permission signals

Only after the earlier checks pass:

  • publish a short /llms.txt if it can be kept accurate with little effort;
  • use current permission syntax to separate answer/search access from model-training choices;
  • keep policy pages direct: answer first, then conditions and exceptions;
  • monitor requests and review the files when URLs or policies change.

Do not make llms.txt a visibility KPI. Its existence does not compensate for blocked crawlers, client-only content or stale product facts.

Pass condition: every published signal is current, reachable and consistent with the policy actually enforced.

6. Treat protocols as optional upside

MCP discovery, Agent Skills, API catalogs and agentic-checkout protocols sit at the bottom of the dependency chain. Check what your commerce platform already provides before building a custom layer.

Prototype only for a named integration or measurable demand. A server card pointing to no working server, or a checkout manifest that drifts from the platform, creates more ambiguity than capability.

Pass condition: a real client can discover, authenticate to and complete the advertised operation in a controlled test.

Re-run and keep evidence

For every change, repeat the original request from outside your own network. Save:

  • request identity and timestamp;
  • status, headers, final URL and body size;
  • a small response sample or checksum;
  • the changed CDN, rendering or data rule;
  • the successful retest.

Run the AI access scanner again after the fixes. A console setting is not proof; the external response is.

Sources