aeollms-txtecommerce

llms.txt: what seven months of server logs say about who actually reads it

Seven months of logs found no verified frontier-lab llms.txt fetches. Keep the file small, current and outside your AI-visibility KPIs.

The short version

Do not fund llms.txt as an AI-visibility project. A seven-month log study across roughly 900 domains found 1,227 requests for the file and no verified fetch from a frontier AI lab; a separate analysis of about 300,000 domains found no citation benefit.

If you publish the file, cap the work at 30 minutes, keep it factual, and monitor your own logs. Spend real effort first on crawler access, server-rendered content, product feeds and structured data.

What the server logs found

The useful question is not how many sites publish llms.txt. It is who actually requests it.

From September 4, 2025 through April 13, 2026, a study covering roughly 900 domains counted 1,227 requests for /llms.txt. About 64.7% came from one commercial data aggregator, Dataprovider.com, and 31.9% came from human browsers. The study found no verified request from a frontier AI lab’s crawler.

A newer single-site dataset points the other way. On August 13, 2026, Marc Lou reported OpenAI and Anthropic activity on TrustMRR; its published 30-day DataFast screenshot showed 527 /llms.txt requests in the AI-answer view, against 6,800 for /mcp. DataFast says it classifies requests server-side from crawler user agents and IP-verification signals, with IP-verified traffic shown by default. This demonstrates that frontier systems can consume the file on one site, but it does not show that the file caused citations, referrals or revenue.

The two datasets can both be true. Consumption may vary sharply by site and implementation. What remains missing is broad, repeatable evidence that publishing llms.txt changes a business outcome.

Publication has not produced a measurable citation lift

A separate November 2025 analysis across roughly 300,000 domains found no measurable relationship between publishing llms.txt and being cited by AI systems. Removing the llms.txt feature from its prediction model improved accuracy.

Google Search also says it ignores the file. Chrome’s Lighthouse added an llms.txt check in May 2026, but the audit treats absence as acceptable and flags server errors rather than promising ranking or citation value.

The operator conclusion is narrow: publishing the file is harmless when it stays cheap and accurate. Treating it as a growth channel is not supported by the current evidence.

Decide whether to publish in five minutes

Publish a small llms.txt file only when all four statements are true:

  1. Your important product, collection, policy and help pages already return usable content to the crawlers you care about.
  2. Product facts in feeds, visible pages, structured data and checkout agree.
  3. A useful file can be written and deployed in about 30 minutes.
  4. Someone owns updating it when URLs, policies or product categories change.

Skip it when any of those conditions fails. A stale manifest can feed a rare reader the wrong answer, while doing nothing for the crawlers that never request it.

What to put in the file

Keep the file short enough to review by eye. Include:

  • one sentence explaining what the store sells;
  • links to the main product or collection entry points;
  • direct links to shipping, returns, warranty and contact policies;
  • a link to current help or FAQ content;
  • only facts that are also true on the linked pages.

Do not copy the entire catalogue into the file. Do not add promotional claims that are absent from the site. Do not describe APIs, checkout tools or agent capabilities that do not exist.

The file is an introduction, not a second source of product truth.

Deploy it without creating a maintenance trap

Serve the file at /llms.txt with a successful response and readable plain text. Then verify it outside your own network and save the response as evidence.

Add three lightweight checks:

  1. Availability: alert on non-200 responses for /llms.txt if you choose to publish it.
  2. Freshness: review the file when policies, navigation or product categories materially change.
  3. Consumption: count requests by verified user agent and network identity in your server or CDN logs.

Use the free AI access scanner to check the higher-priority question first: whether assistants can fetch the store at all. A healthy llms.txt endpoint does not compensate for product pages blocked by a CDN or rendered as an empty application shell.

Measure it honestly

Do not use “file published” as an AI-visibility KPI. Track outcomes that can change a business decision:

  • verified crawler requests;
  • AI-referred sessions and revenue;
  • citations or product mentions measured consistently over time;
  • product-feed eligibility and parity errors;
  • crawler-access failures on pages that matter.

If your logs show no verified consumer after a reasonable period, leave the file alone or remove it. Do not expand the project to justify the first deployment.

Revisit the decision when a major AI lab documents consumption, your own logs show repeat verified fetches, or controlled measurement finds a citation or referral effect. Until then, llms.txt belongs after fetchability, rendering, structured product data and policy accuracy—not before them.

Sources