Does llms.txt Work? 97% of Files Get Zero AI Requests (2026 Data)

Short answer: llms.txt is cheap to publish, but 2026 data says AI crawlers almost never read it, and Google says you do not need it to appear in AI search. Publish one if it takes you minutes, do not expect a ranking lift, and spend your real effort on crawl access, structured data and measuring whether AI engines actually mention you.

What is llms.txt, and why did everyone add one?

llms.txt is a plain-text Markdown file at the root of a site (/llms.txt) that lists your most useful pages with short descriptions, written for language models instead of search crawlers. A longer companion, llms-full.txt, holds the full page content in Markdown. The idea is simple: instead of an AI tool parsing messy HTML, you hand it a clean index.

Adoption has exploded. According to a June 2026 update from Originality.ai, as summarised by PPC Land, llms.txt files across more than 3 million tracked sites grew from 4,088 in June 2025 to 36,120 by May 2026. Plugins did a lot of that work: Common Crawl's analysis of 584,107 llms.txt files from its July 2026 crawl reports that about two thirds were produced by a plugin.

Do AI crawlers actually read llms.txt?

Mostly no. The best evidence is server-log based:

• Ahrefs looked at 137,210 domains using May 2026 data. About 28% published an llms.txt, and 97% of those files received zero requests that month. Ahrefs notes its customers skew technical, so real-world adoption is probably lower. It also measured requests, not whether bots used the content.
• EZY Research placed valid llms.txt files on 83 sites for twelve weeks. In that test, as reported by PPC Land, OpenAI's crawler fetched the file 7 times against 3,990 robots.txt fetches. Meta's crawler was the notable exception with 193 fetches.
• SE Ranking studied 300,000 domains and found one of the 50 most-cited domains had an llms.txt, so there is no visible link to being cited.

Treat the vendor blogs that repeat these numbers with care. The log studies above are more trustworthy than marketing posts selling GEO services.

What does Google say?

Coverage of Google's May 2026 guide on optimizing for generative AI features in Search says it treats AI search optimization as ordinary SEO and that special machine-readable files such as llms.txt are not needed to appear in AI Mode or AI Overviews. I read this through secondary coverage, not the guide itself, so check Google Search Central before quoting it. Note also that this covers Google only. It does not prove that ChatGPT, Perplexity or Claude ignore the file.

Should you still publish one?

The honest framing: it is a harmless, low-cost file with unproven upside. Two things to avoid. Do not pay for a consultant who promises citations from it, and do not let a plugin publish a file you have never read. Common Crawl found some files even contain prompt injections, and about half follow the spec's structure, so a bad file is possible.

How do you create a good llms.txt without writing it by hand?

A good file has an H1 with the site name, a one-line summary in a blockquote, then sections of links with a short description each. Generating it from your sitemap keeps it in sync with the site. The LLMs.txt & llms-full.txt Generator crawls your sitemap or site, pulls titles, descriptions and Markdown content, and outputs both files ready to upload. Check the output, remove pages that should not be public, and upload to the site root.

What should you do instead, or as well?

1. Check AI crawler access. A firewall or CDN rule can block AI bots or even Googlebot without you noticing. See how Cloudflare settings can block crawlers.
2. Audit the basics. The SEO, GEO & AEO Audit scores pages for SEO, GEO and AEO, checks AI-crawler access and llms.txt, validates schema and returns a prioritized fix list.
3. Write answer-first pages. A direct 2 to 3 sentence answer under a question heading is easy to quote for both Google and AI engines.
4. Measure the outcome, not the file. The only number that matters is whether AI engines recommend you. The AI Brand Visibility Tracker asks ChatGPT, Perplexity, Gemini, Claude and Grok your buyers' questions and records mentions, rank, sentiment and cited sources. Our guide to checking whether ChatGPT recommends your competitors walks through it.

All three tools sit in the SEO and AI visibility hub.

How do you test whether your llms.txt does anything?

Do not guess, measure it. Check your server logs for requests to /llms.txt and filter by user agent. If you see no AI retrieval bots after a month, you have your answer, the same result Ahrefs found for most sites. Then track your AI mentions before and after publishing with a fixed list of questions, run on a schedule. If mentions do not move, the file is not what is driving them. Change one thing at a time so you can tell what worked.

Why do the numbers differ so much between studies?

Adoption estimates range widely because each study samples a different set of sites. SE Ranking's 300,000-domain study reported 10.13% adoption. ProGEO.ai found 7.4% of Fortune 500 companies (37 of 500) had one as of March 2026. A curated panel of 219 hosts measured on 3 August 2026 put adoption at 51.8%. These figures are not comparable, because a curated list of technical sites will always look more advanced than the open web. When you read a headline adoption number, ask what sites were in the sample before you draw conclusions for your own business.

What is the practical checklist for this week?

• Open your robots.txt and firewall rules and confirm the AI crawlers you want (and Googlebot) are not blocked by accident.
• Run a page-level audit and fix the top items on the list, usually missing schema, thin answers and weak headings.
• Generate an llms.txt from your sitemap if it takes under an hour, and read it before uploading.
• Write down 10 to 20 questions your buyers ask AI tools, and record today's answers as a baseline.
• Re-run the same questions monthly and compare mentions, ranking position and the sources each engine cites.

This order matters. Access and content quality come first because they affect every search system, while llms.txt is a small bet that only some tools might honour. If a bot shows up in your logs requesting the file, you have learned something real. If none does, you have lost very little.

Where does llms.txt fit in a wider AI-readiness plan?

Think of it as one of three layers. The first is access: can crawlers fetch your pages at all. The second is clarity: does each page state its answer plainly, with schema that describes what it is. The third is reputation: do other sites and communities that AI engines cite mention you. llms.txt only touches the second layer, and only for tools that choose to read it. That is why a tracker that reports cited sources is more useful than any file: it shows you which third-party pages the engines trust in your category, so you know where to earn a mention. For a wider look at how AI answers get built, see how the AI visibility check works step by step.

FAQ

Try it

Generate your file in a few minutes, then audit and track the real outcome: start with the Apify actors, or open the LLMs.txt Generator directly.

Last updated: November 2026