LLMs.txt: Should You Use It for AI Search Visibility in 2026? Here Is What Google Actually Said

On May 15, 2026, Google Search Central released a documentation page titled “Optimising your website for generative AI features on Google Search.” Four days later, Lily Ray, VP of SEO Strategy at Amsive, asked John Mueller a pointed question on Bluesky that the entire SEO community had been thinking: Why does Google use LLMs.txt files and markdown pages on its own properties while telling everyone else they do not need them? 

 

Mueller’s response was the most nuanced take he has given on this topic. And it reframes the entire debate. 

 

Let me break it all down. 

 

What Is LLMs.txt and Why Does It Matter? 

LLMs.txt is a proposed file format, similar in spirit to robots.txt, that websites can place at their root directory to help large language models understand their content. Think of it as a simplified map of your site written in markdown, designed to make it easier for AI systems to find and parse your most important pages. 

 

The idea was proposed in late 2024. Since then, the AEO vs SEO conversation has expanded to include LLMs.txt: people who think it is the future of AI visibility versus people who think it is a waste of time. 

 

Here is where things get interesting. Google’s own developer documentation sites have had LLMs.txt files on them. Multiple Google properties have published markdown versions of their pages. Meanwhile, Google’s official position has been: you do not need any of this. That contradiction has been driving SEOs crazy. 

 

Google’s Official Position (May 15, 2026) 

Google’s new AI optimisation guide is the most direct statement the company has ever made on this topic. It is also the clearest signal yet for anyone pursuing answer engine optimisation alongside traditional search: the playbook has not changed as much as the industry claimed. 

 

What Google says you can skip: 

• LLMs.txt files or any “special” AI-oriented markup 

• Breaking your content into tiny chunks for AI systems 

• Rewriting content in an “AI-friendly” style 

• Special schema or markdown versions of pages 

• Seeking inauthentic brand mentions to influence AI answers 

 

What Google says actually matters: 

• Non-commodity content with first-hand experience and a unique point of view 

• Clean, crawlable technical structure 

• Semantic HTML and good page experience 

• High-quality images and video 

• Following standard SEO best practices 

 

The guide also makes a statement that settles a two-year terminology debate. Google’s exact words: “From Google Search’s perspective, optimising for generative AI search is optimising for the search experience, and thus still SEO.” 

 

In other words, AEO (Answer Engine Optimisation) and GEO (Generative Engine Optimisation) are not separate disciplines. They are SEO. The long-running AEO vs SEO debate is now settled by Google’s own documentation. Google said it. It is now official. 

 

Why This Matters: Google’s AI Features Run on the Same Index 

This is the part most people miss. Google’s AI Overviews and AI Mode do not pull from a separate AI database. They use retrieval-augmented generation (RAG) grounded in Google’s core Search index. AI Mode uses a technique called “query fan-out” that spawns multiple concurrent sub-queries through the same ranking signals used in traditional Search. 

 

Translation: if your page is not good enough to rank in regular search, it will not appear in AI-generated answers either. There is no shortcut, no special file format, and no secret markup that bypasses this. The foundation for optimising for Google AI Overviews is the same foundation you already know: crawlability, authority, and genuinely useful content. 

 

The Lily Ray and John Mueller Exchange (May 19, 2026) 

Four days after Google published the guide, Lily Ray raised the question everyone had been asking. She tagged Mueller on Bluesky and pointed out the irony: Google uses LLMs.txt files and markdown pages on its own sites, despite saying these things are not needed for search performance. 

 

Mueller’s reply was direct: “The short answer is that it’s not done for search. There’s more to websites than just SEO.” 

 

Then he went deeper with a distinction that changes how you should think about this. 

 

Discovery vs. Functionality: The Framework That Changes Everything 

Mueller introduced two concepts that SEOs should pay attention to: 

 

Discovery is about getting found. Using a global search engine to find a website or page. This is traditional SEO. 

 

Functionality is about what happens after someone (or an AI agent) has already found your page. It is about helping them do the task they came to do. 

 

Mueller compared this to CTAs on traditional websites. You do not add CTAs for SEO purposes. You add them because once a visitor has found your page, you want them to take action. Similarly, markdown pages on developer docs are not about getting found in Google Search. They are about helping AI coding tools work more effectively after they have already arrived. This is a crucial distinction for anyone building a generative engine optimisation strategy: the two goals serve different purposes and should not be conflated. 

 

Why Google’s Developer Docs Use Markdown Pages 

Mueller explained specifically why developers.google.com has markdown versions: 

 

AI-assisted coding has gotten extremely popular. Tools like Cursor, Claude Code, GitHub Copilot, and similar coding assistants produce better, more accurate code when they can easily read and parse reference documentation. Providing a simplified markdown version of developer docs helps these tools understand the context of what they are looking at. 

 

But Mueller added an important caveat. He called this “a temporary crutch.” LLMs can already read HTML perfectly well. They have been trained on billions of web pages in standard HTML format. The markdown versions might save some tokens, but they are a convenience, not a requirement. 

 

Why Your Website Probably Does Not Need This 

Mueller was blunt about non-developer sites. For regular websites, even with agentic traffic growing, making markdown versions of your content does not make much sense. If you check your server logs right now, you are likely not seeing significant agentic traffic. 

 

His example was perfect: “Making a markdown version of a shoe’s specs is not going to get you more sales.” Then he added the kicker: “competitors appreciate it tho.” In other words, you would just be making it easier for competitors to scrape your product data. 

 

His closing advice: “Prioritise needs before dreams.” Most sites have far more important SEO work to do than preparing for a hypothetical future that may or may not come. 

 

LLMs.txt: The Full Timeline 

 

 

Wait, Google Lighthouse Now Checks for LLMs.txt? 

Here is where it gets really interesting. Just days after Google’s Search team published a guide saying you do not need LLMs.txt, Google’s Lighthouse tool (version 13.3) shipped an update that includes a new “Agentic Browsing” category. This category checks whether your site has an LLMs.txt file and flags server errors when retrieving it. 

 

The Lighthouse documentation describes LLMs.txt as a way to provide “a machine-readable summary of a website’s content, specifically designed for LLMs and AI agents.” It adds that without the file, “agents may spend more time crawling the site to understand its high-level structure.” 

 

So Google Search says you do not need it. Google Lighthouse says it helps agents browse your site. Both are official Google products. This is not a contradiction if you apply Mueller’s framework. Search (discovery) does not need it. Agent browsing (functionality) might benefit from it. Two different purposes, two different teams at Google, two different recommendations. The same duality applies across the broader AEO vs SEO spectrum. 

 

Google’s Official Guide vs. What the SEO Industry Has Been Selling 

 

 

What About Non-Google AI Platforms? 

This is an important caveat that Google’s guide acknowledges. Google says you do not need LLMs.txt for Google Search. But Google is not the only AI platform. For a complete answer engine optimisation strategy, you need to consider the full landscape. 

 

ChatGPT, Claude, Perplexity, and other AI systems may operate by different rules. Some AI crawlers like ClaudeBot, GPTBot, and OpenAI’s SearchBot do actively request and consume LLMs.txt files and markdown content when available. Dries Buytaert’s experiment in January 2026 showed hundreds of requests from these bots within an hour of making markdown available. 

 

So the answer depends on which AI platforms you care about: 

 

 

So Should You Implement LLMs.txt? 

Here is my honest take based on everything we know right now. 

 

Yes, if you: 

• Run developer documentation or technical reference sites 

• Have a CMS that auto-generates LLMs.txt (zero effort to maintain) 

• Want to help AI coding tools work with your documentation 

• Already have solid SEO fundamentals in place and are looking for marginal gains 

 

No, if you: 

• Run an e-commerce site, local business, or content blog 

• Have not finished basic SEO work (crawlability, indexation, page speed, content quality) 

• Would need to manually create and maintain the file 

• Are doing it because an SEO tool flagged it as a missing item 

• Think it will help you rank in Google AI Overviews (it will not) 

 

What You Should Actually Do Instead 

Google’s guide makes the path forward clear. Here is what moves the needle for AI Search visibility in 2026. The same principles that drive traditional SEO rankings are what power optimising for Google AI Overviews, AI Mode, and the broader shift towards generative engine optimisation. 

 

1. Create non-commodity content. This is the single biggest factor. Google contrasts generic content like “7 Tips for First-Time Homebuyers” with unique content like “Why We Waived the Inspection and Saved Money: A Look Inside the Sewer Line.” The second one has lived experience that AI cannot generate on its own. Write content only you can write. 

 

2. Nail your technical SEO. Crawlability, indexation, semantic HTML, fast page speeds, mobile-friendly design. If Googlebot cannot read your page, neither can Google’s AI systems. This is not optional. 

 

3. Make your site accessible. Google’s new guide mentions that AI agents access websites by analysing screenshots, DOM structure, and the accessibility tree. A site that is accessible to humans is also accessible to AI agents. This is becoming doubly important. 

 

4. Use structured data where it makes sense. Not for AI specifically, but for rich results and helping search engines understand your content. Do not over-invest here at the expense of content quality. 

 

5. Stop chasing AI-specific hacks. No chunking. No AI-specific rewriting. No LLMs.txt for ranking purposes. No buying fake brand mentions. Google’s spam policies now explicitly apply to AI Overviews and AI Mode. 

 

The Bottom Line 

The SEO industry spent two years panicking about AI Search, rebranding traditional SEO work as AEO (Answer Engine Optimisation) and GEO (Generative Engine Optimisation), and selling courses on tactics that Google has now officially said do not matter. 

 

Google’s position is clear: what gets you ranked in search also gets you cited in AI. The playbook for optimising for Google AI Overviews is the same playbook that good marketers and SEOs have always followed. Not a new discipline. The AEO vs SEO question has been answered: they are one and the same. 

 

LLMs.txt has a legitimate use case for developer documentation and AI coding tools. For everyone else, it is a distraction from the work that actually matters. A complete generative engine optimisation guide does not start with file formats. It starts with content quality. 

 

Mueller said it best: prioritise needs before dreams. 

 

Focus on creating content that only you can create, make sure Google can crawl and index it, and stop worrying about file formats that no major AI platform has asked you to implement. 

 

That is how you win in AI Search in 2026.