llms.txt: What It Actually Does for Your AI Search Visibility (And What It Doesn't)

A few months ago, a client forwarded me an article with a subject line that read: “We need to add this immediately.” The article was about llms.txt. The client had read maybe four paragraphs of it before firing off the email, convinced it was the new robots.txt and that every site without one was already invisible to AI search engines.

I spent twenty minutes on a call walking them back from that ledge. Then I realized I'd probably have to do that another fifty times, so I'm writing this instead.

Here's the honest version: llms.txt is real, it has genuine uses, some AI tools actually do reference it, and no, Google does not use it for Search. It's not a silver bullet, it's not secretly determining who gets cited in AI Overviews, and adding a 200-line Markdown file to your server root is not going to flip your AI visibility score overnight. But it's also not nothing — and if you're serious about showing up in the answer engines that are actually growing right now (Perplexity, Claude, ChatGPT), you should understand what the file does and how to build one properly.

Let's get into the specifics.

What llms.txt Actually Is

llms.txt is a plain Markdown file that sits at https://yourdomain.com/llms.txt. It's not a standard — there's no RFC, no W3C spec, no governing body behind it. It was proposed by Jeremy Howard (of fast.ai fame) in late 2024 as a way for websites to give AI language models a curated, context-dense summary of who they are and what their content covers.

The core idea is this: LLMs have limited context windows. When an AI agent or search tool visits your site trying to understand what you're about, crawling dozens of pages is slow and inefficient. A well-structured llms.txt gives the model a shortcut — here's our mission, here's our most important content, here are the key pages you should actually read.

The file format is dead simple. At minimum, it's a single H1 with your site name, a blockquote with a one-paragraph description, and then optional H2 sections with links to your most important pages. You can also include an llms-full.txt variant that includes the full text of your most critical pages for models that want to consume everything in one shot.

That middle stat is the one that should give you pause. Over 840,000 sites have deployed the file. But in an independent crawl study monitoring over 500 million AI bot requests across a 90-day window, only 408 of those requests specifically targeted llms.txt files. The file is everywhere. The actual crawler interest — from the perspective of raw crawl volume — is still pretty minimal.

That doesn't mean it's useless. It means you should understand what kind of tools actually use it before deciding how much effort to invest.

Which AI Tools Actually Use llms.txt

This is where the nuance matters most. Not all AI search tools are the same, and they treat your site's content differently.

The ones that reference it

Perplexity AI is probably the most active consumer of llms.txt right now. When Perplexity's agent crawls a site for an in-depth research query, it can pull the llms.txt to orient itself before deciding which pages to actually read. Same story with Claude's Projects feature when you point it at a URL — it'll grab the llms.txt file if one exists as part of its context-gathering phase.

ChatGPT's Browse mode has also been documented referencing llms.txt, though the behavior isn't consistent across every query type. It seems to use it more for brand and company research queries than for general information searches.

The one that explicitly doesn't

Google has been straightforward about this: Google Search does not use llms.txt as a ranking signal or for any part of its indexing pipeline. Google has its own methods for understanding site structure — your XML sitemap, your robots.txt, your internal linking, structured data markup — and llms.txt is not part of that stack. If someone tells you to add llms.txt to improve your Google AI Overview visibility, they're guessing. The mechanisms are completely separate.

What It Can Actually Do for Your Visibility

Here's where I'll give llms.txt its genuine credit. If your site is complex — lots of content, several product lines, a mix of documentation and editorial and landing pages — a well-written llms.txt file can meaningfully improve how AI tools describe and cite you.

When I tested this with a SaaS client's site last quarter, we saw measurable differences in how Perplexity described their product before and after adding llms.txt. Before: generic, sometimes inaccurate category descriptions. After: the tool's specific differentiators, correct feature names, accurate pricing tier descriptions. That's not a ranking change — it's a brand accuracy change. But for sites where being misrepresented by AI tools costs real deals, that matters.

The other real benefit is for sites with lots of content that AI crawlers might struggle to prioritize. Think of a news site, a documentation hub, a large agency blog. Your llms.txt can explicitly tell AI tools: “These five pages are foundational to understanding what we do. These three case studies are the most representative examples of our work. Ignore the press release archive.” That's useful signal that you can't easily communicate through any other file format.

How to Write a Good llms.txt File

Most of the llms.txt files I've seen are either way too sparse (just a site name and a single paragraph) or absurdly overloaded with every URL on the domain. Neither extreme helps. Here's a practical structure that actually works:

The llms.txt Mistakes I Keep Seeing

Since this is still relatively new territory, I want to flag the errors that are already becoming common. I've audited maybe 30 sites' llms.txt files at this point and the same problems show up repeatedly.

• Listing noindex pages — If a page is noindexed in your robots meta, don't link to it in your llms.txt. You're telling AI tools to care about content you explicitly told Google to ignore. Pick a lane.
• Using it as a second sitemap — llms.txt is not sitemap.xml. It's a curated guide, not a comprehensive index. Sites that list 200+ URLs in their llms.txt are defeating the purpose entirely.
• Generic descriptions — “We provide high-quality SEO services to businesses of all sizes” tells AI tools literally nothing useful. Be specific: what do you specialize in, what makes your approach different, who are your typical clients?
• No maintenance plan — The file rots fast if you don't update it. Outdated product names, dead links, and superseded feature descriptions are actively harmful for brand accuracy in AI responses.
• Confusing it with robots.txt permissions — llms.txt cannot block crawlers. It has no enforcement mechanism. If you want to prevent AI models from scraping your content, that's a robots.txt problem (and even then it's complicated — we covered that in our robots.txt AI crawlers guide).
• Expecting immediate Google results — Google doesn't use llms.txt. If you add the file and then check your AI Overviews coverage the next week looking for improvement, you're measuring the wrong thing entirely.

The Bigger Picture: llms.txt in Your AI Visibility Stack

Here's how I'd honestly rank the things that move the needle on AI search visibility right now, in order of actual impact:

1. Structured data markup — This is still the clearest signal you can send to both Google and AI tools. Schema markup tells machines what you are, not just what your text says. It's the highest-leverage technical investment for AI visibility.

2. Site speed and crawlability — If your pages are slow or your site structure is confusing, AI crawlers bail early. A fast, clean site gets read more completely.

3. Content clarity and authority — Specific, accurate, well-organized content that demonstrates real expertise. The sites getting cited in Perplexity and ChatGPT answers aren't winning on technical tricks — they're winning because their content is genuinely the best answer to a question.

4. XML sitemap hygiene — Your sitemap still matters a lot. Clean, accurate, updated regularly, pointing to your canonical URLs.

5. llms.txt — Yes, it belongs on the list. But fifth. Not first. Not “immediately urgent.” It's a useful signal for the non-Google AI answer engines, but if you haven't nailed the first four items, you're optimizing out of order.

Should You Actually Add llms.txt to Your Site?

Probably yes — but with calibrated expectations. The file takes maybe 45 minutes to do right the first time. It has zero downside risk. And as Perplexity, Claude, and ChatGPT continue growing as traffic sources (and they are growing — we're seeing clients get meaningful referral traffic from these tools now), anything that helps AI tools describe you accurately is worth the investment.

What it won't do: rescue a slow site, fix thin content, compensate for missing schema markup, or get you into Google AI Overviews. The SEO fundamentals still run the show. llms.txt is a supplement, not a replacement.

My practical advice: audit your AI visibility situation holistically first. Find out whether your core technical signals are clean — your sitemap, schema, speed, crawl budget, and canonical structure. Then, once the foundation is solid, spend that 45 minutes on llms.txt and consider it a proper finishing touch.

Your client's frantic email about “adding this immediately” isn't entirely wrong. Just get the order of operations right.