A file format barely two years old has gone from niche experiment to a checkbox in Google’s own developer tools. The number of websites publishing an llms.txt file grew 8.8 times in the past year, according to a year-long tracking study by Originality.ai, yet the same research found that 97% of those files show no evidence of ever being requested by an AI crawler.
That gap between rising adoption and unproven usage sits at the center of the llms.txt debate. Google recently added an llms.txt check to Chrome’s Lighthouse tool under a new “Agentic Browsing” audit category even though Google has separately said the file offers no benefit for traditional SEO.
So is llms.txt worth the effort? The honest answer depends on what kind of site is asking.
What llms.txt Does
llms.txt is a markdown file placed at a domain’s root (for example, example.com/llms.txt) that gives AI systems a curated, structured summary of a site’s most important content. Google’s Lighthouse documentation describes it as an emerging convention used to provide a machine-readable summary of a website’s content, specifically designed for large language models (LLMs) and AI agents, adding that without the file, agents may spend more time crawling a site to understand its high-level structure and primary content.
The distinction from robots.txt is worth underscoring: robots.txt tells crawlers where not to go, while llms.txt is designed to point agents toward what matters most.
How Adoption Looks Right Now
The data is more mixed than the format’s growing profile suggests, and quality varies by source.
Independent analyst Casey Burridge queried HTTP Archive’s crawl dataset, covering millions of live websites, and found that llms.txt adoption among the top 10,000 domains grew from 1.04% in July 2025 to 5.61% by June 2026. Adoption was nearly flat across site-size tiers, suggesting the format is spreading broadly rather than concentrating among elite sites. One exception: Shopify’s 78.1% adoption rate reflects a platform-level default rollout in April and May 2026, not organic merchant decisions. WordPress’s 8.7% figure is considered a more representative signal of actual human adoption.
Enterprise adoption looks even more limited. A ProGEO.ai study found that only 7.4% of Fortune 500 companies had implemented llms.txt as of March 2026, compared to 92.8% adoption of robots.txt.
SE Ranking’s separate analysis of nearly 300,000 domains put overall adoption at 10.13% and found no statistical correlation between having an llms.txt file and how often a domain was cited in LLM responses. Removing the llms.txt variable from a predictive model in that study actually improved its accuracy, suggesting the file added noise rather than signal for citation prediction.
Despite thin evidence of AI platforms reading the file, adoption is concentrated among specific types of sites. Documentation-heavy, developer-facing platforms have moved the fastest: Anthropic, Cursor, Cloudflare, and Hugging Face have published llms.txt files, and documentation host Mintlify rolled out the format across its hosted sites in November 2024, bringing thousands of documentation sites into compliance at once. That pattern extends to IDE-based coding tools, which increasingly check for llms.txt when pointed at a documentation site.
Consumer-facing brands and major AI platforms themselves tell a different story. According to Burridge, none of the major consumer AI platforms, including OpenAI and Google, serve a valid llms.txt from their own marketing or consumer-facing domains, even as their developer documentation sites do.
What Google’s Lighthouse Update Changes
The new check sits inside a Lighthouse audit category called Agentic Browsing, which evaluates how well a site is constructed for machine interaction through a set of deterministic audits. Unlike Lighthouse’s familiar 0-100 scoring, this category returns a fractional score reflecting how many agentic readiness checks a site passes, as standards for the agentic web are still emerging.
This creates a distinction worth understanding. Google’s SEO guidance maintains that the file isn’t necessary for search visibility, yet Chrome’s developer tooling now checks for it anyway. That isn’t necessarily a contradiction: Google’s SEO guidance is specific to the Google Search ecosystem, not the broader world of LLMs, AI agents, and machine-readable content discovery. Discovery through a search engine and functionality for an agent that has already landed on a page are two different jobs, and llms.txt is built for the latter.
How to Implement llms.txt Successfully
Two real-world Lighthouse audits illustrate where implementations tend to fail.
One site owner testing nohacks.co found the audit failed with the error “File does not appear to contain any links” despite the file listing accurate, working URLs. The cause was formatting: Lighthouse’s parser only recognizes links written in markdown bracket-parenthesis syntax, not plain URLs or dash-separated lists. The fix took roughly five minutes per link.
Another practitioner running the audit on seo-kreativ.de found the site’s Agentic Browsing score improved from 33% to 67% after fixing several flagged issues, including a Yoast-generated llms.txt file that initially failed for not following recommendations.
A few practices can help avoid these pitfalls:
- Place the llms.txt file at the domain root so agents can find it without additional guidance.
- Follow markdown syntax precisely. Links must use proper bracket-parenthesis formatting. A technically accurate file with plain-text links can still fail an audit if a parser doesn’t register them as links.
- Test before publishing. A missing file returns a “not applicable” result in Lighthouse rather than a failure, but a malformed file can register as an outright fail. Running it through Lighthouse’s Agentic Browsing audit or a reference markdown parser confirms it’s usable rather than just present.
- Keep it accurate and current. Passing a technical audit only confirms a file is parseable, not that it honestly represents a site. Sites generating the file automatically through a CMS plugin should periodically confirm it still reflects current structure and priorities.
The data suggests llms.txt functions less as a search visibility lever and more as baseline infrastructure for a narrower set of use cases: documentation sites, developer tools, and coding assistants that already look for it. For most other sites, it remains a low-cost, low-yield bet: not expensive to implement correctly but not yet proven to move any measurable metric. What Google’s Lighthouse update signals is that agent readiness is becoming a distinct evaluation criterion from search visibility, one that sites aiming to be legible to AI agents may want to track separately from traditional SEO performance.
FAQ
What’s the difference between llms.txt and robots.txt?
The two files do opposite jobs. Robots.txt tells crawlers which parts of a site they shouldn’t access, while llms.txt points AI systems and agents toward the content that matters most, offering a curated, markdown-formatted summary of a site’s key pages. Having an llms.txt file doesn’t replace robots.txt or change any crawler permissions, so sites that want AI crawlers to access their content still need to allow them in robots.txt.
Which websites should have an llms.txt file?
Documentation sites, developer platforms, and software tools benefit most, as coding assistants and IDE-based tools increasingly look for llms.txt when they’re pointed at documentation. For most other sites, including consumer brands, it’s a low-cost, low-yield addition: inexpensive to implement correctly but not yet proven to move measurable metrics. Sites that choose to publish one should format links in proper markdown syntax, test the file with Chrome Lighthouse’s Agentic Browsing audit, and keep it updated as site structure changes.
Ready to Stop Guessing and Start Growing?
Let’s build a digital marketing program that aligns with your revenue goals and strengthens your bottom line.