· Ben · craft · 6 min
Your markdown repo is already the best thing you can hand an answer engine
97% of llms.txt files get zero requests. Here's who actually fetches the ones that do, what changed in the v2 spec, and why generating llms.txt for a markdown blog is a build step you already have most of.
Ahrefs crawled 137,000 domains in May 2026 and found that 97% of llms.txt files got zero requests. Not low traffic. Zero. Of roughly 38,000 domains serving a valid file, about 1,100 saw a single fetch. And 12% of the requests that did land came from tools auditing, scanning or studying the format. GEO readiness checkers at 5%, validators at 3%, research crawlers at 2%, the largest of which was a prompt-injection survey. Validators alone out-fetched retrieval bots and assistants combined.
So no, this is not a ranking file. SE Ranking looked at 300,000 domains and found no link between publishing llms.txt and being cited in LLM answers. Otterly logged 62,100 AI bot visits to one site over 90 days and counted 84 hits on /llms.txt, about 0.1%, against roughly 265 for an average content page. It lost to a PDF.
That's the honest version, and most posts on this query stop there. Here's the part they skip: the 19% of requests that did come from AI bots aren't distributed the way you'd guess, and if your blog is a folder of markdown, the expensive half of this proposal is already sitting in your repo.
Coding agents are the audience, not answer engines
Break the Ahrefs traffic down and the shape is clear. 96% of requests were bots, most of them not AI at all: SEO audit tools 21%, unidentified 14%, general web crawlers including Googlebot 13%, tech profilers like BuiltWith 11%. The AI slice splits into coding agents 10%, training crawlers 5%, assistants 2%, and retrieval bots around 1%. The top two named bots were Claude-Code and GPTBot. Slackbot fetched llms.txt more often than PerplexityBot did.
Percentages cover the ~3% of files that received any request at all: about 1,100 domains and 22,000 requests. The 12 bot categories sum to 96%; humans are the remaining 4%. Source: Ahrefs.
Read that as a job description. The thing that pulls your llms.txt is usually a coding agent that already has a reason to be on your site. Someone pointed it at your docs, or it's resolving a library you ship. It isn't a discovery channel. It's a reception desk.
The adoption pattern agrees. Casey Bergman's HTTP Archive analysis puts valid llms.txt on 5.61% of the top 10,000 sites in June 2026, up from 1.04% a year earlier, with Shopify at 78.1% purely because Shopify shipped the file to every store by default. The interesting column is who publishes deliberately: chatgpt.com, claude.ai, gemini.google.com, openai.com and anthropic.com have no llms.txt. docs.anthropic.com, docs.x.ai, docs.perplexity.ai, docs.mistral.ai and docs.cohere.com all do. (The v2 spec also lists developers.openai.com/llms.txt; the crawl checked platform.openai.com and found nothing, so treat any single lab URL as a moving target.) Nobody is doing this on the marketing site. Everyone is doing it on the docs.
Google says both things, and both are true
Chrome ships a Lighthouse audit for llms.txt under agentic browsing. Its stated reasoning: without the file, agents "may spend more time crawling the site to understand its high-level structure." A missing file scores N/A rather than a fail, because the spec is optional. Across the entire Ahrefs dataset, that audit generated about 22 requests.
Meanwhile John Mueller, on Search Off the Record, called llms.txt "not done for search" and a "temporary crutch, perhaps to save some tokens" for AI coding tools. He's right that a file you write about yourself can't help a system tell you apart from a competitor who wrote a nicer one about themselves. He also conceded the one case that matters: "If someone is already on your website, maybe some kind of automated system is helpful."
Saving tokens for an agent that is already reading your site is a dismissal from a search perspective and a spec from mine. That's the whole feature. Don't buy it as distribution.
The v2 spec moved the value into the .md file
The spec was revised on 10 August 2026, and most of the "ultimate guides" ranking for llms.txt on a markdown blog still teach v1. What changed matters more than what the guides cover.
Gone: the context-expansion tooling (llms_txt2ctx) as part of the proposal, and the special machine-readable semantics of the Optional section. Added: discoverability via link relations, rel="alternate" type="text/markdown" pointing at the markdown version of a page and rel="describedby" pointing at the llms.txt that covers it, both usable as <link> elements or an HTTP Link: header. Also added: page.md is now legal alongside v1's page.html.md, and the file can live at any subpath, with the most specific one winning, so /docs/llms.txt covers everything under /docs/. The spec explicitly refuses /.well-known/ because that's root-only and plenty of authors control only a path.
The file itself is small. One required H1, an optional blockquote summary, optional prose, then H2 sections of [name](url): notes.
# Contentcron
> AI blog content for teams whose posts live in a git repo as Markdown or MDX.
> Articles arrive as pull requests on a contentcron/<slug> branch.
## Blog
- [Introducing Contentcron](https://contentcron.com/blog/introducing-contentcron.md): What the pipeline does, from crawl to PR.
- [Review content the way you review code](https://contentcron.com/blog/review-content-the-way-you-review-code.md): Why PR review beats a CMS approval queue.
## Product
- [Pricing](https://contentcron.com/pricing.md): Per project, not per word.
The link list points at .md URLs, not HTML. That's the half that's actually worth something, and it's the half a database-backed CMS has to build from scratch.
Which is why a git blog wins by default
A WordPress site installs a plugin to reconstruct markdown from HTML that was rendered from markdown someone typed into an editor. Contentful or Sanity needs a serialization layer to turn structured blocks back into text, which is the same reason your CMS can't take a pull request. There's no file. Your repo skips both conversions. The content is already the format.
And the file list is already in your frontmatter. Title, description, date, tags, slug: that's every field the generator needs.
The generator is a build step, not a file you maintain
Read the content directory, parse frontmatter, emit the H1 and blockquote from config, one H2 per collection, one [title](url): description line per post sorted by date descending. Thirty lines, plus whatever you're already using to read the folder. If you've been wiring content collections with Velite, you have the parsed objects in hand and this is a map and a join.
Run it on every build. A hand-maintained llms.txt goes stale exactly the way the blog did: the last entry is from four months ago and it's the one announcing your seed round.
Serving the .md variant
Add a route that returns the raw source with Content-Type: text/markdown. Strip MDX components or replace them with their text equivalent, render the frontmatter as a short header so the title and date survive, leave the rest alone. In Next.js that's a route handler at /blog/[slug].md; in Astro an endpoint with getStaticPaths; in Hugo an output format. One pattern, three spellings.
The two lines that make it discoverable
<link rel="alternate" type="text/markdown" href="/blog/my-post.md">
<link rel="describedby" href="/llms.txt">
Or do it at the CDN, which also covers the .md responses themselves:
Link: </blog/my-post.md>; rel="alternate"; type="text/markdown",
</llms.txt>; rel="describedby"
What goes in the blockquote, and what stays out
Write the summary an agent needs to tell you apart from the other nine tabs it has open. What the product is, what stack it assumes, what the docs cover. Not your positioning statement. Mueller's objection is that self-reported files can't differentiate sites, and every blockquote that says "the industry's leading platform" is the evidence for his case.
Two absolutes. Don't dump your entire archive into llms-full.txt. v2 dropped the tooling that made concatenation part of the proposal, and a 400KB wall of text is a worse answer than a 60-line index. And don't generate an entry per tag page, per pagination page, per author archive. The file exists so a machine can skip crawling those.
Keep it honest in CI
Broken is worse than absent. In the Ahrefs data, 404 /llms.txt paths drew no AI traffic at all; the humans hitting them looked like people checking competitors.
So add a check that fails the build when a published post has no entry, when a linked .md route returns anything other than 200 and text/markdown, or when the H1 or blockquote goes missing. It's the same instinct as reviewing content the way you review code: if the artifact ships from the repo, the repo gets to reject it.
What this won't do
It won't get you cited. It won't move rankings. And it may not be the format that survives. Mueller's own estimate for agentic conventions settling is six months to a year, maybe longer, with WebMCP and agents.md in the same conversation.
What it does: makes your content cheap to read correctly for the machines that arrive with intent, and costs you one build step you can delete in an afternoon if the spec dies. That's the trade. It's a fine trade at that price, and a terrible one if you're buying it as traffic.
The generator only stays correct if the frontmatter stays correct, which is the part that quietly rots on most blogs. Contentcron infers your frontmatter fields, date format and slug style from the posts already in the repo, so new articles land as PRs with the same shape the generator reads. We don't emit llms.txt for you, and I'm not going to pretend otherwise. If your posts are already markdown in a git repo, the expensive part of this is done. Try it on your repo, first article free, no card.