· Ben · craft · 8 min

Generate llms.txt from your content collection at build time

Your frontmatter already has the title, description and tags. Generate llms.txt in the build step instead of crawling your own site. Route handlers for Astro, Next.js, Hugo and Eleventy, plus the evidence on whether anything reads it.

Every post in your repo already has a title, a description, a date and a tags array. You typed them. They're sitting in frontmatter at the top of an MDX file.

Search for how to generate llms.txt and page one tells you to render all of that to HTML, hand the URL to a hosted crawler, let it parse your metadata back out of the rendered page, and download a static file to commit by hand. Then subscribe to a re-crawl schedule so the file doesn't go stale.

That's a lossy round trip back to data you never lost. llms.txt is a build artifact. It's a pure function of your content collection, the same collection your sitemap already iterates. Write the function once and the file is correct on every deploy, or the deploy fails.

Here's the code for four frameworks, then the honest part: what the measured evidence says about whether anything actually reads it.

The spec is smaller than the tooling around it

Jeremy Howard proposed llms.txt on September 3, 2024. The argument was narrow: context windows can't hold a whole website, and converting HTML to clean text is imprecise, so put one markdown file at the root that gives a model background, guidance and links.

The format, per llmstxt.org: an H1 with the project name, which is the only required section. Then a blockquote summary. Then zero or more markdown sections of any type except headings. Then zero or more H2-delimited file lists, where each entry is a markdown hyperlink, optionally followed by a colon and a note.

That's it. No JSON schema, no validator that anyone official maintains, no required fields beyond the H1.

llms-full.txt isn't in the spec at all. It's a community extension that grew out of an internal llms-ctx-full.txt pattern in FastHTML and spread after Mintlify rolled it out platform-wide in November 2024. Useful, sometimes. Not standard.

Your content collection is already the source of truth

Look at what the spec wants per entry: a title, a URL, a one-line note. Look at what getCollection('blog') hands you: data.title, the slug, data.description.

The entire file is a .map() over frontmatter with a header glued on top. There is no crawling step because there is nothing to recover.

This is the same argument as treating your markdown repo as the thing you hand an answer engine in the first place. The repo is the structured data. Everything downstream is a projection of it.

Astro does it in one route file and one library function

The .txt.ts extension builds to the URL /llms.txt. Keep the route thin and put assembly in src/lib/ so you can unit-test the string.

// src/pages/llms.txt.ts
import type { APIRoute } from 'astro';
import { getCollection } from 'astro:content';
import { buildLlmsTxt } from '../lib/llms-txt';

export const GET: APIRoute = async ({ site }) => {
  const posts = (await getCollection('blog', ({ data }) => !data.draft))
    .sort((a, b) => b.data.date.valueOf() - a.data.date.valueOf());

  return new Response(buildLlmsTxt(posts, site!), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

// src/lib/llms-txt.ts
export function buildLlmsTxt(posts: Post[], site: URL) {
  return [
    '# Contentcron',
    '',
    '> AI blog content, shipped as pull requests. The content engine for blogs that live in Git.',
    '',
    '## Blog',
    '',
    ...posts.map(
      (p) =>
        `- [${p.data.title}](${new URL(`/blog/${p.slug}/`, site)}): ${p.data.description}`
    ),
  ].join('\n');
}

The draft filter matters. A crawler can't see your unpublished posts, but getCollection can, and shipping a link list that includes drafts is a fast way to leak a launch.

In Next.js, a route handler plus force-static

Same shape, different runtime. Set the content type explicitly, because the default is text/html and some fetchers will treat that as a signal to parse rather than read.

// app/llms.txt/route.ts
import { allPosts } from '@/.velite';

export const dynamic = 'force-static';

export async function GET() {
  const posts = allPosts
    .filter((p) => p.published)
    .sort((a, b) => +new Date(b.date) - +new Date(a.date));

  const body = [
    '# Contentcron',
    '',
    '> AI blog content, shipped as pull requests.',
    '',
    '## Blog',
    '',
    ...posts.map((p) => `- [${p.title}](https://contentcron.com${p.permalink}): ${p.description}`),
  ].join('\n');

  return new Response(body, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=0, must-revalidate',
    },
  });
}

force-static is the line that turns this from a serverless invocation into a file on the CDN. If you're wiring up a typed collection for the first time, the Velite setup gives you the allPosts import this route depends on.

Next's own documentation is the proof by example here: pages carry a note that the same URL is available as Markdown with .md appended, with an index at /docs/13/llms.txt.

Hugo and Eleventy do it with output formats

Hugo does this with two custom output formats and zero build plugins.

# hugo.toml
[outputFormats.llms]
  mediaType   = "text/plain"
  baseName    = "llms"
  isPlainText = true

[outputFormats.md]
  mediaType   = "text/markdown"
  baseName    = "index"
  isPlainText = true

[outputs]
  home = ["HTML", "RSS", "llms"]
  page = ["HTML", "md"]

That gives the site root a /llms.txt and every page an index.md sibling. In layouts/index.llms.txt, use .RawContent rather than .Content. The first is your original Markdown, the second is rendered HTML, and rendering Markdown to HTML just to strip the tags again defeats the point. Native support is still an open feature request upstream, so for now this is the path.

Eleventy is a permalink and a loop:

---
permalink: /llms.txt
eleventyExcludeFromCollections: true
---
# {{ metadata.title }}

> {{ metadata.description }}

## Blog
{% for post in collections.post | reverse -%}
- [{{ post.data.title }}]({{ metadata.url }}{{ post.url }}): {{ post.data.description }}
{% endfor %}

Group by tag, cap the list, and actually use Optional

This is where generating beats crawling by a mile. A crawler sees your nav. Your build sees data.tags, so H2 sections come free:

const byTag = Object.groupBy(posts, (p) => p.data.tags[0] ?? 'Blog');

Cap the list while you're in there. The token spread across published files is wide: per the llms.txt directory, Anthropic's file is 892 tokens and Cursor's is about 2K, while Cloudflare's is 49K and the Vercel AI SDK's llms.txt alone is 293K. The median file on a curated 219-host panel is 14.0 KB. Fifty recent posts is plenty. The archive is what the sitemap is for.

Then there's the Optional H2, which the spec defines as material an agent can skip when its context budget is tight, and which almost nobody ships. Changelog entries, tag index pages, the old posts you keep for URL stability. One if statement in the mapper. Free signal, and you're the one deciding what gets dropped instead of a truncation heuristic.

The .md sibling is the half worth building

The other half of the proposal is per-page clean-markdown mirrors: the same URL with .md appended, or index.html.md for extensionless URLs. It's more work and far less widely implemented. It's also the part with a measurable payoff. Fern's number for HTML-to-markdown conversion is over 90% reduction in token consumption.

It also has a named reader. Coding agents already on your docs page, right now, burning context on your nav and cookie banner.

Astro's route-extension trick does this too. The filename extension before .ts becomes the built URL, so [slug].md.ts gives you /blog/my-post.md:

// src/pages/blog/[slug].md.ts
export async function getStaticPaths() {
  const posts = await getCollection('blog', ({ data }) => !data.draft);
  return posts.map((post) => ({ params: { slug: post.slug }, props: { post } }));
}

export const GET: APIRoute = ({ props }) =>
  new Response(props.post.body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  });

getStaticPaths is required because Astro needs to know at build time which pages to emit. Add <link rel="alternate" type="text/markdown" href="..."> in your head and an agent can find it without guessing.

Generate llms.txt next to your sitemap and fail the build on dead links

Same input, same trigger. If sitemap.xml generation lives in your build, llms.txt belongs in the same file, reading the same collection query. Two exports, one source.

A stale map is worse than no map, and the industry's answer to staleness is a re-crawl subscription. A generated file can't go stale. It's regenerated on every deploy or it isn't deployed.

Then treat a dead link in it like a dead link on the site. The community validators check the basics (root location, H1 immediately followed by a blockquote, standard [Title](URL) links, plus sampling for 404s), but you don't need a service for that. You have the URL list in memory at build time. Assert every entry resolves to a page you also generated, and throw if one doesn't.

What the evidence actually says

Here's the part most guides leave out, and it's the reason this post is about build steps rather than strategy.

A server-log study across roughly 900 domains, running September 2025 to April 2026, counted every request for llms.txt-family paths. The total is small. The breakdown is the number worth reading.

0 Requests to llms.txt files from a verified AI-lab crawler, out of 1,227 logged across ~900 domains in seven months.: Server logs, 4 Sep 2025 – 13 Apr 2026 (191 days): 1,227 requests to /llms.txt and variants, ~6/day. Requests touched 107 of ~900 domains, so roughly 88% saw none (derived from the study's own counts). The largest single requester was commercial data aggregator Dataprovider.com at 794 requests; no OpenAI, Anthropic or Google bots appeared. One operator's estate — at far larger scale, Ahrefs found 97% of llms.txt files got zero requests. Source: seekio.pl, 2026. (AI bots ignore LLMS.txt but scan the internet at scale – 2 studies, 1 conclusion)

The second-largest requester was ordinary browser traffic from humans checking the file existed.

A separate citation study across roughly 300,000 domains, published November 2025, ran correlation tests plus a gradient-boosted model and found that removing the llms.txt feature improved the model's accuracy. The file behaved as noise.

Google has said it twice. John Mueller, June 2025: "FWIW no AI system currently uses llms.txt." Search Relations later argued a self-reported manifest can't differentiate sites. Meanwhile Chrome shipped a Lighthouse audit for it in May 2026, filed under a new agentic browsing category alongside WebMCP. Two teams, one file, opposite conclusions. There's also a counter-datapoint worth one line: log analysis posted by Ray Martinez showed OpenAI pulling llms.txt on a few of his sites every fifteen minutes, apparently checking freshness.

So: the honest read is that adoption is cohort-shaped and the return is unproven. On a curated developer-weighted panel measured this month, 51.8% of reachable hosts ship one. In the Tranco top 1,000, it's 8.7%.

Which is exactly the argument for generating it. An afternoon of hand-curation plus a monthly maintenance chore does not survive those numbers. Forty lines wired to a collection query does, comfortably, because after the first deploy the ongoing cost is zero and the option value is free.

Ship a clean 404, not a 500

If you decide against it, or you remove it later, return a plain 404. The Lighthouse audit only flags a page when the server returns an error fetching /llms.txt; a clean 404 marks it Not Applicable. A 500 or a timeout is the one outcome that costs you something.

If you do ship it, make it findable. Mirror at /.well-known/llms.txt and announce it with a Link: </llms.txt>; rel="llms-txt" header, or X-Llms-Txt: /llms.txt. An agent shouldn't have to guess the path.

Where this lands if your posts arrive as pull requests

This whole approach has one dependency: frontmatter that stays consistent. The generator reads title, description, date and tags, so a post that ships with a missing description puts a bare link in your file, and a post with a novel field name puts nothing at all.

That's the constraint Contentcron writes to. Every article is full MDX with frontmatter inferred from your existing posts — field names, date formats, slug style — so the route handler you wrote six months ago keeps returning a correct file without anyone touching it. And because the article lands as a PR, the llms.txt diff shows up in the same review as the prose. A machine-readable file is one of the few diffs that's genuinely pleasant to read, and reviewing content the way you review code means you see it before it's live.

An article page toggled between rendered preview and raw MDX source, showing frontmatter matched to the repo's existing posts

If your blog is a folder of markdown and you've been putting this off because the tooling looked like a subscription, it isn't. It's a route file and a .map(). Write it once, wire it to the build. Then stop thinking about it.

← All posts