Developer docs

Generate and publish llms.txt for a site.

Use the app to turn a public website into a spec-compliant llms.txt file, review what was included, and serve the latest version from a stable URL.

llms.txt basics

What an llms.txt file does

An llms.txt file is a plain-text map for language models. It gives models a concise description of the site and a curated list of URLs that are worth reading, such as docs, API references, changelogs, pricing pages, or support material.

The file does not replace your sitemap, robots.txt, or public docs. It gives LLMs a smaller entry point that explains which pages matter and why, so a model can choose better sources before answering questions about your product.

You can find the official spec here

Quickstart

Generate a file for any public site

Start with the homepage or the canonical docs URL for the site you care about. The app discovers crawlable pages, generates the file, validates it against the spec, and takes you to a result page with the rendered output.

For developer workflows, the hosted URL is usually the important artifact. You can paste it into a pull request, add it to release checklist documentation, or proxy it from your own /llms.txt path when you are ready to serve it from the site.

Paste a public URL

Start from the homepage or a canonical docs URL. The crawler normalizes the origin, discovers crawl candidates, and starts a bounded run.

Watch crawl progress

Progress moves through discovery, crawling, generation, and validation so it is clear whether the system is still gathering pages or publishing the file.

Review the result

The result page shows the rendered llms.txt, copy and download actions, the stable hosted URL, and the page inventory used to produce the file.

Keep it current

Monitoring can regenerate the file when site structure or page metadata changes. Version history records what changed and supports inline diffs.

Connect your domain

Proxy /llms.txt from infrastructure you control

The hosted URL is the source of truth for generated files. To make https://example.com/llms.txt resolve on a real domain, the site owner must add a proxy, rewrite, or route in their DNS, CDN, hosting platform, or origin server.

This app cannot install that route unless it controls the site. For v1, connecting a domain means copying the hosted URL from the result page and installing the matching snippet in the platform that serves the site.

Code starts for common setups:

Cloudflare Worker

Use a Worker route on the site owner's zone, such as example.com/llms.txt, when Cloudflare controls the domain.

worker.js

const SOURCE =
  "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt";

export default {
  async fetch() {
    const upstream = await fetch(SOURCE);

    return new Response(upstream.body, {
      status: upstream.status,
      headers: {
        "content-type": "text/markdown; charset=utf-8",
        "cache-control": "public, max-age=300",
      },
    });
  },
};

Vercel rewrite

Add a rewrite when the site is deployed on Vercel and /llms.txt can be routed at the platform layer.

vercel.json

{
  "rewrites": [
    {
      "source": "/llms.txt",
      "destination": "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt"
    }
  ]
}

Next.js route handler

Use a route handler when you want cache control in application code or when platform rewrites are not available.

app/llms.txt/route.ts

export async function GET() {
  const upstream = await fetch(
    "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt",
    { next: { revalidate: 300 } },
  );

  return new Response(upstream.body, {
    status: upstream.status,
    headers: {
      "content-type": "text/markdown; charset=utf-8",
      "cache-control": "public, max-age=300",
    },
  });
}

Netlify proxy

Use a forced 200 rewrite so the visitor stays on the site's own /llms.txt URL.

netlify.toml

[[redirects]]
from = "/llms.txt"
to = "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt"
status = 200
force = true

Nginx

Proxy the exact file path from the origin server or a reverse proxy in front of it.

nginx.conf

location = /llms.txt {
  proxy_pass https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt;
  proxy_set_header Host our-api.example.com;
  proxy_ssl_server_name on;
  add_header Cache-Control "public, max-age=300" always;
}

Apache

Enable mod_proxy and map the exact /llms.txt path to the generated hosted URL.

.htaccess or vhost.conf

ProxyPass "/llms.txt" "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt"
ProxyPassReverse "/llms.txt" "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt"
Header set Cache-Control "public, max-age=300"

Using the application

Features developers can build around

The app is designed for a practical loop: generate, inspect, publish, and keep the file current. You do not need an account to generate a file, and you can decide later whether to use the hosted URL directly or bring the content into your own deployment.

Stable hosted URL

Every generated site gets a shareable URL that always resolves to the latest validated llms.txt. Use it from docs, deployment checks, reverse proxies, or internal runbooks.

curl "https://our-api.example.com/sites/https%3A%2F%2Fexample.com/llms.txt"

Automatic updates

Monitoring starts on by default. When the site structure or page metadata changes, the app can crawl again, publish a new version, and keep the hosted URL pointed at the current file.

stable URL: latest validated version
monitoring: crawl when the site changes
history: compare previous versions

Reviewable inventory

The result page shows which pages were discovered and how they were classified before the file was rendered, so developers can catch missing docs, noisy pages, or thin generated output.

docs pages: included
reference pages: included
low-signal URLs: optional or omitted

Version history

Each publish is stored as a version. Use the timeline and diffs to see what changed between crawls before you wire the file into a production docs workflow.

v3: current
v2: 2 pages added, 1 modified
v1: first generated file