Making the documentation findable.

ApiGlow renders in the browser, under hash routes. Crawlers handle that badly. AI crawlers — GPTBot, ClaudeBot, PerplexityBot — fetch HTML and run no JavaScript. Google runs JavaScript but treats every #/… route as the same URL, so it can index the home view and nothing below it.

Two mechanisms answer this: the document head every install manages at runtime, and the static mirror the apiglow bake CLI writes to disk.

What every install does

The app updates the document head on each route change, with no configuration:

• <title> — {route} — {API title}: the operation summary (or GET /path), the docs page title, the workflow name. The app owns the title once it has booted.
• Meta description — plain text derived from the route’s own content, capped around 160 characters. Created if the host page has none, updated in place otherwise.
• JSON-LD — one block per route: APIReference for endpoints, TechArticle for docs pages and scenarios, WebSite on the home view. Routes that no type fits, such as the audit and the first-call page, emit none.

There is no runtime canonical. Under hash routing every route shares one server URL, and crawlers strip the fragment, so a per-route canonical would name the same address every time. Canonicals exist in the baked snapshots.

Staying out of the index

seo.index: false injects <meta name="robots" content="noindex"> before first paint:

{ "seo": { "index": false } }

The key is root-only in multi-spec: one page is served at one URL, and a crawler reads that HTML without choosing a spec.

To cover files the meta tag cannot reach — the schema itself, baked .md mirrors — send an X-Robots-Tag: noindex response header for them.

A noindex is a request to well-behaved crawlers, not a protection. Documentation that must not be read needs an auth wall.

The bake: a static mirror from one command

apiglow bake is a companion CLI for the docs author. It ships in the npm package ("bin": { "apiglow": "dist/cli.js" }, Node ≥ 24 — the same binary runs apiglow audit in CI) and reads the same JSON config your host page inlines:

apiglow bake --config apidoc.config.json --site-url https://docs.example.com/ --out public/

It writes one file tree, deployed next to the app page by whatever already deploys your site:

sitemap.xml
llms.txt
llms-full.txt
overview.html
op/{operationId}.html        op/{operationId}.md
page/{slug}.html             page/{slug}.md
scenario/{scenarioId}.html   scenario/{scenarioId}.md
scenario/{scenarioId}.arazzo.json

Each route gets an HTML snapshot: canonical to itself, a <link rel="alternate" type="text/markdown"> to its sibling, JSON-LD, no scripts, and a link back into the interactive app. Beside it sits a Markdown mirror, produced by the same generator as the in-app “Copy page”, with {{var}} left literal. Each scenario also publishes its Arazzo recipe, the file a CI runner executes unchanged.

overview.html is the crawler’s entry point and links to every snapshot. In multi-spec, everything nests under s/{specId}/…, and the three root files cover all specs.

This is not a prerender and not static-site generation: no DOM runs in Node, and the reader’s install stays one script tag with no build step. It is the app’s export layer written to disk. When a snapshot and the app disagree, the generator is what gets fixed.

What agents get out of it

Without a bake, llms.txt and llms-full.txt exist only as in-browser downloads whose links are hash routes, which an agent cannot fetch. Baked, both are served files: llms.txt links the .md mirrors per the llmstxt.org convention, and each llms-full.txt section carries a Source: line pointing at its served page. The agent surface guide covers what is inside the files.

Running it in CI

The bake resolves its two kinds of addresses differently. Files the config names are read from disk, relative to the config file — a leading / included, since only a URL with a scheme is fetched. Every URL written into a generated file resolves against --site-url. That is what lets the bake run before anything is deployed.

What the bake degrades, and what it stops on

• A schema that will not load ends the run. Everything else derives from it.
• A docs page carried by contentId lives in host-page HTML that no Node process sees. It is named in the warnings and dropped from the map, the sitemap and the tree.
• Raw HTML inside Markdown is escaped rather than rendered in snapshots — there is no DOM sanitizer in Node. A baked page can render less richly than the app.
• A scenario whose schema is inlined (openapi.spec) gets no Arazzo recipe, because the recipe would name a source no runner can fetch. This is warned, not silent.
• The reader’s user overlay is never baked.
• The sitemap carries no <lastmod>: the generator has no clock, and a bake date would describe the build rather than the documentation.
• The bake emits no robots.txt and no index.html. Those stay yours.

Re-run the bake whenever the schema, the config, the prose pages or the scenarios change — in practice, from the CI job that deploys the site.