feat(ui): add llms.txt generation for npm packages#1382
feat(ui): add llms.txt generation for npm packages#1382lukeocodes wants to merge 22 commits intonpmx-dev:mainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds end-to-end LLM documentation support: new shared types (AgentFile, LlmsTxtResult) and a re-export in shared/types/index.ts; server utilities to discover and fetch agent files, assemble llms.txt/llms_full.txt and package README content; a new middleware (server/middleware/llm-docs.ts) to serve package, org and root llms routes and raw README .md; vitest test alias for Possibly related issues
Suggested reviewers
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
lukeocodes
changed the title
|
@danielroe when merging please add @BYK as a co-author from #151 |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
server/middleware/llm-docs.ts (1)
35-135: Split the middleware into smaller helpers.
This handler covers multiple routes and behaviours in one block and is well over the “~50 lines” guideline. Consider extracting.mdhandling, root handling, org handling, and package parsing into dedicated functions.As per coding guidelines: Keep functions focused and manageable (generally under 50 lines).
Add AgentFile and LlmsTxtResult interfaces for llms.txt generation and export from the shared types barrel.
Add discoverAgentFiles, fetchAgentFiles, generateLlmsTxt, and handleLlmsTxt orchestrator for llms.txt generation from npm packages.
Serve llms.txt at /api/registry/llms-txt/[...pkg] following existing registry API patterns with cached event handler and SWR.
Cover discoverAgentFiles, fetchAgentFiles, and generateLlmsTxt including root files, directory scanning, graceful failures, scoped packages, and full/minimal output generation.
Use Nitro server routes at /package/.../llms.txt instead of an API route with middleware rewriting. Single handler re-exported across four route files for unscoped, scoped, and versioned URL patterns.
…txt content Add createPackageLlmsTxtHandler factory for DRY route creation. handleLlmsTxt now accepts includeAgentFiles option to control whether agent instruction files are included (llms_full.txt) or omitted (llms.txt). Add handleOrgLlmsTxt for org-level package listings and generateRootLlmsTxt for the root /llms.txt discovery page. Simplify route handlers to single-line factory calls.
Add server middleware to handle llms.txt routes that Nitro's radix3 file-based router cannot resolve (parameterized intermediate segments don't match literal children). Handles versioned package paths, org-level package listings, and root /llms.txt discovery page. Remove broken versioned route files and add llms_full.txt routes.
Extend canonical redirect regexes with optional /llms.txt and /llms_full.txt suffix capture groups so shorthand URLs like /nuxt/llms.txt redirect to /package/nuxt/llms.txt. Add explicit /llms.txt root path skip to prevent it matching as a package name.
Add ISR rules for llms_full.txt and root /llms.txt routes in nuxt.config.ts. Add #server alias to vitest config for resolving server utility imports in unit tests.
Test route pattern inclusion, example links, base URL substitution, and trailing newline for the root /llms.txt discovery page output.
Vercel ISR glob rules (/package/**/llms.txt) create catch-all serverless functions that intercept requests before Nitro's file-based routes can resolve them, breaking scoped packages and versioned paths. Move all llms.txt/llms_full.txt handling into the middleware, remove ISR route rules, and delete file-based route files.
Fix strict TypeScript errors: add fallback for split()[0] possibly undefined, narrow regex match group types, use non-null assertions for Record lookups after in-check, and use Nuxt's auto-generated Packument type instead of @npm/types import.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Versioned .md paths (e.g. /package/nuxt/v/3.16.2.md) conflict with Vercel's ISR route rules which match /package/:name/v/:version and intercept the request before middleware can handle it. Keep .md for latest-only (unscoped and scoped).
lukeocodes
force-pushed
the
feat/llms-txt
branch
from
80b1fe1 to
456017b
Compare
Use hyphenated naming convention (llms-full.txt) to match the community standard used by Anthropic, GitBook, Fern, and others.
|
@danielroe @patak-cat ticky ticky bang bang? <3 |
Add <link rel="alternate"> tags to HTML head for LLM-friendly document discovery on homepage, org pages, and package pages.
|
From what I can see, there is no llms.txt for a packages API docs, right? |
Nope, that seems out of scope for a initial version though |
|
fair enough — though I would love to see that in the future |
yeh me too, this felt a little invasive for me for a single PR so i thought i'd limit it much more than what i've done |
| const full = path.endsWith('/llms-full.txt') | ||
| const suffix = full ? '/llms-full.txt' : '/llms.txt' | ||
|
|
||
| // Root /llms.txt |
There was a problem hiding this comment.
a thing that I personally dislike is the amount of comments that explain very simple statements.
If a comment is needed, the code should be made more readable.
|
I just noticed this happening on paths that I haven't opened before Screencast.From.2026-03-05.17-49-51.webm
|
3 participants
Summary
Adds
llms.txtandllms_full.txtsupport across all package URL patterns.Supported
llms.txtroutes/llms.txt/package/nuxt/llms.txt/package/nuxt/v/3.15.0/llms.txt/package/@deepgram/sdk/llms.txt/package/@deepgram/sdk/v/4.0.0/llms.txt/package/@deepgram/llms.txt/package/nuxt/llms_full.txt/nuxt/llms.txt→/package/nuxt/llms.txt/nuxt/v/3.15.0/llms.txt→/package/nuxt/v/3.15.0/llms.txtllms.txt— README + metadata onlyllms_full.txt— README + metadata + agent instruction files (CLAUDE.md, .cursorrules, etc.)Supported
.mdroutes/package/nuxt.md/package/@deepgram/sdk.md/nuxt.md→/package/nuxt.md.md— README onlyTest plan
/llms.txtreturns 200 withtext/markdowncontent typellms_full.txtincludes agent instruction files,llms.txtdoes not/nuxt/llms.txt) redirect 301 to canonical paths