DX-2659: Trim llm files#697
Merged
Merged
Conversation
Apply lossless presentational cuts and replace SDK command reference
pages with one-line link bullets.
Body normalization changes (in normalizeMdxBody):
- Stop injecting `theme={"system"}` into code fences and strip any
pre-existing instance. It's Mintlify-specific styling with no value
to an LLM.
- Drop `<Frame>` / `<Frame caption="…">` / `</Frame>` wrapper lines
(both standalone and inline forms). The image inside survives.
- Strip image source URLs:
`` → `![alt]()`
`<img … src="…" …>` → `<img … >`
AI consumers don't follow image links, and badge / asset URLs are
often hundreds of bytes each. Code-fence contents are preserved.
Output framing changes (in writeLlmsFullTxt):
- Per-page block reduced to one blank line on each side (was three
after Source and two before next page).
- A global `collapseBlankRuns` pass squashes any 2+ blank-line run to
one. Affects both our framing and body-internal padding.
SDK command reference pages (~390 pages under `*/sdks/<lang>/commands/`)
no longer get full bodies in llms-full.txt. Instead, each appears as a
single `- [TITLE](url.md): description` bullet at its natural sort
position, matching llms.txt's bullet format. The bundle still announces
every command (title, description, URL); AI can follow the link if it
needs the full reference. Saves ~19K lines without losing discoverability.
llms.txt remains byte-identical to upstream.
Extend FULL_TXT_SKIP to also match any path segment named `quickstarts`,
`examples`, or `recipes`. Affected pages now emit a single
`- [title](url.md): description` bullet at their sort position instead
of a full page body, matching how SDK command pages were already handled.
Targets:
- framework quickstarts (redis/quickstarts/*, qstash/quickstarts/*,
workflow/quickstarts/*)
- SDK examples (qstash/sdks/ts/examples/*, qstash/sdks/py/examples/*)
- long tutorial pages (redis/search/recipes/*, workflow/examples/*,
qstash/recipes/*)
These are code-heavy walkthroughs — high token cost but low reference
value when pre-bundled. AI consumers still see title + description +
URL for every one and can fetch the full page on demand.
llms-full.txt size:
before: 66,215 lines / 2.05 MB
after: 52,716 lines / 1.66 MB
cumulative reduction from upstream-matched output: 40% lines / 35% bytes
llms.txt remains byte-identical to upstream.
Two related changes:
1. Both llms.txt and llms-full.txt now start with the contents of
`_snippets/redis/start-redis-snippet.mdx` — an AI-targeted tip
telling agents how to grab a free scratch Redis DB. Surfaced at
the top of the bundle so it's the first thing a consuming agent
sees.
2. `<Snippet file="..." />` references inside MDX bodies are now
resolved during normalization. Both the single-line form and the
line-broken form
<Snippet
file="redis/ioredisnote.mdx"
/>
are matched; the referenced file from `_snippets/` is inlined in
place of the tag. Expansion runs iteratively (up to 5 passes) so
nested Snippet references resolve. Missing files leave the
original tag in place so breakage stays visible.
Previously these tags were emitted to llms-full.txt verbatim and would
have been useless to an AI consumer that can't fetch
`_snippets/<file>` (it isn't a navigable URL).
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trims llm files added in #696