Skip to content

feat: derive Anthropic max_tokens from context budget instead of hardcoding 4096#559

Merged
nashsu merged 2 commits into
nashsu:mainfrom
AndrewDongminYoo:feat/anthropic-max-tokens-from-budget
Jul 15, 2026
Merged

feat: derive Anthropic max_tokens from context budget instead of hardcoding 4096#559
nashsu merged 2 commits into
nashsu:mainfrom
AndrewDongminYoo:feat/anthropic-max-tokens-from-budget

Conversation

@AndrewDongminYoo

Copy link
Copy Markdown
Contributor

What

getProviderConfig() now derives the default Anthropic max_tokens from the configured context window instead of letting the wire request fall back to 4096.

This completes a follow-up that context-budget.ts already documents as pending:

// The response reserve is a "passive" reservation: we don't pass
// `max_tokens: responseReserve / 3` to the LLM (yet — that's a
// follow-up). We just refuse to fill above (maxCtx - responseReserve)

Why

computeContextBudget() sizes a 15% response reserve so the model has room to answer, but that reserve was passive — it only capped how full we packed the context. The actual Anthropic request still defaulted to max_tokens: 4096 regardless of the configured context window, so a large context could be assembled but the response was silently clipped to 4096 tokens.

How

  • Compute anthropicBudgetTokens from the same responseReserve (chars → tokens at ~3 chars/token), capped at 16 384 to stay within typical per-request limits.
  • Pass it as the default max_tokens for all three Anthropic-wire providers (anthropic, minimax, custom/anthropic_messages).
  • Explicit caller overrides (e.g. max_tokens: 300 for routing decisions) are spread after the default, so they still take precedence.
  • Update the now-outdated context-budget.ts doc comment to reflect that the follow-up is done.

Effect on a 200K-char context: default max_tokens 4096 → 10 240.

Testing

  • src/lib/__tests__/llm-providers.test.ts — 50 pass / 0 fail (updated to assert the derived default and that overrides win).

…coding 4096

The response reserve computed by computeContextBudget() was a passive
reservation — it sized the context budget correctly but never enforced
a matching max_tokens on the wire. Anthropic API calls defaulted to 4096
tokens regardless of the configured context window.

Now getProviderConfig() computes anthropicBudgetTokens from the same
15% response reserve (responseReserve / 3 chars-per-token, capped at
16 384) and passes it as the default max_tokens for all three Anthropic-
wire providers (anthropic, minimax, custom/anthropic_messages). Explicit
overrides from callers (e.g. max_tokens: 300 for routing decisions) are
spread after the budget value so they still take precedence.

Effect on a 200K-char context: default max_tokens 4096 → 10 240.
The passive-reservation comment said passing max_tokens was an
unfinished follow-up; getProviderConfig() now derives it from
responseReserve, so update the doc to match.
@nashsu
nashsu merged commit d189150 into nashsu:main Jul 15, 2026
3 checks passed
@AndrewDongminYoo
AndrewDongminYoo deleted the feat/anthropic-max-tokens-from-budget branch July 15, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants