Skip to content

feat(llm): make Anthropic prompt caching configurable - #1697

Open
OUAO-FRANK wants to merge 5 commits into
alibaba:mainfrom
OUAO-FRANK:fix/700-anthropic-prompt-caching
Open

OUAO-FRANK wants to merge 5 commits into
alibaba:mainfrom
OUAO-FRANK:fix/700-anthropic-prompt-caching

Conversation

@OUAO-FRANK

Copy link
Copy Markdown
Contributor

Summary

  • add an llm.prompt_caching setting for Anthropic-compatible endpoints
  • allow a provider-level prompt_caching override
  • omit all cache_control breakpoints when prompt caching is disabled
  • add resolver, configuration, and request-builder regression tests

This keeps prompt caching enabled by default for backward compatibility while allowing gateways that reject Anthropic's cache_control field to opt out.

Validation

  • git diff --check passed
  • Go tests could not be run locally because the Go toolchain is unavailable in this environment

The implementation and tests were reviewed manually. This PR was developed with GPT-5.6 through Codex, using Git and GitHub CLI; I reviewed the complete diff and validation results before submission.

Fixes #700

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 1 issue(s) in this PR.

  • ✅ Successfully posted inline: 1 comment(s)

Comment thread internal/llm/client.go Outdated
Comment on lines 1632 to 1638
if c.promptCachingEnabled() && len(systemBlocks) > 0 {
systemBlocks[len(systemBlocks)-1].CacheControl = anthropic.NewCacheControlEphemeralParam()
params.System = systemBlocks
}
if len(tools) > 0 {
if c.promptCachingEnabled() && len(tools) > 0 {
tools[len(tools)-1].OfTool.CacheControl = anthropic.NewCacheControlEphemeralParam()
params.Tools = tools

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · critical
Critical bug: When promptCachingEnabled() returns false, params.System and params.Tools are never assigned, which means system prompts and tool definitions are silently dropped from the API request. Previously (before this change), these were always set when non-empty regardless of caching.

The fix should separate the cache-control mutation from the assignment. For example:

if len(systemBlocks) > 0 {
    if c.promptCachingEnabled() {
        systemBlocks[len(systemBlocks)-1].CacheControl = anthropic.NewCacheControlEphemeralParam()
    }
    params.System = systemBlocks
}
if len(tools) > 0 {
    if c.promptCachingEnabled() {
        tools[len(tools)-1].OfTool.CacheControl = anthropic.NewCacheControlEphemeralParam()
    }
    params.Tools = tools
    // ... tool choice logic
}

The same pattern applies to the messages block below — while messages are always set via the struct literal, the cache_control breakpoint on the last message should also be guarded without preventing any essential assignment.

@NanaseInori NanaseInori left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resolver correctly distinguishes unset from false, and the client gates all three automatically added Anthropic cache breakpoints.

One behavior to adjust or clarify: applyCreateCustomProvider explicitly writes PromptCaching=true for every new Anthropic custom provider. With llm.prompt_caching=false, creating a provider through the TUI therefore re-enables caching, although the user never chose a provider-specific override. Leaving the pointer nil would inherit the global setting and still retain the historical enabled default when no setting exists.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unrecognizedProperty cache_control

2 participants