Repository navigation
Conversation
…nterface with scoped, identity-aware key bindings (#7176) ## Summary Session stickiness was implemented directly inside `selectKeyFromProviderForModel` as an inline block of KV store logic. This PR extracts that logic into a `SessionAffinity` interface, ships a default implementation (`keyAffinity`) that preserves the existing behaviour, and allows deployments to register their own policy through `BifrostConfig.SessionAffinity`. ## Changes - Introduced `schemas.SessionAffinity` interface with a single `ResolveKey` method, added to `BifrostConfig` as an optional field. - Extracted all session stickiness logic from `selectKeyFromProviderForModelWithPool` into `core/sessionaffinity.go` as `keyAffinity`, the default implementation installed when `BifrostConfig.SessionAffinity` is nil. - `keyAffinity` scopes session bindings to the virtual key and user from the request's grant identity, so a session ID reused by a different caller does not share bindings. The old implementation scoped only by provider, model, and session ID. - Session state keys are now built by `sessionStateKey`, which hashes the session ID, virtual key ID, user ID, provider, and model together under a `session:v2:` prefix. The old `buildSessionKey` helper and `hashSHA256` utility are removed. - `getCachedKeyFromStore` is removed; its logic lives in `keyAffinity.lookup` and `sessionStateString`. - The `sonic` import is moved from `bifrost.go` into `sessionaffinity.go` where it is actually used. - Added `core/sessionaffinity_test.go` covering key scoping, TTL handling, bind-then-reuse, stale binding replacement, decline conditions, concurrent first-request convergence, and per-caller isolation. - Updated the session stickiness documentation to mention `BifrostConfig.SessionAffinity` and the identity-scoped binding behaviour. ## Type of change - [ ] Bug fix - [x] Feature - [x] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test ```sh go test ./core/... -run TestSessionAffinity go test ./core/... -run TestSelectKeyFromProviderForModel_SessionStickiness go test ./core/... -run TestSelectKeyFromProviderForModel_NoStickinessWithoutSessionID go test ./core/... ``` To exercise a custom policy, implement `schemas.SessionAffinity` and pass it as `BifrostConfig.SessionAffinity`. Requests carrying `BifrostContextKeySessionID` will be routed through `ResolveKey`; returning `ok=false` falls through to ordinary key selection. ## Breaking changes - [x] Yes - [ ] No Session state keys written by the previous implementation (`session:<provider>:<hashed-session-id>:<hashed-model>`) are not read by the new implementation (`session:v2:key:<hash-of-identity+session+provider+model>`). Existing KV entries will be ignored and will expire naturally; sessions will rebind on their next request. ## Security considerations Session IDs, virtual key IDs, user IDs, provider names, and model names are all hashed before being written to the KV store key, so no caller-supplied text or PII appears in store keys. Bindings are scoped to the grant identity attributed to the request, preventing session ID collisions across callers. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…h `x-bf-session-affinity` header opt-out (#7177) ## Summary Session affinity previously bound a session to a key at selection time, before any request was served. This meant the binding could be written by a request that ultimately failed, and the session had no concept of which provider or route served it — only which key was selected. This PR rewrites session affinity so that bindings are written only when a request is successfully served, introduces route-level affinity (provider stickiness alongside key stickiness), and exposes the full lifecycle to custom implementations through a revised `SessionAffinity` interface. ## Changes - **`SessionAffinity` interface extended**: Added `ResolveRoute` and `Observe` methods. `ResolveRoute` is called after routing hooks run and can reorder the provider/fallback chain. `Observe` is called once per request with the route and key that served it, or the error it ended in. Bindings are now written only from `Observe`, never from `ResolveKey`. - **Route-level stickiness**: The shipped `sessionAffinity` (formerly `keyAffinity`) now tracks which provider last served a session for a given requested route. On subsequent requests, if that provider appears anywhere in the chain, it is moved to the front. If it is already first, the chain is left unchanged and no routing decision is recorded. - **Bindings written on outcome, not on selection**: `ResolveKey` no longer selects a key for an unbound session or writes to the store. It only answers when a binding already exists and the bound key is still eligible. `Observe` writes the route and key bindings after a request is served, using `SetNX` for a first binding (so concurrent first requests converge on whichever finishes first) and `Set` for a refresh or move. - **`IsSessionAffinityActive` guard**: A new `BifrostContextKeySessionAffinity` context key (`x-bf-session-affinity` header) lets callers opt individual requests out of affinity. `IsSessionAffinityActive` checks both that a session ID is present and that affinity has not been switched off. Core consults this before every affinity call, so custom implementations never see requests that do not take part. - **`resolveSessionRoute` and `observeSessionOutcome`**: Two new internal helpers on `Bifrost` wire the new interface methods into `handleRequest` and `handleStreamRequest`. `resolveSessionRoute` runs after pre-request hooks and applies the chain the affinity returns. `observeSessionOutcome` is deferred and fires once with the served route and key, or the error. - **Key selection guard**: `selectKeyFromProviderForModelWithPool` now checks `IsSessionAffinityActive` before calling `ResolveKey`, so requests without a session or with affinity switched off never reach the affinity implementation. - **`RoutingEngineSessionAffinity` constant**: Added to the routing engine log vocabulary so the request trail records when a session moved a provider to the front or reused a key. - **Tests updated throughout**: `TestSelectKeyFromProviderForModel_SessionStickiness` and the no-rotation test are updated to reflect the new lifecycle. New tests cover `ResolveRoute`, `Observe`, `IsSessionAffinityActive`, `resolveSessionRoute`, `observeSessionOutcome`, and the key-pool guard. ## Type of change - [ ] Bug fix - [x] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... -run TestSessionAffinity go test ./core/... -run TestSelectKeyFromProviderForModel_SessionStickiness go test ./core/... -run TestResolveSessionRoute go test ./core/... -run TestObserveSessionOutcome go test ./core/... -run TestKeyPoolAsksAffinityOnlyForRequestsThatTakePart go test ./... ``` To exercise route affinity end-to-end, send requests with `x-bf-session-id` set and confirm that after the first request is served, subsequent requests to the same session are routed to the same provider even when the routing chain would otherwise propose a different one. To opt a request out, send `x-bf-session-affinity: off` and confirm the session binding is neither consulted nor updated for that request. ## Breaking changes - [x] Yes - [ ] No The `SessionAffinity` interface has two new required methods: `ResolveRoute` and `Observe`. Any custom implementation registered via `BifrostConfig.SessionAffinity` must implement both. `ResolveKey` no longer receives unbound sessions and will not be called for requests that do not carry a session ID or have affinity switched off. ## Related issues ## Security considerations The `x-bf-session-affinity` header is parsed and validated before being applied; unrecognised values are ignored with a warning. No session state is written for failed requests, reducing the risk of poisoning a session binding through a deliberately induced failure. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…affinity control and document provider-level session binding behavior (#7178) ## Summary Introduces a `BifrostContextKeySessionAffinity` context key (and corresponding `x-bf-session-affinity` HTTP header) that lets callers opt individual requests out of session-driven routing. Previously, session stickiness was all-or-nothing once a session ID was present; now a single request can bypass both provider-level and key-level affinity without affecting the session state for other requests. Documentation is updated to reflect the two-level affinity model (provider ordering and key selection) and the UI gains a `session-affinity` routing engine entry with its own label, color, and icon. ## Changes - Added `BifrostContextKeySessionAffinity` (`x-bf-session-affinity`) as a new `bool` context key; `on` (default) preserves existing behavior, `off` skips affinity for that request and writes no session state. - Clarified session stickiness docs to describe the two-level model: provider affinity (tried first when routing still offers it) and key affinity (used while the key remains eligible), with bindings written only on success and refreshed by every served request. - Updated session TTL description from "session-to-key binding" to "session bindings" to reflect that bindings now cover both provider and key. - Added `session-affinity` as a recognized routing engine in the UI, with an amber color scheme and a `Link2` icon, so it appears correctly in routing logs and engine-used badges. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [x] UI (React) - [x] Docs ## How to test ```sh # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` To validate the affinity switch: 1. Make a request with a session ID and confirm the session binds to a provider and key. 2. Make a subsequent request with the same session ID and `x-bf-session-affinity: off`; confirm routing logs do not include the `session-affinity` engine and the bound provider/key is not preferred. 3. Make a follow-up request with `x-bf-session-affinity: on` (or omit the header); confirm the binding from step 1 is still in effect (not overwritten by step 2). ## Screenshots/Recordings The `session-affinity` engine badge now appears in routing logs with an amber background and a chain-link icon, consistent with the other engine badges. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No new secrets or PII surfaces. The affinity switch is a per-request routing hint with no elevated privilege; disabling it for a request simply falls back to standard routing and key selection. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [x] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…lpers; unify complexity session keying under shared session state (#7179) ## Summary Several internal session affinity helpers were private to the `core` package, preventing external components (such as plugins) from building on the same session-scoping and key-derivation logic. This PR promotes those helpers to exported symbols and migrates the complexity routing plugin to use them, eliminating a duplicate, divergent implementation of session key scoping. ## Changes - `newSessionAffinity`, `sessionStateKey`, `sessionStateString`, `sessionTTLFromContext`, and `routeString` are now exported as `NewSessionAffinity`, `SessionStateKey`, `SessionStateString`, `SessionTTLFromContext`, and `RouteStateValue`. - `ParseRouteState` is introduced as the exported inverse of `RouteStateValue`, replacing an inline `strings.Cut` call and adding a validity check on the parsed provider. - The complexity routing plugin's `buildComplexitySessionKey` (which maintained its own SHA-256 scoping logic, virtual-key/user scope classification, and key prefix) is removed. `complexitySessionKey` now delegates directly to `bifrost.SessionStateKey` with a `"complexity"` kind, inheriting the canonical scoping and hashing from core. - The `virtualKeyID` parameter is removed from `computeComplexity` since the key is now derived entirely from the context. - Tests are updated throughout to call the exported names and to reflect the new key prefix (`session:v2:complexity:` instead of `complexity-session:v1:`). ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... ./plugins/routing/... ``` Verify that session stickiness tests pass end-to-end and that complexity routing session keys are scoped correctly per virtual key, user, and session ID. ## Breaking changes - [x] Yes - [ ] No The complexity session store key format changes from `complexity-session:v1:<hash>` to `session:v2:complexity:<hash>`. Any in-flight session state written under the old prefix will not be found after this change, causing affected sessions to start a new complexity epoch. No code migration is required; sessions will re-establish naturally on the next request. Any code outside this repository that called the previously unexported helpers by copying them will need to be updated to use the exported versions. ## Related issues ## Security considerations Session keys continue to hash all caller-supplied identifiers (virtual key ID, user ID, session ID) with SHA-256 before writing to the KV store, so no PII or caller-controlled text appears in stored keys. The complexity plugin now inherits this guarantee from core rather than maintaining a parallel implementation. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…agent, and request-options docs (#7210) ## Summary Adds a dedicated Session Affinity documentation page and cross-links it from every relevant location in the docs. Previously, session stickiness behavior was only described inline within Request Options with no standalone reference, making it hard to discover and understand how it composes with routing engines, Adaptive Load Balancing, and the circuit breaker. ## Changes - Added `docs/providers/session-affinity.mdx`: a full reference page covering the two-level affinity model (provider and key), binding lifecycle, how affinity fits into the routing order, health-aware invalidation (Enterprise), controls, observability trail entries, walk-throughs for common scenarios, and known limits. - Registered the new page in `docs.json` navigation under the Providers section. - Added `<Tip>` callouts to the Claude Code, Codex CLI, and OpenCode pages noting that each harness already sends a session header, so affinity works with no configuration, with a link to the new page. - Updated the Adaptive Load Balancing page to clarify that the system is not session-aware and that Session Affinity runs after it, linking to the new page. - Updated the Complexity Router page to note that affinity and the router are independent but use the same session identity and run alongside each other. - Updated the Provider Routing page to explain where in the routing order affinity sits and what it changes versus what routing decides. - Updated the Request Options page to point to the new guide for the full binding lifecycle and composition details. ## Type of change - [ ] Bug fix - [ ] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test Review the rendered docs to confirm: - `providers/session-affinity` resolves and appears in the sidebar under Providers. - The `<Tip>` blocks appear on the Claude Code, Codex CLI, and OpenCode pages with working links. - Cross-links from Adaptive Load Balancing, Complexity Router, Provider Routing, and Request Options all resolve to the new page. - The walk-throughs, lifecycle table, and controls table render correctly. ## Breaking changes - [x] No ## Related issues ## Security considerations None. This is documentation only. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [x] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
…nes-used, log unweighted provider exclusions, clarify docs (#7289) ## Summary Session affinity now records its decision in the routing trail and engines-used list even when the bound provider already matches what routing proposed, making it possible to distinguish "session agreed with routing" from "session was never consulted." This PR also improves the governance plugin's routing log to report the weighted candidate pool rather than the broader eligible pool, and logs providers excluded for having no weight assigned. ## Changes - `sessionAffinity.ResolveRoute` now appends `RoutingEngineSessionAffinity` to the engines-used list and writes a log entry (`Session stays on <provider> for <model>, which routing also proposed`) before the early return when the bound provider is already first in the chain, matching the behaviour of the reorder path. - The test for "bound provider already first" is updated to assert that the engine is recorded and the trail contains the agreement message, rather than asserting the opposite. - `GovernancePlugin.LoadBalanceProvider` now logs each provider excluded for having no weight assigned, for the same reason every other exclusion is named: the counts in the trail are otherwise impossible to reconcile. - The governance selection log now reports the weighted pool (`from %d weighted`) instead of the eligible pool (`from %d eligible`), because the roll ran over the weighted set and the eligible set includes candidates the selection could never have landed on. - Documentation updated to note that the UI labels the `session-affinity` engine **Session**, to add the new trail entry for agreement with routing, to clarify that the circuit-breaker and load-balancer entries are Enterprise-only, to document that governance refusals before routing leave the session binding untouched, and to correct the description of what the provider level records. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [x] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [x] Docs ## How to test ```sh go test ./core/... ./plugins/governance/... ``` Verify that a request whose session is already bound to the provider routing proposes first now shows `session-affinity` in the engines-used list and a `which routing also proposed` entry in the routing trail. Verify that a provider with no weight assigned produces an exclusion log entry in the governance trail. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
OpenAI Responses web search can return specialized API sources
({"type":"api","name":"oai-weather"}) that carry a name and no URL. The typed
source schema only modeled type and a required url, so decode dropped name and
re-encode fabricated an empty "url":""; the OpenAI request-side source
sanitization rebuilt sources without name as well. url is now omitempty and
name survives both the schema round-trip and the sanitization path.
Affected packages:
- core/schemas/responses.go - add Name to web search action sources, make url omitempty
- core/providers/openai/types.go - keep Name in the OpenAI source sanitization path
- core/schemas/responses_test.go - round-trip regression tests
- core/providers/openai/responses_test.go - sanitization regression test
- core/changelog.md - changelog entry
- tests/e2e/api/collections/provider-harness.json - replay regression case
Closes #7356
…#7360) The Anthropic Messages API accepts a message's content either as an array of content blocks or as a plain string, and clients mix the two: a rolling cache breakpoint produces block arrays for the turns carrying cache_control and bare strings for the rest. BedrockMessage.Content is typed []BedrockContentBlock, so one bare string anywhere in messages[] fails the standard unmarshal and diverts the whole request into the AI21 fallback. That branch rebuilt the messages but never called applyMessageContentCacheControl, so every cache_control in the request was dropped, not only the one on the string message. Prompt caching went off silently, with only the system cachePoint surviving on its separate path. The fallback now makes the same translation the standard path does. Fixes #7336
## Summary Extracts repeated label-value splicing logic in the telemetry plugin into a shared `spliceLabelValues` helper, and fixes an over-allocated map in the governance plugin. ## Changes - Introduced `spliceLabelValues(promLabelValues []string, nDefault int, extras ...string) []string` in `plugins/telemetry/utils.go`. This helper inserts metric-specific extra label values (e.g. `is_success`, `status_code`+`error_type`, `cache_type`) between the default labels and custom labels without mutating the input slice, replacing three identical inline `make`/`append` patterns in `PostLLMHook`. - Added `TestSpliceLabelValues` to pin the exact label-value ordering for latency, error, and cache-hit metrics. Because Prometheus metric vectors match values to label names purely by position, any reordering silently attributes values to the wrong labels; the test guards against this. - Fixed `alreadyCharged` map initialization in `reportBatchModelUsage` (governance plugin): the capacity hint `len(usage.BudgetIDs)+len(usage.RateLimitIDs)` was incorrect because `RateLimitIDs` entries are stored under a `"ratelimit:"` prefix and the map is populated in a separate loop, so the hint was misleading. The map is now initialized without a capacity hint. ## Type of change - [ ] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./plugins/telemetry/... ./plugins/governance/... ``` The new `TestSpliceLabelValues` test verifies: - Extra values are inserted at the correct position between default and custom labels. - Multiple extras preserve their relative order. - The original input slice is never mutated. - Edge cases where there are no custom labels or no extras behave correctly. ## Screenshots/Recordings N/A ## Breaking changes - [x] No ## Related issues N/A ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary A permit that grants every provider names no provider permits for those providers, so the methods that return a list of providers had nothing to enumerate and would return only the providers that happened to have explicit overrides. This made the listing and routing layers refuse providers that the request path correctly admitted. This PR completes those lists from the deployment's configured provider set when an allow-all permit is in effect. ## Changes - Added a `ProviderLister` function type and `WithConfiguredProviders` option to `Access`, so a resolver can supply the deployment's provider set without affecting resolvers that have nothing to give. - `ProvidersForModel` and `GrantedProvidersForModel` now call `providersGrantedByAllowAll`, which enumerates configured providers not already named by any provider permit and checks each through `IsModelAllowed` — the same gate the request path uses — so the two cannot drift. - A provider that a provider permit explicitly refused is tracked in a `named` set and is not reopened by the allow-all path. - An intersecting scope is not widened: the allow-all completion still passes through `IsModelAllowed`, which applies composition, so a project-scoped intersection that permits only one provider continues to permit only that one. - `GovernancePlugin.ResolveAccess` now passes `configuredProviders()` to `NewAccess`. The lister reads the in-memory store at call time rather than capturing a snapshot, so a provider added after resolution is granted by the same rule. - The `listManagementModels` handler path in the HTTP transport is covered by a new test confirming that a caller permitted every provider sees models from providers it holds no explicit permit for. ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Plugins ## How to test ```sh go test ./framework/grant/... ./transports/bifrost-http/handlers/... ./plugins/governance/... ``` The new `TestAllowAllProvidersCompletesTheProviderLists` suite in `framework/grant/access_test.go` covers the four cases: granted providers include one no permit names, candidates include that provider unweighted and unrestricted, a provider its own permit refused is not reopened, and an intersecting scope is not widened. `TestListModels_VKFilterListsProviderAllowedOnlyByAllowAll` covers the HTTP handler path end-to-end. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations The allow-all completion path gates every candidate through `IsModelAllowed`, which enforces blacklists and composition mode, so no provider or model is admitted beyond what the existing permission logic already permits. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds support for new typesafe models. We are also introducing new /v1/decisions endpoint for systemone class of models ## Changes - Adds new typesafe provider - Adds new /v1/decisions endpoint along with BifrostDecisionRequest and BifrostDecisionResponse - Adds new /typesafe integration ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [x] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [x] UI (React) - [x] Docs ## How to test Describe the steps to validate this change. Include commands and expected outcomes. ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` If adding new configs or environment variables, document them here. ## Screenshots/Recordings If UI changes, add before/after screenshots or short clips. ## Breaking changes - [ ] Yes - [x] No If yes, describe impact and migration instructions. ## Related issues NA ## Security considerations NA ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
…ers (#7361) ## Summary Providers that lack native decision support previously returned an `unsupported_operation` error. This PR introduces a transparent emulation layer that intercepts that error and re-answers the decision request through the same provider's chat model using forced tool-calling (or native structured output where available). This covers both the case where an LLM is named directly as the decision model and the case where an LLM is reached as a fallback after a native provider fails. ## Changes - When `handleProviderRequest` receives an `unsupported_operation` error from `provider.Decision`, it now calls `emulateDecisionViaChat` before propagating the error, allowing any tool-capable chat model to fulfill the request transparently. - `core/decisionemulation.go` implements the emulation: it builds a forced `emit_decision` function tool from the question set, calls `provider.ChatCompletion`, extracts the tool-call arguments (with a fallback to native structured-output content), and maps the result back to `BifrostDecisionResponse`. - `core/providers/utils/decisionemulation.go` provides the reusable schema and parsing layer: `BuildDecisionTool` encodes all question kinds (`noul`, `choice`, `score`) as a single JSON-schema function tool; `ParseDecisionAnswers` decodes and validates the tool-call arguments against each question's kind, rejecting out-of-range, wrong-typed, or missing answers. - `DecisionEmulation` test scenario and `DecisionEmulationModel` config field added to the integration test harness; `RunDecisionEmulationTest` exercises the emulation path end-to-end. - OpenAI provider test config opts into `DecisionEmulation` using `gpt-4o-mini`. - E2E Postman collection extended with three new cases: LLM as primary decision model, LLM as fallback after a failing primary, and the `/typesafe/v1/systemone` native-shape emulation path. - Python and TypeScript integration tests added under `TestDecisionLLMEmulation` / `Decision LLM emulation`, covering primary emulation, fallback, and a side-by-side comparison against the native `jev` judgment model. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh # Unit tests for the schema builder and answer parser go test ./core/providers/utils/... # Full provider integration test (requires OPENAI_API_KEY) go test ./core/providers/openai/... -run TestOpenAI/DecisionEmulation -v # Python integration tests (requires a running Bifrost instance) cd tests/integrations/python pytest tests/test_typesafe.py::TestDecisionLLMEmulation -v # TypeScript integration tests cd tests/integrations/typescript npx jest tests/test-typesafe.test.ts --testNamePattern="Decision LLM emulation" # E2E Postman collection # Import tests/e2e/api/collections/provider-harness.json and run folder # "106. Decision LLM emulation and fallback" ``` Set `DecisionEmulationModel` in the provider test config to a tool-capable chat model (e.g. `gpt-4o-mini`) and enable `Scenarios.DecisionEmulation: true`. ## Screenshots/Recordings N/A ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations The emulation path forwards the decision `state` payload as a chat message to the configured LLM provider. No new secrets or credentials are introduced; the same provider key used for chat completions is reused. Callers should be aware that state content is sent to the chat endpoint rather than a dedicated judgment endpoint. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Tightens the decision emulation pipeline so that probability distributions are always exactly normalized before being returned to callers, enforces that a chosen option must have the highest probability in its distribution, and fixes a double-gutter rendering bug in the Monaco code editor. ## Changes - Renamed `validateProbabilities` to `normalizeProbabilities`, which now returns a rescaled copy of the distribution whenever the raw sum is within the 5% tolerance but not already exact. This ensures callers always receive probabilities that sum to exactly 1, and that score values derived from the weighted sum are consistent with the normalized distribution. - Added a post-normalization check in `ParseDecisionAnswers` that rejects any choice answer whose selected option does not have the highest (or tied-highest) probability in the distribution. - Updated the decision system prompt and JSON schema descriptions to instruct the model that probabilities must sum to 1 (not "about 1") and that the choice field must select the option with the highest probability. - Disabled `glyphMargin` unconditionally in the Monaco editor and reduced `lineDecorationsWidth` from 18 to 2 when folding is enabled. The glyph margin is a separate breakpoint column that was never used; enabling it alongside folding was doubling the left gutter width. Folding arrows render in the `lineDecorations` column, which Monaco already widens automatically. ## Type of change - [x] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./... # UI cd ui pnpm i || npm i pnpm test || npm test pnpm build || npm run build ``` Verify that: - A choice answer whose selected option does not have the highest probability is rejected with an error containing "highest probability". - A near-one distribution (e.g. summing to 0.99 or 1.01) is accepted and the returned probabilities sum to exactly 1. - A distribution whose sum deviates by more than 5% is still rejected. - The Monaco editor no longer shows a doubled left gutter when folding is enabled. ## Screenshots/Recordings Before: Monaco editor with folding enabled rendered a wide double gutter on the left side. After: Gutter is compact; folding arrows appear in the line decorations column as expected. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations None. No auth, secrets, PII, or sandboxing changes. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
### TL;DR Simplified the `cloneBifrostReq` logic by replacing deep, field-by-field cloning with shallow struct copies, and added tests to verify the original request is not mutated across primary and fallback attempts. ### What changed? The overly granular cloning functions (`cloneTextCompletionParameters`, `cloneChatToolChoice`, `cloneChatWebSearchOptions`, `cloneResponsesToolChoice`, `cloneResponsesTool`, `cloneOrderedMap`, `cloneAnyValue`, etc.) have been removed. In their place, the code now performs a shallow struct copy (`cloned := *params`) and only deep-copies the specific fields that are actually written to during the plugin's `PreLLMHook` — namely `Reasoning`, `ToolChoice`, and `Tools`. This means fields the plugin never mutates are shared between the original and the clone, which is safe and avoids unnecessary allocations. Two new tests were added to `requestcopy_test.go`: - `TestCloneBifrostReq_OriginalUntouched` — asserts that `PreLLMHook` does not mutate the caller's original request. - `TestCloneBifrostReq_FallbackSeesOriginalTools` — simulates core's fallback path to confirm that tool rewrites applied for a primary provider do not bleed into a subsequent fallback attempt. ### How to test? Run the new tests directly: ```bash go test ./plugins/compat/... -run TestCloneBifrostReq ``` ### Why make this change? The previous implementation cloned every nested field defensively, including many that the plugin never modifies. This added significant complexity and maintenance burden. Since the plugin only writes to `Reasoning`, `ToolChoice`, and `Tools`, only those fields need to be isolated in the clone. The new tests explicitly guard against the mutation scenarios that motivated the original deep-copy approach, ensuring correctness without the overhead.
… preserving them for fallbacks (#7182) ## Summary Bedrock `cachePoint` markers embedded in a caller's request were leaking into non-Bedrock provider wire bodies (e.g. OpenAI primary attempts) and being silently dropped by the compat plugin, which also mutated the shared request object and broke Bedrock fallbacks that needed those markers. This PR moves the stripping responsibility into the dispatch layer using a copy-on-write approach, ensuring the original request is never mutated and each provider sees exactly what it should. ## Changes - **`promptCacheChatRequest`** now strips Bedrock `cachePoint` markers before forwarding to any non-Bedrock provider, using copy-on-write so the caller's original request retains its markers for subsequent fallbacks. - **`StripChatCachePoints`** utility added in `providers/utils/promptcache.go` to perform the copy-on-write strip, cloning only the messages that carry a marker and dropping standalone marker-only blocks entirely. - **`ToResponsesMessages`** in `schemas/mux.go` now skips standalone `cachePoint`-only blocks (type `""`) when converting chat messages to Responses format, preventing empty-type blocks from being serialized. - **`convertBifrostReasoningToBedrockReasoning`** gains a `stripBlockSignatures` parameter. For models that do not verify signatures (e.g. Nova), signatures are stripped from the outgoing wire body while the caller's in-memory block retains its signature for potential fallbacks. - The `dropCachePoint`, `dropCacheControlFromResponsesMessages`, and `applyBedrockResponsesCompatibility` functions have been removed from the compat plugin (`plugins/compat/dropparams.go`). Their responsibilities are now handled correctly and non-destructively at the dispatch layer. - Tests added for the end-to-end OpenAI → Bedrock fallback scenario, the `StripChatCachePoints` utility, the `promptCacheChatRequest` copy-on-write guarantee, and Nova's signature-stripping behavior. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [x] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... ./core/providers/bedrock/... ./core/providers/utils/... ./plugins/compat/... ``` The new `TestChatCachePoint_StrippedForPrimaryKeptForBedrockFallback` end-to-end test spins up a fake OpenAI server (returns 500) and a fake Bedrock server (returns 200), sends a request with a `cachePoint` block, and asserts: - The OpenAI wire body does **not** contain `cachePoint`. - The Bedrock fallback wire body **does** contain `cachePoint`. - The caller's original request is not mutated. ## Breaking changes - [x] No The compat plugin no longer drops `cachePoint` blocks or `cache_control` fields from Bedrock Responses requests via `dropparams`. Any caller relying on that side-effect to sanitize requests should now rely on the dispatch-layer stripping, which is transparent and non-destructive. ## Related issues Relates to #6624 (unsigned reasoning replay). ## Security considerations No auth, secrets, or PII implications. The change ensures provider-specific markers are not forwarded to unintended providers, which reduces the risk of unexpected API errors or information leakage in multi-provider fallback chains. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
Bedrock's Converse API rejects JSON tool results that contain an empty-string object key (e.g. `{"": "value"}`), returning `"The format of the value at ...toolResult.content.N.json is invalid"`. This surfaces in practice with tools like Cursor's `list_directory`, which produces extensionless filenames as empty-string keys. This PR detects that condition and falls back to sending the content as a text block instead, which the model reads identically without triggering the API rejection.
## Changes
- Added `jsonHasEmptyObjectKey` — a token-stream scanner that walks JSON without building a full value tree, reporting whether any object key at any nesting depth is an empty string.
- Added a guard in `tryParseJSONIntoContentBlock` that invokes `jsonHasEmptyObjectKey` on compacted JSON objects and arrays, falling back to a `BedrockContentBlock{Text: ...}` when an empty key is found rather than attempting to send a `json` block that Converse will reject.
- Refactored `convertToolMessages` to delegate to the existing `tryParseJSONIntoContentBlock` helper instead of duplicating the JSON-handling logic inline, removing ~30 lines of redundant code.
## Type of change
- [x] Bug fix
- [ ] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [ ] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [ ] UI (React)
- [ ] Docs
## How to test
```sh
go test ./core/providers/bedrock/...
```
To validate end-to-end, send a tool result containing an empty-string object key (e.g. `{"": "extensionless-file"}`) through a Bedrock-backed model (e.g. `us.anthropic.claude-haiku-4-5`). Prior to this fix the request would fail with an invalid format error; after this fix it succeeds and the model receives the content as a text block.
## Breaking changes
- [x] No
## Security considerations
No auth, secrets, or PII implications. The change only affects how tool result content blocks are serialized before being sent to the Bedrock Converse API.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [ ] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [ ] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
* [fix]: core - preserve Grok 4.7 xhigh reasoning effort Affected packages: - core/schemas - core/providers/openai - core * [test]: core - consolidate reasoning effort E2E coverage Affected packages: - core
## Summary Adds end-to-end passthrough support for Claude Code's auto-mode server-side classifier (`safeguards` / `safeguard_results`). When Bifrost proxies Claude Code traffic, the upstream Anthropic API sends opaque `safeguards` fields on requests and `safeguard_results` fields on responses and stream events. Previously these were silently dropped, breaking Claude Code's auto-mode behavior. This change captures, carries, and restores them byte-for-byte through the Bifrost pipeline while stripping them fail-closed on every non-Anthropic provider. ## Changes - Added `Safeguards json.RawMessage` to `AnthropicMessageRequest` and `SafeguardResults json.RawMessage` to `AnthropicMessageResponse` and `AnthropicStreamEvent`, with compact-JSON normalization on unmarshal. - Added `SafeguardResults json.RawMessage` to `BifrostResponsesResponse` and `BifrostResponsesStreamResponse` so the fields survive the Bifrost internal representation without being copied by `WithDefaults` (OpenAI-shaped surfaces never see them). - Introduced `ResponsesStreamResponseTypeProviderRawEvent`, a Bifrost-synthesized stream chunk type that acts as a raw-only carrier when an upstream SSE event cannot be modeled (e.g. a future or unknown Anthropic event type). The Anthropic passthrough egress forwards the raw frame verbatim; all other surfaces drop it, as does `WithDefaults`. - When `message_start` carries `safeguard_results` nested inside `message.safeguard_results`, the field is forwarded into the Bifrost stream chunk and restored onto the re-rendered `message_start` event on egress. - `safeguard_results` appearing top-level on any stream event is attached to the corresponding `BifrostResponsesStreamResponse` chunk and restored by `ToAnthropicResponsesStreamResponse` onto the first re-rendered frame. - Added `Safeguards bool` to `ProviderFeatureSupport`, enabled only for the direct Anthropic provider. `stripUnsupportedAnthropicFields` and `StripUnsupportedFieldsFromRawBody` both strip `safeguards` on every other provider. - `ToBifrostResponsesRequest` round-trips `safeguards` through `ExtraParams`; `ToAnthropicResponsesRequest` restores it from `ExtraParams` back onto the typed field with type-switch handling for `json.RawMessage`, `[]byte`, and arbitrary values. - Gemini's `ToGeminiResponsesStreamResponse` now skips `ResponsesStreamResponseTypeProviderRawEvent` alongside the existing lifecycle no-ops. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./... ``` To validate end-to-end, proxy a Claude Code auto-mode request through Bifrost targeting the direct Anthropic provider and confirm: 1. The outbound request to Anthropic contains the `safeguards` field unchanged. 2. Non-streaming responses include `safeguard_results` in the Anthropic-shaped reply. 3. Streaming responses include `safeguard_results` on the appropriate frame (e.g. `message_start` or a top-level event). 4. Routing the same request through a non-Anthropic provider (e.g. Vertex, Bedrock) strips `safeguards` from the outbound request without error. 5. OpenAI-shaped egress surfaces never emit `safeguard_results`. ## Breaking changes - [ ] Yes - [x] No ## Security considerations `safeguards` / `safeguard_results` are forwarded opaquely as `json.RawMessage` and never logged or inspected beyond JSON compaction. They are stripped fail-closed on all non-Anthropic providers, so no undocumented fields leak to third-party backends. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Two production heap leaks are fixed in this PR. First, `StripEmptyThinkingBlocks` was rewriting the entire request body once per deleted block (O(blocks × body)), accounting for 23.7% of all bytes ever allocated in production. Second, `handleStreaming` only cancelled the request context on write errors, leaving a `startClientDisconnectWatcher` goroutine and the entire request-scoped `BifrostContext` alive until the client closed its connection — two production pods showed ~15 leaked contexts per open connection, holding ~2 GB of a 2.66 GB heap that GC could not reclaim. A third smaller fix eliminates a full decode of the Anthropic request body (messages array and all) that was done purely to decide which beta headers to inject, then immediately discarded, holding ~526 MB live. ## Changes - **`StripEmptyThinkingBlocks` rewrite (O(n) → O(1) in block count):** The function now does a single scan pass to identify which messages need rewriting and which blocks to keep, then rebuilds the messages array once using `bytes.Buffer` and raw gjson results. Previously it called `DeleteJSONField` (which calls sjson, which reserialises the whole body) once per block. A new allocation-bounding test (`TestStripEmptyThinkingBlocks_AllocationIsLinear`) pins this: it fails loudly if the implementation regresses to quadratic allocation. - **`handleStreaming` cancels on every exit path:** A `defer cancel()` is registered inside the producer goroutine so the request context is cancelled on normal stream completion, not only on write errors. This ends the `startClientDisconnectWatcher` goroutine that `ConvertToBifrostContext` starts per request. A new regression test (`Test_handleStreamingCancelsContextOnNormalCompletion`) verifies cancel is called even when no write ever fails. Existing tests are updated to use a `cancelRecorder` (channel-based, race-safe) instead of a plain bool, and their assertions are updated to expect cancel on all exit paths. - **Beta header injection without full decode:** `AddMissingBetaHeadersToContextFromRawBody` replaces the probe-unmarshal in `BuildAnthropicResponsesRequestBody`. It strips the `messages` array before decoding the top-level fields, and recovers the two signals the gating reads from messages (`scopedCacheControl`, `fileSource`) via a gjson scan (`scanRawMessagesForBetaSignals`). The internal `addMissingBetaHeadersToContext` is refactored to accept an `anthropicMessageBetaSignals` struct so both the typed and raw-body entry points share identical gating logic. `TestAddMissingBetaHeadersFromRawBody_MatchesTypedPath` asserts the two paths produce identical headers across every signal combination and all three providers. - **UI:** The log details sheet breakpoint for the 40% max-width is changed from `xl` to `2xl`. ## Type of change - [x] Bug fix - [ ] Feature - [x] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go test ./core/providers/anthropic/... -run TestStripEmptyThinkingBlocks go test ./core/providers/anthropic/... -run TestStripEmptyThinkingBlocks_AllocationIsLinear go test ./core/providers/anthropic/... -run TestAddMissingBetaHeadersFromRawBody_MatchesTypedPath go test ./transports/bifrost-http/integrations/... -run Test_handleStreamingCancelsContextOnNormalCompletion go test -race ./transports/bifrost-http/integrations/... go test ./... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` The allocation test will fail with a message describing the O(blocks × body) regression if `StripEmptyThinkingBlocks` is reverted to per-block deletes. The cancel test will fail with a 5-second timeout if `handleStreaming` returns without calling cancel on normal completion. ## Breaking changes - [ ] Yes - [x] No ## Related issues Production heap profiles showing 23.7% allocation in `StripEmptyThinkingBlocks`, ~526 MB live from the beta-header probe decode, and ~2 GB of unreclaimable heap from leaked watcher goroutines. ## Security considerations No auth, secrets, PII, or sandboxing changes. The beta header logic is refactored but functionally identical, verified by the parity test across all providers and signal combinations. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
…e_preview` (#7426) OpenAI's computer-use tool for GPT-6 Astra and GPT-5.6 is `{"type":"computer"}` with no display/environment fields. normalizeResponsesToolType folded every `computer*` type into `computer_use_preview`, so OpenAI rejected the request: "Tool 'computer_use_preview' is not supported with gpt-6-astra". Leave the bare `computer` type untouched and round-trip it without extra fields. Dated Anthropic variants (`computer_20250124`) still normalize to `computer_use_preview` as before. Fixes #7425
Use ordinary struct decoding for ResponsesRequest and select the string or array decoding path before parsing input and content. Embed ResponsesParameters by value to preserve null handling. Add decoding boundary tests, request preparation benchmarks, and a core changelog entry.
…sMessage (#7422) ## Summary `CacheControl`, `ProviderNativeParts`, and several `ResponsesToolMessage` sub-fields were not being deep copied in `DeepCopyResponsesMessage` and `deepCopyResponsesMessageContentBlock`, meaning callers sharing these structs could inadvertently mutate each other's pointer values and slice backing arrays. ## Changes - Added deep copy logic for `CacheControl` (including `TTL` and `Scope` pointer fields) on both `ResponsesMessage` and `ResponsesMessageContentBlock` - Added deep copy for `ProviderNativeParts` (`json.RawMessage`) on `ResponsesMessage` - Added deep copy for the bare-string `ResponsesToolCallActionStr` on `ResponsesToolMessageActionStruct` - Added deep copy for `ResponsesComputerToolCall` (including `PendingSafetyChecks` slice) and `ResponsesComputerToolCallOutput` (including `AcknowledgedSafetyChecks` with pointer fields) - Added deep copy for `ResponsesCodeInterpreterToolCall` (including `Code`, `Outputs`, and nested log/image variants) - Added deep copy for `ResponsesMCPToolCall`, `ResponsesImageGenerationCall` (all optional string fields), `ResponsesMCPListTools` (including `InputSchema` and `Annotations` maps and `Description`), `ResponsesMCPApprovalResponse`, `ResponsesAdvisorCall`, `ResponsesToolSearchCall`, and `ResponsesCodeExecutionCall` (including `Lines`, `Files`, and `Caller`) - Added deep copy for `Citations`, `PromptCacheBreakpoint`, and `FileType` in `deepCopyResponsesMessageContentBlock` - Added deep copy for Anthropic-specific annotation fields (`StartCharIndex`, `EndCharIndex`, `StartPageNumber`, `EndPageNumber`, `StartBlockIndex`, `EndBlockIndex`, `Source`, `EncryptedIndex`) - Added deep copy for `ResponsesOutputMessageContentRenderedContent`, `ResponsesOutputMessageContentCompaction`, and `ResponsesOutputMessageContentFallback` ## Type of change - [x] Bug fix ## Affected areas - [x] Core (Go) ## How to test ```sh go test ./core/schemas/... ``` Verify that mutating any copied pointer field or slice on a `ResponsesMessage` copy does not affect the original struct, as exercised by `TestDeepCopyResponsesMessagePreservesCacheControls` and `TestDeepCopyResponsesMessagePreservesExtendedFields`. ## Breaking changes - [x] No ## Security considerations None. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
Adds continuous performance measurement with CodSpeed over the paths every request pays for: - core/schemas: chat response/chunk and tool-schema JSON round-trips - core/providers/openai: inbound parse, neutral lowering, wire raising, marshal - core/providers/anthropic: Messages request/response and stream-event conversion - core/providers/utils: SSE data and event framing readers - core/keyselectors: weighted key selection at 3, 25 and 200 keys The workflow runs these plus the benchmarks already in framework/ and plugins/ on CodSpeed macro runners with the walltime instrument, the only instrument the Go integration supports. Co-authored-by: CodSpeed Bot <no-reply@codspeed.io>
## Summary Claude Code injects an `x-anthropic-billing-header` metadata block into the system prompt before sending requests. When Bifrost routes those requests to non-Anthropic model families (e.g. GPT, Gemini), the billing block leaks into the outbound prompt, polluting the context sent to those providers. This PR strips the billing block at Messages ingress, before the input is shared across attempts, and restores it only for Anthropic-family dispatch — including cross-family fallbacks and alias-resolved targets. ## Changes - **`core/schemas/billingheader.go`** — Introduces `ExtractAnthropicBillingHeader` and `WithAnthropicBillingHeader` on `BifrostResponsesRequest`. Extraction compacts the owned system block slice once at ingress (O(H) space, no prompt string copies). Restoration produces a shallow, attempt-local copy that re-inserts the billing blocks at their original positions, preserving cache markers and block ordering. - **`core/schemas/responses.go`** — Adds a private `anthropicBillingHeader` field to `BifrostResponsesRequest` to carry extracted metadata across retries and fallbacks without serialization. - **`core/billingheader.go`** — Adds `restoreResponsesBillingHeader`, which resolves the model family (via alias, model name, or base provider) and calls `WithAnthropicBillingHeader` only for Anthropic-family attempts. Raw passthrough requests are left untouched since they already carry the original body. - **`core/bifrost.go`** — Calls `restoreResponsesBillingHeader` as the first step in `prepareResponsesRequest`, before prompt-cache injection and tool flattening. - **`transports/bifrost-http/integrations/anthropic.go`** — Calls `ExtractAnthropicBillingHeader` immediately after conversion and content-block normalization, while the input slice is still exclusively owned, before hooks or fallback attempts can share it. - **Tests** — Unit tests cover extraction idempotency, block position and cache-marker restoration, non-Anthropic zero-copy fast path, raw passthrough, all four dispatch paths (responses/chat × sync/stream), and a broad matrix of provider/model/alias combinations. A benchmark confirms non-Anthropic attempts do not allocate or inspect message payloads. E2E Postman collection adds three cases asserting the outbound wire for GPT strips the header while Anthropic preserves it. **Design decisions:** - Extraction happens once at ingress on the owned slice to avoid copying the full message list on every non-Anthropic attempt. - The private field survives shallow struct copies (used by core fallback paths) so metadata is never lost across retries. - `isStandaloneBillingHeader` matches only lines that are exclusively a billing header (no mixed instruction text), leaving real system content intact. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/... ./transports/... ``` The E2E Postman collection entry `109. Claude Code billing header filtering (billing-header)` covers three live cases: - GPT-4o-mini with a versioned billing header → outbound wire must not contain `x-anthropic-billing-header:` - GPT-4o-mini with a nonce billing header → same assertion - Anthropic Claude Haiku → outbound wire must preserve both billing and instruction blocks in order ## Breaking changes - [ ] Yes - [x] No ## Security considerations The billing header contains client attribution metadata (`cc_version`, `cc_entrypoint`, `cch` nonce). Stripping it before forwarding to third-party providers prevents unintended disclosure of Claude Code session metadata to non-Anthropic endpoints. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary
DeepSeek rejects tool schemas containing `\0` with a 400 error ("is not a regex"), and `moonshotai.kimi-k3` on Bedrock rejects both `\0` and lookaround assertions (`(?!`, `(?=`, `(?<=`, `(?<!`) by returning HTTP 200 with an empty event stream. This caused every Claude Code request routed to kimi-k3 to silently fail, since Claude Code's `ArtifactData` tool ships four lookaround patterns. This PR adds a targeted, copy-on-write tool-schema regex normalization pass on the shared Responses dispatch path that rewrites `\0` → `\x00` and strips lookaround assertions, but only for Moonshot and DeepSeek models. All other models receive their schemas byte-identical, preserving prompt-cache keys.
## Changes
- **`core/bifrost.go`**: Calls `normalizeResponsesToolSchemas` in `prepareResponsesRequest` before any provider-specific handling. Adds `normalizeResponsesToolSchemas`, `toolSchemaPatternRewriter`, and the pre-built `relaxedPatternRewriter` (composed of `NormalizeRegexNULEscape` + `StripRegexLookaround`). The rewrite is gated on `IsMoonshotModel` or `IsDeepSeekModel` on the canonical model name, so kimi behind an OpenAI-compatible gateway is still treated as kimi, and Claude on Bedrock is still left untouched.
- **`core/providers/utils/utils.go`**: Adds `NormalizeRegexNULEscape` (rewrites `\0` → `\x00`, skipping legacy octal escapes like `\012` and escaped backslashes like `\\0`), `StripRegexLookaround` (removes `(?=...)`, `(?!...)`, `(?<=...)`, `(?<!...)` groups, leaving non-capturing `(?:...)` and named groups alone, and returning the pattern unchanged if unbalanced), `ComposePatternRewriters`, `RewriteToolSchemaPatterns`, and `RewriteResponsesToolSchemas`. All walkers use copy-on-write: schemas with nothing to rewrite are returned as the original pointer with no allocation.
- **`core/schemas/utils.go`**: Adds `IsMoonshotModel`, which matches `moonshotai` and `kimi` substrings to cover Bedrock inference-profile prefixes, Groq-style `moonshotai/kimi-k2-instruct`, and bare `kimi-k3` behind custom gateways.
- **E2E harness**: Adds collection group 109 with five cases asserting via `x-bf-send-back-raw-request` that DeepSeek and kimi-k3 receive rewritten patterns, while OpenAI, Anthropic, and Gemini receive the original bytes unchanged.
## Type of change
- [x] Bug fix
## Affected areas
- [x] Core (Go)
- [x] Providers/Integrations
## How to test
```sh
go test ./core/... ./core/providers/utils/...
```
The unit tests cover:
- `NormalizeRegexNULEscape`: NUL rewrite, octal exclusion, escaped-backslash exclusion, idempotency.
- `StripRegexLookaround`: all four lookaround forms, nested groups, escaped parens, character classes containing `(`, unbalanced patterns returned unchanged, non-capturing and named groups left alone.
- `RewriteToolSchemaPatterns` and `RewriteResponsesToolSchemas`: copy-on-write correctness (unchanged schemas return the same pointer, changed schemas never mutate the caller's data, key order is preserved).
- `normalizeResponsesToolSchemas`: only Moonshot and DeepSeek models get a rewritten copy; every other model gets its own request pointer back.
E2E collection group 109 validates live wire behavior against DeepSeek, kimi-k3 on Bedrock, OpenAI, Anthropic, and Gemini.
## Breaking changes
- [x] No
## Security considerations
The rewrite is purely structural (regex syntax normalization) and operates only on tool schema `pattern` strings. No secrets, auth tokens, or user content are touched. The positive-list approach means an unknown backend that rejects a pattern fails loudly rather than being silently rewritten.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
## Summary
Adds Claude Opus 5.5 to the set of models that reject `thinking:{"type":"disabled"}` and forced tool choice (`tool_choice: "any"/"tool"`), and refactors `RejectsDisabledThinking` to accept `ModelCaps` instead of a raw model string so the datasheet's `supports_reasoning_disable` field can override the name-based fallback in both directions without a code release.
## Changes
- `IsOpus55Plus` added to `core/schemas/utils.go` to identify Claude Opus 5.5+ by name, matching Bedrock/Vertex/date-suffixed variants.
- `DefaultCanDisableReasoning` updated to return `false` for Opus 5.5+ in addition to the Fable/Mythos family.
- `DefaultSupportsForcedToolChoice` updated to return `false` for Opus 5.5+ in addition to Fable 5.1+.
- `RejectsDisabledThinking` signature changed from `(model string, effort *string)` to `(caps ModelCaps, effort *string)`. It now reads `caps.CanDisableReasoning(DefaultCanDisableReasoning(model))` for the unconditional gate, so a datasheet row can flip the passthrough in either direction without touching code. The per-request effort carve-out for Opus 5 is unchanged.
- Both call sites in `utils.go` (`stripUnsupportedAnthropicFields` and `StripUnsupportedFieldsFromRawBody`) updated to pass `caps` instead of `model`.
- Tests extended with `claude-opus-5-5` in the `alwaysOn` and forced-tool-choice model lists, plus two new sub-tests verifying that a datasheet row overrides the name-based fallback in both directions (`datasheet_outranks_the_name_fallback` and `datasheet_disables_on_a_model_the_fallback_allows`).
## Type of change
- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI
## Affected areas
- [x] Core (Go)
- [ ] Transports (HTTP)
- [x] Providers/Integrations
- [ ] Plugins
- [ ] UI (React)
- [ ] Docs
## How to test
```sh
go test ./core/providers/anthropic/... ./core/schemas/...
```
Expected: all existing tests pass, and the two new sub-tests under `TestDisabledThinkingStrip` (`datasheet_outranks_the_name_fallback`, `datasheet_disables_on_a_model_the_fallback_allows`) and the extended `TestForcedToolChoice_DroppedWhenUnsupported` pass for `claude-opus-5-5`.
## Breaking changes
- [x] Yes
- [ ] No
`RejectsDisabledThinking` now takes `schemas.ModelCaps` instead of a `string`. Any caller outside this repo that references this function directly will need to update its call site to pass a `ModelCaps` value.
## Related issues
## Security considerations
None.
## Checklist
- [ ] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
…hile blocking link-local targets via pre-proxy destination guard (#7437) ## Summary MCP HTTP/SSE connections were not honoring `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` from the process environment. This was introduced in core v1.8.5 when the proxy selector was nulled out to prevent SSRF bypass via proxy. Deployments with no direct egress (locked-down VPCs, pods behind a corporate proxy) could not reach public MCP servers, causing clients to remain `Unstable` with `TLS handshake timeout` or `context deadline exceeded` errors. ## Changes - Restores `http.ProxyFromEnvironment` on the MCP HTTP transport by keeping the cloned `DefaultTransport` proxy selector intact instead of setting it to `nil`. - Introduces `mcpProxySelector`, a wrapper around the transport's proxy selector that validates the request destination against the private network policy (`ResolvePrivateNetworkTarget`) before allowing the proxy to be contacted. This closes the SSRF gap that originally motivated nulling the selector: a link-local or unspecified destination is refused before the proxy is ever dialed, regardless of what the proxy would do with it. - Extracts the DNS resolution and address policy check from `PrivateNetworkDialContext` into a new exported `ResolvePrivateNetworkTarget` function in `core/network/ssrf.go`, shared by both the dial-time guard and the proxy selector. - Renames `TestBuildTLSHTTPClientNeverRoutesThroughProxy` to `TestBuildTLSHTTPClientRefusesProxyingLinkLocalTarget` to reflect its narrower scope, and adds three new tests: - `TestBuildTLSHTTPClientRoutesThroughConfiguredProxy` — proves a permitted destination is actually routed through the proxy and never dialed directly. - `TestBuildTLSHTTPClientBypassedProxyDialsDirect` — proves a `NO_PROXY`-style bypass still reaches the destination directly and the dial-time guard still blocks link-local on that path. - `TestMCPProxySelectorPassthrough` — pins the nil-passthrough and error-passthrough behaviors of `mcpProxySelector`. - Adds a `TestResolvePrivateNetworkTargetPolicy` unit test for the extracted function. - Documents the egress proxy behavior and the affected version range in `docs/mcp/connecting-to-servers.mdx`, including a note that `network_config.proxy_config` does not apply to MCP connections. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [x] Docs ## How to test ```sh go test ./core/mcp/... ./core/network/... ``` To validate proxy routing end-to-end, set `HTTP_PROXY` or `HTTPS_PROXY` to a reachable proxy in the process environment and confirm an MCP server connection that previously timed out now succeeds. To confirm the SSRF guard is intact, point a connection string at `169.254.169.254` with a proxy configured and confirm the request is refused with `blocked connection to link-local address` before the proxy is contacted. ## Breaking changes - [ ] Yes - [x] No ## Security considerations The SSRF guard for MCP connections is preserved on both the direct and proxied paths. On the proxied path, `mcpProxySelector` resolves the destination hostname and applies the link-local/unspecified block before the proxy is contacted, so a malicious connection string cannot use a configured proxy to reach a blocked address. The proxy itself is trusted operator configuration (process environment), not request input. DNS re-resolution on the proxy side is inherent to proxying and is an accepted trade-off documented in the code. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable
## Summary Anthropic's Claude Opus 5.5 rejects every dated `computer_*` tool type and instead requires the new `computer_toolset_20260801` entry. Unlike the dated tools, a toolset entry carries no `name` and no `display_*` geometry — sending either is a 400. Additionally, Anthropic requires that `toolset_name` appear on both halves of a call/result pair or neither, meaning a client speaking a dialect without the field (e.g. an OpenAI-shaped `function_call_output`) would produce a mismatched pair and a 400. This PR adds full support for the toolset generation across the typed path, the raw passthrough path, and the streaming path. ## Changes - **New `ComputerUseGenToolset20260801` generation constant** — `ComputerUseGeneration()` now returns this for Opus 5.5 on surfaces that have dropped the dated tools (Anthropic API and Vertex). AWS Bedrock and Azure still serve `computer_20251124` for the same model, so the split is scoped to the provider as well as the model via `toolsetOnlyComputerUseProvider()`. - **Toolset entry emitted bare** — `convertBifrostToolToAnthropic()` and `NormalizedToolSpec()` return a tool with only `Type` set when the generation is `20260801`. `AnthropicTool.Name` is now `omitempty` so an empty name is omitted from the wire rather than sent as `""`. - **Chat path guard fixed** — `convertServerToolToAnthropic()` previously dropped any tool it could not name. A toolset entry legitimately has no name, so an early return is added before the guard for that case. - **`toolset_name` threaded through the full message lifecycle** — The field is now carried on `AnthropicContentBlock` and `ResponsesToolMessage`. `ConvertBifrostMessagesToAnthropicMessages()` builds a `toolsetNameByToolUseID` map from assistant tool-use blocks and backfills the value onto any matching tool-result block that arrived without it. - **Streaming path updated** — `ToBifrostResponsesStream()` copies `ToolsetName` from the inbound `content_block_start` into the neutral `ResponsesToolMessage`, and `toAnthropicResponsesStreamEvents()` writes it back onto the outbound `content_block_start`. - **Raw passthrough path normalized** — `AnthropicProviderRequestDefaultsMap` now sets `RemapToolVersions: true` for the native Anthropic provider. `RemapRawToolVersionsForProvider()` strips `name`, `display_width_px`, `display_height_px`, `display_number`, and `enable_zoom` when upgrading a dated entry to the toolset form. - **`convertAnthropicToolToBifrost()` handles the toolset type** — A `computer_toolset_20260801` entry round-trips back to `ResponsesToolTypeComputerUsePreview` so the typed path can re-derive the correct generation on the way out. - **New test file `computertoolset_test.go`** — Covers: bare emission, dated-tool upgrade for Opus 5.5, Opus 5 keeping the dated tool, symmetric `toolset_name` replay, backfill onto results from clients without the field, no invention for plain tool calls, datasheet overriding the fallback, the chat-path guard fix, streaming round-trip, and both raw-body upgrade paths. ## Type of change - [x] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go test ./core/providers/anthropic/... -run TestComputerToolset go test ./core/providers/anthropic/... -run TestComputerUseGeneration go test ./core/providers/anthropic/... -run TestNormalizedToolSpec go test ./core/providers/anthropic/... -run TestRawBodyBuilderKeepsToolsAndSetsBetaHeaders go test ./core/providers/anthropic/... ``` Expected: all tests pass. The `TestComputerToolset_*` suite validates the full round-trip for the toolset generation including streaming, raw passthrough, and `toolset_name` backfill. ## Breaking changes - [x] No `AnthropicTool.Name` is now `omitempty`. Any code that previously relied on an empty `name` field being serialized as `""` on the wire will now omit it, which is the correct behaviour for toolset entries and harmless for all other tools that always set a name. ## Security considerations None. No auth, secrets, PII, or sandboxing changes. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
Co-authored-by: tejas ghatte <tejas@tejass-MacBook-Pro.local>
|
|
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Repository: maximhq/bifrost/.coderabbit.yaml Review profile: CHILL Plan: Team Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (296)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Features
/v1/decisionsendpoint and/typesafeintegration. Providers without native decision support now answer decision requests through forced tool-calling on their chat model, whether used as the primary or as a fallback. Probabilities are normalized to sum to exactly 1 and the chosen option must be the most likely one. Decision requests are priced from the datasheet and logged with their answers (typesafe models support, new decisions endpoint #7355, feat(decision): emulate decisions via LLM tool-calling for all providers #7361, benchmarking and api struct fixes #7384, test fixes #7440)x-bf-session-idor the session header that Claude Code, Codex CLI or OpenCode already send) stays on the provider and key that last served it. Affinity only reorders the chain routing built and never restores a provider routing excluded. The logs UI shows it as a routing enginecomputer_toolset_20260801on the Anthropic API and Vertex, while Bedrock and Azure keepcomputer_20251124.toolset_nameis carried on both halves of each call/result pair across typed, raw passthrough and streaming paths. Disabled thinking and forced tool choice are rejected for Opus 5.5+, and the datasheetsupports_reasoning_disablefield can override this (fix: claude opus 5-5 changes #7433, fix: computer toolset compatibility #7434, fix: computer toolset model capability #7441)safeguardsandsafeguard_resultsare forwarded byte-for-byte on requests, responses and stream events to the direct Anthropic provider and stripped on every other provider. Thedangerous-tool-useandauto-mode-classifierbetas are gated the same way. Unknown Anthropic SSE events are forwarded raw on the Anthropic passthrough (adds safeguard support for passthrough requests #7393, test fixes #7440)🐞 Fixed
\0to\x00, lookaround assertions stripped) for Moonshot and DeepSeek models only. kimi-k3 on Bedrock no longer returns an empty stream, and every other model gets byte-identical schemas (kimi and deepseek support for claude code #7430)x-anthropic-billing-headersystem block is removed at Messages ingress and restored only for Anthropic-family attempts, including fallbacks and alias targets, so it no longer pollutes GPT or Gemini prompts (readact anthropic billing headers to avoid cache spoiling #7431)HTTP_PROXY,HTTPS_PROXYandNO_PROXYagain (broken since core v1.8.5). Link-local and unspecified destinations are refused before the proxy is dialed (fix: restoreHTTP_PROXY/HTTPS_PROXYsupport for MCP connections while blocking link-local targets via pre-proxy destination guard #7437)computerTool - The bare{"type":"computer"}tool is no longer rewritten tocomputer_use_preview, which fixes computer use on GPT-6 Astra and GPT-5.6 (fix(responses): stop rewriting OpenAI computer tool to computer_use_preview #7426) (thanks @abhishekgahlot2!)StripEmptyThinkingBlocksrewrites the body once instead of once per block, and Anthropic beta-header gating no longer decodes the full request body (memory fixes for streaming cancellation #7406)cachePointmarkers are stripped copy-on-write for non-Bedrock providers and kept for Bedrock fallbacks. The compat plugin no longer mutates the shared request. Nova reasoning signatures are stripped on the wire only (fix: strip bedrock cachepoint markers for non-bedrock providers while preserving them for fallbacks #7182)list_directory) are sent as text, so Converse no longer rejects them (adds bedrock unknown json document handling #7396)cache_controlwhen any message's content is a plain string (fix(bedrock): keep cache_control when a message's content is a string #7360) (thanks @basil-k-aji-dev!)nameand no longer emit an emptyurl([fix]: preserve API web search source names #7358) (thanks @g-yixuan!)xhighreasoning effort is no longer downgraded tohigh([fix]: preserve Grok 4.7 xhigh reasoning effort #7403) (thanks @nettee!)data:prefix (test fixes #7440)DeepCopyResponsesMessagenow deep copies cache controls, provider-native parts, computer/MCP/code-interpreter tool fields and annotations, so copies no longer share pointers with the original (fix(schemas): deep copy extended Responses fields in DeepCopyResponsesMessage #7422)🗄️ Database Migrations
🐙 Closed GitHub Issues
{"type":"computer"}is rewritten tocomputer_use_preview, breaking GPT-6 Astra / GPT-5.6 computer use