providers/openrouter: pin the serving backend, record which fleet served - #107
Open
lec77 wants to merge 1 commit into
Open
providers/openrouter: pin the serving backend, record which fleet served#107lec77 wants to merge 1 commit into
lec77 wants to merge 1 commit into
Conversation
OpenRouter routes a model id to whichever upstream fleet it likes, and it re-routes between sessions — so two runs of the same skill on the same model id can land on different backends. We measured a session pair where the prompt-cache hit rate went 0% -> 75% and mean call latency 1.8s -> 3.1s, with per-task scores moving by up to 58 points on byte-identical inputs (permutation test p < 0.0005). A benchmark repeated a few weeks later is then comparing serving paths, not skills, with nothing in the results to say so. SKVM_OPENROUTER_PROVIDER (comma-separated, e.g. "deepinfra,nebius") now sets provider.order with allow_fallbacks:false on both request builders; unset keeps OpenRouter's default routing. Every response also records the fleet that served it as LLMResponse.providerName, so a run's serving path is diagnosable after the fact rather than inferred from cache-hit rates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
OpenRouter routes a model id to whichever upstream fleet it likes, and it re-routes between sessions. Two runs of the same skill against the same model id can therefore execute on different backends, with different quantization, different context handling, and different prompt-cache behaviour.
That is not a small effect. In a measured session pair on one model id, the prompt-cache hit rate went 0% → 75%, mean call latency 1.8s → 3.1s, and per-task scores moved by up to 58 points on byte-identical inputs (permutation test p < 0.0005). A benchmark repeated a few weeks later can be comparing serving paths rather than skills, and nothing in the results says which.
Change
SKVM_OPENROUTER_PROVIDER(comma-separated, e.g.deepinfra,nebius) setsprovider.orderwithallow_fallbacks: falseon both request builders. Unset keeps today's behaviour exactly — OpenRouter's own routing.LLMResponse.providerName, from OpenRouter'sproviderfield. A run's serving path becomes diagnosable after the fact instead of being inferred from cache-hit rates.Opt-in, so nothing changes for existing users until they set the variable.
Verification
bunx tsc --noEmitclean;bun test1297 pass / 0 fail. New cases intest/providers/openrouter.test.tscover the pinned request body, the unset case (noproviderkey at all), andproviderNameparsing.Note: touches the same
parseResponsereturn andLLMResponseshape as #106, so whichever lands second needs a one-line rebase.🤖 Generated with Claude Code