Skip to content

providers/openrouter: pin the serving backend, record which fleet served - #107

Open
lec77 wants to merge 1 commit into
SJTU-IPADS:mainfrom
lec77:providers-pin-openrouter-backend
Open

providers/openrouter: pin the serving backend, record which fleet served#107
lec77 wants to merge 1 commit into
SJTU-IPADS:mainfrom
lec77:providers-pin-openrouter-backend

Conversation

@lec77

@lec77 lec77 commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Problem

OpenRouter routes a model id to whichever upstream fleet it likes, and it re-routes between sessions. Two runs of the same skill against the same model id can therefore execute on different backends, with different quantization, different context handling, and different prompt-cache behaviour.

That is not a small effect. In a measured session pair on one model id, the prompt-cache hit rate went 0% → 75%, mean call latency 1.8s → 3.1s, and per-task scores moved by up to 58 points on byte-identical inputs (permutation test p < 0.0005). A benchmark repeated a few weeks later can be comparing serving paths rather than skills, and nothing in the results says which.

Change

  • SKVM_OPENROUTER_PROVIDER (comma-separated, e.g. deepinfra,nebius) sets provider.order with allow_fallbacks: false on both request builders. Unset keeps today's behaviour exactly — OpenRouter's own routing.
  • Every response records the fleet that actually served it as LLMResponse.providerName, from OpenRouter's provider field. A run's serving path becomes diagnosable after the fact instead of being inferred from cache-hit rates.

Opt-in, so nothing changes for existing users until they set the variable.

Verification

bunx tsc --noEmit clean; bun test 1297 pass / 0 fail. New cases in test/providers/openrouter.test.ts cover the pinned request body, the unset case (no provider key at all), and providerName parsing.

Note: touches the same parseResponse return and LLMResponse shape as #106, so whichever lands second needs a one-line rebase.

🤖 Generated with Claude Code

OpenRouter routes a model id to whichever upstream fleet it likes, and it
re-routes between sessions — so two runs of the same skill on the same model id
can land on different backends. We measured a session pair where the prompt-cache
hit rate went 0% -> 75% and mean call latency 1.8s -> 3.1s, with per-task scores
moving by up to 58 points on byte-identical inputs (permutation test p < 0.0005).
A benchmark repeated a few weeks later is then comparing serving paths, not
skills, with nothing in the results to say so.

SKVM_OPENROUTER_PROVIDER (comma-separated, e.g. "deepinfra,nebius") now sets
provider.order with allow_fallbacks:false on both request builders; unset keeps
OpenRouter's default routing. Every response also records the fleet that served
it as LLMResponse.providerName, so a run's serving path is diagnosable after the
fact rather than inferred from cache-hit rates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant