Skip to content

fix(openrouter-models): stop treating output limit as throughput - #178

Open
yazcaleb wants to merge 2 commits into
OpenRouterTeam:mainfrom
yazcaleb:fix/openrouter-models-throughput-13
Open

fix(openrouter-models): stop treating output limit as throughput#178
yazcaleb wants to merge 2 commits into
OpenRouterTeam:mainfrom
yazcaleb:fix/openrouter-models-throughput-13

Conversation

@yazcaleb

@yazcaleb yazcaleb commented Aug 27, 2026

Copy link
Copy Markdown

Problem

list-models.ts and compare-models.ts accepted --sort speed and --sort throughput, but ranked models by top_provider.max_completion_tokens. That field is an output-capacity limit, not generation speed, so the commands silently returned the wrong ordering.

Closes #13

Root cause

The model list response does not contain tokens-per-second performance data. Actual throughput is live and provider-specific under GET /models/{id}/endpoints.

Changes

  • Remove model-level speed and throughput sorting from list and compare commands
  • Reject the legacy sort names with a direct route to get-endpoints.ts --sort throughput
  • Keep CLI help and SKILL.md guidance aligned on the supported sort modes
  • Add reverse-ordered provider endpoints where the lower-output-capacity endpoint generates faster
  • Add CLI regression coverage for both removed aliases and the live endpoint workflow

Testing

  • npm ci
  • npm test (6 tests)
  • git diff --check
  • Mutation check: reversing endpoint throughput sorting fails the regression test

Regression risk

Low. Existing newest, price, and context sorting is unchanged. The intentional compatibility change is that the two misleading sort names now fail with an actionable error instead of returning a false ranking.

Live acceptance

  • On main, list-models.ts --sort throughput returned 415 live models with a zero exit code. The first models had a 1,800,000-token output limit.
  • On this branch, the same command exits with code 1 and directs the user to live endpoint throughput data.
  • On main, compare-models.ts ... --sort speed placed the 1,800,000-token model before the 16,384-token model. On this branch, it rejects that false comparison.
  • list-models.ts --sort context fetched 415 live models and returned them in descending context-length order.
  • The live endpoint API returned provider lists for four current models. Their throughput_last_30m values were null, so a numeric live throughput order was not available to observe. The deterministic endpoint fixture covers that data case.

Installed acceptance

gh skill install yazcaleb/skills openrouter-models --pin de43c1b installed the skill into a clean directory. From that installed copy:

  • Both removed sort aliases exited with code 1 and printed the get-endpoints.ts route.
  • The updated help listed only the supported model-level sorts.
  • The live models request returned 415 models in descending context-length order.
  • The live endpoint request returned three providers for openai/gpt-4o-mini.
  • Those providers had null 30-minute throughput values, so the live numeric ordering constraint still applies.

max_completion_tokens is capacity, not tokens/sec. Reject legacy model-level
speed and throughput sorts and direct users to live endpoint performance data.

Fixes OpenRouterTeam#13
Put two endpoints in reverse throughput order within one model fixture. The
regression test now fails if endpoint throughput sorting is removed or reversed.
@yazcaleb

Copy link
Copy Markdown
Author

@jackyliang-openrouter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: --sort throughput/speed in list-models.ts and compare-models.ts sorts by max_completion_tokens, not real throughput

1 participant