fix: extract Google Gemini token usage and price cached tokens correctly - #8
Open
MukundaKatta wants to merge 1 commit into
Open
fix: extract Google Gemini token usage and price cached tokens correctly#8MukundaKatta wants to merge 1 commit into
MukundaKatta wants to merge 1 commit into
Conversation
The README, TypeScript types, and CHANGELOG all advertised that `defaultExtractUsage` (and therefore `measureLLM`) supported Google Gemini responses, but the extractor only inspected a `usage` key. The Gemini SDK (`@google/genai`) returns usage under `usageMetadata` with `promptTokenCount` / `candidatesTokenCount` / `cachedContentTokenCount`, so a Gemini call returned `null` and its tokens were silently recorded as zero cost — exactly the under-reporting this library exists to catch. Changes: - `defaultExtractUsage` now recognizes the Gemini `usageMetadata` shape (including when nested under a `response` property) alongside the existing Anthropic / OpenAI `usage` shapes. - When a `usage`/`usageMetadata` object is present but contains no recognized token fields, return `null` instead of a misleading all-zero record. - Add `cacheRead` rates to the Gemini 2.5/2.0 PRICING entries so cached context tokens are priced at the (much lower) cached rate rather than falling back to the full input rate. - Add a runnable Gemini call to examples/demo.js. - Stabilize the previously flaky "measure tracks latency" test, which asserted `elapsed >= 5` against a separately-sampled wall clock; it now asserts on the library's own recorded `step.latencyMs`. - Docs: README API section, `src/index.d.ts` doc comment, and CHANGELOG updated; test count 26 -> 29. Validation: `npm run lint` clean; `npm test` 29/29 (run 5x, no flakes); `npm run test:examples` runs and shows the Gemini step priced correctly; cost-under-report CI smoke check still passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The README, TypeScript types, and CHANGELOG all advertised that
defaultExtractUsage(and thereforemeasureLLM) supported Google Gemini responses, but the extractor only ever inspected ausagekey. The Gemini SDK (@google/genai) returns token usage underusageMetadata(promptTokenCount/candidatesTokenCount/cachedContentTokenCount), so a real Gemini response returnednulland its tokens were silently recorded as zero cost — exactly the under-reporting this library exists to prevent.Changes
defaultExtractUsagenow recognizes the GeminiusageMetadatashape (including when nested under aresponseproperty) alongside the existing Anthropic / OpenAIusageshapes.usage/usageMetadataobject is present but contains no recognized token fields, the extractor returnsnullinstead of a misleading all-zero record.cacheReadrates to the Gemini 2.5/2.0PRICINGentries so cached-context tokens are priced at the (much lower) cached rate rather than falling back to the full input rate.measure tracks latencytest assertedelapsed >= 5against a separately-sampled wall clock; Node timers can fire a hair early relative toperformance.now(), so it failed intermittently. It now asserts on the library's own recordedstep.latencyMs.response-nested usage, and the no-known-fields guard. Test count 26 → 29.src/index.d.tsdoc comment, CHANGELOG, and a runnable Gemini call inexamples/demo.js.Validation
npm run lint— clean (syntax check of src + tests).npm test— 29/29 passing, run 5 times consecutively with no flakes.npm run test:examples— runs; the newrerank [gemini-2.5-flash]step is now priced ($0.000285) instead of$0.claude-sonnet-4) still passes.coveragejob usesc8v11, which is incompatible with the very latest local Node (v26); it fails identically on unmodifiedmainand works on CI's Node 22. Not affected by this change.🤖 Generated with Claude Code