Skip to content

Streaming /v1/messages: message_start reports input_tokens 0 and no cache fields #718

Open

Description

@gdevenyi

What happened

A streamed /v1/messages response always opens with message_start.message.usage = {"input_tokens": 0, "output_tokens": 0} and no cache fields, even under --enable-cache-report. The real counts only arrive in the final message_delta. Anthropic's streaming format reports the prompt's input_tokens and cache fields in message_start, so a client that reads input usage there sees 0 prompt and 0 cache-read tokens on every streamed turn, while the non-streaming response reports the real values.

--enable-cache-report message_start usage (expected) message_start usage (got)
off input_tokens: 1200 input_tokens: 0
on input_tokens: 176, cache_read_input_tokens: 1024 input_tokens: 0 (no cache field)

Minimal reproduction (CPU only, PYTHONPATH=python):

import asyncio, json
from freetoken.server import anthropic_api as A
from freetoken.server.generation import ContentDelta, GenDone
async def ev():
    yield ContentDelta("hi")
    yield GenDone("stop", 1200, 5, cached_tokens=1024)
async def main():
    for cr in (False, True):
        async for f in A.anthropic_event_stream(ev(), "m", 1, cache_report=cr):
            d = json.loads(f.splitlines()[1][6:])
            if d["type"] == "message_start":
                print(f"cache_report={cr} message_start:", d["message"]["usage"])
            elif d["type"] == "message_delta":
                print(f"cache_report={cr} message_delta:", d["usage"])
asyncio.run(main())
cache_report=False message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=False message_delta: {'input_tokens': 1200, 'output_tokens': 5}
cache_report=True message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=True message_delta: {'input_tokens': 176, 'output_tokens': 5, 'cache_read_input_tokens': 1024}

Cause (current main, 9b68e00)

  • python/freetoken/server/anthropic_api.py:475-482: anthropic_event_stream sends message_start before it reads any event, with a hard-coded AnthropicUsage(input_tokens=0, output_tokens=0). The comment there says "input_tokens filled in by the terminal message_delta".
  • python/freetoken/server/anthropic_api.py:582-588: the real usage, built by _anthropic_usage (L417-428), is only sent on GenDone.
  • The counts are available long before GenDone. The scheduler sends PromptAdmittedMsg(prompt_tokens, cached_tokens) when it admits the prompt (python/freetoken/scheduler/scheduler.py:885). The tokenizer turns that into an empty ack (python/freetoken/tokenizer/server.py:43-51), and _generate_events_impl adds it up (python/freetoken/server/generation.py:702-704). But no GenEvent carries these counts out, so the Anthropic streamer cannot use them.

Proposed fix

Have _generate_events_impl emit a small event (for example PromptAdmitted(prompt_tokens, cached_tokens)) when the admission ack arrives. anthropic_event_stream would then send message_start on that event, filling usage through _anthropic_usage(prompt, 0, cached, cache_report), so its input_tokens/cache_read_input_tokens split matches the final message_delta. If a keepalive is due before admission, message_start would go out first with the current zeros, as it does today. The OpenAI and Responses streamers can ignore the new event.

Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB; the reproduction above is CPU-only on main 9b68e00.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions