Repository navigation
Streaming /v1/messages: message_start reports input_tokens 0 and no cache fields #718
Description
What happened
A streamed /v1/messages response always opens with message_start.message.usage = {"input_tokens": 0, "output_tokens": 0} and no cache fields, even under --enable-cache-report. The real counts only arrive in the final message_delta. Anthropic's streaming format reports the prompt's input_tokens and cache fields in message_start, so a client that reads input usage there sees 0 prompt and 0 cache-read tokens on every streamed turn, while the non-streaming response reports the real values.
--enable-cache-report |
message_start usage (expected) |
message_start usage (got) |
|---|---|---|
| off | input_tokens: 1200 |
input_tokens: 0 |
| on | input_tokens: 176, cache_read_input_tokens: 1024 |
input_tokens: 0 (no cache field) |
Minimal reproduction (CPU only, PYTHONPATH=python):
import asyncio, json
from freetoken.server import anthropic_api as A
from freetoken.server.generation import ContentDelta, GenDone
async def ev():
yield ContentDelta("hi")
yield GenDone("stop", 1200, 5, cached_tokens=1024)
async def main():
for cr in (False, True):
async for f in A.anthropic_event_stream(ev(), "m", 1, cache_report=cr):
d = json.loads(f.splitlines()[1][6:])
if d["type"] == "message_start":
print(f"cache_report={cr} message_start:", d["message"]["usage"])
elif d["type"] == "message_delta":
print(f"cache_report={cr} message_delta:", d["usage"])
asyncio.run(main())cache_report=False message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=False message_delta: {'input_tokens': 1200, 'output_tokens': 5}
cache_report=True message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=True message_delta: {'input_tokens': 176, 'output_tokens': 5, 'cache_read_input_tokens': 1024}
Cause (current main, 9b68e00)
python/freetoken/server/anthropic_api.py:475-482:anthropic_event_streamsendsmessage_startbefore it reads any event, with a hard-codedAnthropicUsage(input_tokens=0, output_tokens=0). The comment there says "input_tokens filled in by the terminal message_delta".python/freetoken/server/anthropic_api.py:582-588: the real usage, built by_anthropic_usage(L417-428), is only sent onGenDone.- The counts are available long before
GenDone. The scheduler sendsPromptAdmittedMsg(prompt_tokens, cached_tokens)when it admits the prompt (python/freetoken/scheduler/scheduler.py:885). The tokenizer turns that into an empty ack (python/freetoken/tokenizer/server.py:43-51), and_generate_events_impladds it up (python/freetoken/server/generation.py:702-704). But noGenEventcarries these counts out, so the Anthropic streamer cannot use them.
Proposed fix
Have _generate_events_impl emit a small event (for example PromptAdmitted(prompt_tokens, cached_tokens)) when the admission ack arrives. anthropic_event_stream would then send message_start on that event, filling usage through _anthropic_usage(prompt, 0, cached, cache_report), so its input_tokens/cache_read_input_tokens split matches the final message_delta. If a keepalive is due before admission, message_start would go out first with the current zeros, as it does today. The OpenAI and Responses streamers can ignore the new event.
Environment
Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB; the reproduction above is CPU-only on main 9b68e00.
Activity