Skip to content

/v1/responses reports usage.output_tokens_details.reasoning_tokens = 0 even when the reply has reasoning #717

Open

Description

@gdevenyi

What happened

With thinking on, /v1/responses returns a reasoning output item holding the full trace, but usage.output_tokens_details.reasoning_tokens is always 0, both non-streaming and in the streamed terminal events. A client that budgets or reports from this field is told no reasoning happened. When max_output_tokens cuts a reply off, the client cannot tell from usage how much of the budget the reasoning used. /v1/chat/completions has no completion_tokens_details at all, so it gives no count either (#307 and a comment on #302 mention this in passing).

Minimal reproduction

PYTHONPATH=python python -c "
from freetoken.server.generation import GenResult
from freetoken.server.responses_api import build_responses_response, ResponsesRequest
req = ResponsesRequest(model='m', input='hi')
r = GenResult(reasoning='Let me think about this carefully. '*20, content='4', tool_calls=[],
              finish_reason='stop', prompt_tokens=12, completion_tokens=180)
u = build_responses_response(r, req, 'resp_1', 0).usage
print('output_tokens', u.output_tokens, 'reasoning_tokens', u.output_tokens_details.reasoning_tokens)
"
output_tokens 180 reasoning_tokens 0

Cause (current main, 9b68e00)

  • python/freetoken/server/responses_api.py:481-490: _usage() hard-codes OutputTokensDetails(reasoning_tokens=0). Every Responses usage object is built by it: the non-stream builder (:452) and the stream's response.incomplete / response.completed / response.failed events (:717, :726, :739, :754).
  • Nothing on the generation path counts reasoning tokens. GenDone (python/freetoken/server/generation.py:112) and GenResult (:124-132) carry only prompt_tokens/completion_tokens, and openai_api.py:637-646 _usage() has no completion_tokens_details.

Proposed fix

Count reasoning tokens once in the generation path, carry the count on GenResult/GenDone, and report it as output_tokens_details.reasoning_tokens (Responses) and completion_tokens_details.reasoning_tokens (chat completions). The counter could start in the same state as the reasoning parser (force_reasoning, generation.py:389-422), so that a template that pre-opens <think> is counted from the first token. It would stop at the think-end token. If generation ends before that token, all completion tokens would count as reasoning, so a reply truncated mid-reasoning reports its real count rather than 0. With no reasoning parser, or thinking off, the count stays 0.

Environment

Seen serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB; the reproduction above is CPU-only on main 9b68e00.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions