Non-streaming requests keep decoding after the client disconnects #644

Activity

1. gdevenyi commented on Oct 13, 2026

   

   gdevenyi

   on Oct 13, 2026

   ContributorAuthor

   More actions

   Tested in the 1080 Ti deployment branch, rebuilt today on main 9c635a1 (torch 2.14.1+cu126, Triton 3.8, GTX 1080 Ti sm_61, Qwen3.8-Flash-Next-NVFP4 at 262K): main now wraps /v1/messages and /v1/responses in the disconnect watcher; the branch adds the n>1 abort test and the pure-ASGI request recorder. tests/server/test_stream_cancellation.py and tests/server/test_request_recorder_disconnect.py pass.
2. SUKUNA-AI commented on Oct 19, 2026

   

   SUKUNA-AI

   on Oct 19, 2026

   More actions

   A data point from a different deployment: a single RTX 5070 Ti 16 GB, FreeToken 0.1.3 plus a fork, serving only
   /v1/chat/completions behind our own ASGI wrapper. We worked around the same root cause from outside FreeToken, and
   it agrees with the diagnosis here.

   Still present on main 4d2e2e2. The non-streaming branches await generate_full without watching the connection
   (server/openai_api.py:212-215, server/anthropic_api.py:139-142). stream_with_cancellation is used only on the
   streaming branches (openai_api.py:209, anthropic_api.py:136, responses_api.py:172). _record_request_middleware
   is still an @app.middleware("http") (api_server.py:485), i.e. a BaseHTTPMiddleware. The earlier comment says
   /v1/messages and /v1/responses are wrapped on main; on 4d2e2e2 we only see the wrapper on their streaming
   branches. We may be looking at the wrong place.

   What we run (serving/guard.py, serving/serve_guarded.py in our repo; not part of FreeToken):

   • A pure ASGI middleware, added with app.add_middleware(...) so it sits outside the BaseHTTPMiddleware. It reads
     the request body itself, replays it to the app, and runs a watcher task on the real receive(). On
     http.disconnect it dispatches state.abort_user(uid) and only then cancels the app task. Our first version
     cancelled first; the non-streaming waiter then unwound before the abort was dispatched.
   • Abort dispatch runs in its own retained, de-duplicated task and is awaited for a bounded time
     (asyncio.shield). The cleanup task is itself shielded, so a second server-side cancellation cannot abandon it.
   • openai_api.generate_full is wrapped so that a CancelledError also dispatches the abort from its own task. An
     AnyIO cancel scope can cancel every awaited checkpoint inside the handler, including an inline await abort_user.

   Evidence.

   • A CPU unit test (serving/test-guard.py): a fake /v1/chat/completions app that sleeps is cancelled when the
     client sends http.disconnect, and the generation slot is released.
   • On the GPU: in our 2-hour soak, 29/29 client cancellations passed, and the next request was served after a median
     of 0.37 s from the disconnect (max 0.38 s).
   • A live non-streaming disconnect check exists in our runbook but has not been run, so the non-streaming case is
     covered only by the unit test above.

   We think the pure-ASGI recorder from the linked commits plus an explicit watcher on the non-streaming branches is
   the right fix inside FreeToken; our wrapper is just evidence that the approach works. One detail from our side: send
   the abort before cancelling the waiter.

   Our implementation, for reference (fork of 0.1.3, Apache-2.0; paths in the fork):

   • serving/guard.py
   • serving/serve_guarded.py
   • serving/test-guard.py