Repository navigation
Engine wedges under sustained agentic load: active=1, tps=0 until restart (Windows, Qwen3.6-35B-A3B-FP8) #453
Description
Before you start
- I have read the FAQ and my problem is not answered there.
- I have read the Roadmap and this is not already planned there.
- I have searched existing issues and found no duplicate.
- I am on the latest release, or on a freshly rebuilt
mainwhen building from source.
What happened
The FreeToken engine (daemon --auto-restart, engine serving) periodically wedges mid-generation: it reports maintenance: serving, keeps the request active (active=1) but produces zero tokens per second for 10–20+ minutes, until killed/restarted. Reproduced 7 times in one evening (2026-09-12, timestamps below) under sustained agentic load (an OpenAI-compatible agent client sending long-context turns). Thinking ON/OFF makes no difference — reproduced with enable_thinking=false on every request.
Environment
FreeToken: 0.1.2+g816c324d0 (engine+daemon, pip/git build)
OS: Windows 11 (10.0.26200), Python 3.12
GPU: NVIDIA RTX 5060 Ti 16 GB (driver 616.64), CPU 12 threads usable, 64 GB RAM
Model: Qwen3.6-35B-A3B-FP8 (FTW shards)
Serve args: --moe-backend auto --max-running-requests 2 --num-tokens 73728 --memory-ratio 0.95 --moe-cpu-threads 12 --max-output-tokens 8192 --cache-type radix --enable-cache-report --enable-special-token-ckpt --sampling-defaults model
Load profile: one long-lived agent client (chat-completions, streaming), turns of 10–60k prompt tokens, continuous for hours.
Symptom (identical every time)
Engine healthy at ~20+ tok/s decode.
At some point a request stalls: health shows engineRunning: true / serving, but throughput drops to 0 with 1 active request.
A canary request sent directly to the engine times out (30s+) while the engine claims serving.
Only a restart (stop serve + reload) recovers it; the wedged request never completes.
Typical cadence under load: every 15–40 minutes.
Log excerpt (external watchdog observing the daemon health API):
2026-09-12 18:11:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists. 2026-09-12 18:31:28 Stuck-state first observation (active=1 tps=0) — ... 2026-09-12 18:37:12 Canary failed while engine claims serving: The request was aborted: The operation has timed out. — restarting engine. 2026-09-12 18:47:13 Canary failed ... restarting engine. 2026-09-12 18:56:27 Stuck-state first observation (active=1 tps=0) — ... 2026-09-12 19:02:12 Canary failed ... restarting engine. 2026-09-12 20:11:27 Stuck-state first observation (active=1 tps=0) — ... 2026-09-12 20:17:12 Canary failed ... restarting engine. 2026-09-12 20:51:27 Stuck-state first observation (active=1 tps=0) — ... 2026-09-12 20:57:12 Canary failed ... restarting engine.
(All 7 wedges within ~3 hours; two more earlier the same day.)
What we already ruled out
Not thinking/reasoning: reproduced with enable_thinking=false forced on every request (chat_template_kwargs), including the 20:51 wedge.
Not the client: the wedge persists even with the client disconnected; a fresh minimal request (10 tokens) also hangs until restart.
Not OOM: RAM ~20 GB free during wedges, VRAM steady, no crashes/logs at wedge time.
Not cache rebuild: wedges happen mid-conversation long after load/cache build completes.
Suspicions (unverified)
A deadlock/livelock in the MoE CPU-offload streaming path (PCIe H2D ~6.5 GB/s here) under long-context decode.
Interaction of --enable-special-token-ckpt or radix checkpointing with long prompts.
Happy to run instrumented builds or share full engine logs — the setup reproduces the issue reliably under agentic load.
How did you install FreeToken
pip / uv wheel
FreeToken version
FreeToken 0.1.2+g816c324d0 (daemon+serve via pip venv).
OS
Windows 11
OS details
Windows 11 native (10.0.26200), no WSL. Python 3.12.14.
GPU and driver
NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM, driver 616.64
CPU and system RAM
Intel Core i5-10400F, 64 GB DDR4-2667 (2×32, Samsung)
Checkpoint
Qwen3.6-35B-A3B-FP8 (FTW shards)
Command
Full log
2026-08-27 21:16:30 Engine needs heal: running=True model=S:\LmStudio\Models\gpt-oss-20b cache=loading -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 21:16:32 Heal switch issued for gpt-oss-20b.
2026-08-27 21:17:47 Engine needs heal: running=False model=S:\LmStudio\Models\gpt-oss-20b cache= -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 21:17:47 Heal switch issued for gpt-oss-20b.
2026-08-27 21:26:29 Engine needs heal: running=True model=S:\LmStudio\Models\gpt-oss-20b cache=loading -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 21:26:29 Heal switch issued for gpt-oss-20b.
2026-08-27 21:31:29 Engine needs heal: running=False model=S:\LmStudio\Models\gpt-oss-20b cache= -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 21:31:29 Heal switch issued for gpt-oss-20b.
2026-08-27 21:36:30 Engine needs heal: running=True model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=loading -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 21:36:30 Heal switch issued for gpt-oss-20b.
2026-08-27 22:01:35 Engine needs heal: running=True model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=loading -> S:\LmStudio\Models\gpt-oss-20b
2026-08-27 22:01:36 Heal switch issued for gpt-oss-20b.
2026-08-27 22:36:27 Engine needs heal: running=True model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=stopping -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-08-27 22:36:28 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-08-29 12:01:31 Engine needs heal: running=False model= cache= -> S:\LmStudio\Models\gpt-oss-20b
2026-08-29 12:01:32 Heal switch issued for gpt-oss-20b.
2026-09-01 22:21:31 Engine needs heal: running=False model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-01 22:21:31 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-01 22:36:30 Engine needs heal: running=False model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-01 22:36:30 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-04 09:31:47 Canary failed while engine claims serving: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. — restarting engine.
2026-09-04 09:31:47 Recovery switch issued.
2026-09-06 13:44:02 Engine needs heal: running= model= cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-06 14:01:31 Engine needs heal: running=False model= cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-06 14:01:31 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-07 08:51:29 Engine needs heal: running=False model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-07 08:51:29 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-07 16:16:29 Engine needs heal: running=False model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache= -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-07 16:16:29 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-07 22:31:27 Engine needs heal: running=True model=M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=loading -> S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-07 22:31:28 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-07 23:21:28 Engine needs heal: running=True model=S:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=serving -> M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-07 23:21:33 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-11 08:06:29 Engine needs heal: running=False model=M:\LmStudio\Models\Ornith-1.5-35B-A3B-FP8 cache= -> M:\LmStudio\Models\Ornith-1.5-35B-A3B-FP8
2026-09-11 08:06:29 Heal switch issued for Ornith-1.5-35B-A3B-FP8.
2026-09-11 10:11:32 Engine needs heal: running=True model=S:\LmStudio\Models\gpt-oss-120b cache=loading -> S:\LmStudio\Models\gpt-oss-120b
2026-09-11 10:11:36 Heal switch issued for gpt-oss-120b.
2026-09-12 03:12:13 Canary failed while engine claims serving: The request was aborted: The operation has timed out. — restarting engine.
2026-09-12 03:12:20 Recovery switch issued.
2026-09-12 03:16:16 Engine needs heal: running=False model=M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache= -> M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-12 03:16:16 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-12 03:16:28 Engine needs heal: running=True model=M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8 cache=loading -> M:\LmStudio\Models\Qwen3.6-35B-A3B-FP8
2026-09-12 03:16:29 Heal switch issued for Qwen3.6-35B-A3B-FP8.
2026-09-12 13:16:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 17:56:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 18:11:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 18:31:28 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 18:37:12 Canary failed while engine claims serving: The request was aborted: The operation has timed out. — restarting engine.
2026-09-12 18:37:21 Recovery switch issued.
2026-09-12 18:41:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 18:47:13 Canary failed while engine claims serving: The request was aborted: The operation has timed out. — restarting engine.
2026-09-12 18:47:21 Recovery switch issued.
2026-09-12 18:56:27 Stuck-state first observation (active=1 tps=0) — will canary on next cycle if it persists.
2026-09-12 19:02:12 Canary failed while engine claims serving: The request was aborted: The operation has timed out. — restarting engine.
2026-09-12 19:02:20 Recovery switch issued.
Anything else
No response
Activity
fix(engine): pipeline ranks agree on the KV pool, and check the prefi…
feat(scheduler): say why the head of the prefill queue is not being a…
feat(distributed): log a wait between ranks that has gone on too long
minatoAI commentedon Oct 15, 2026
Hi -- this may or may not be your bug, but it looks close enough to be worth a comment: same
platform, same shape, and I have before/after numbers from a Windows box (Win11, RTX 5070 Ti
16 GB, gemma4 26B-A4B NVFP4 offload).
Same box, same usage, two states of
expandable_segments:(hourly medians, MB)
only ever falls
stable, and comes back up
¹ all 576 of those sub-30 samples are the first second after a prefill, not degradation.
In the OFF state the GPU stayed at 99% utilisation while drawing about half its normal power and
the serve log said nothing beyond its startup lines -- from the outside that reads exactly as
active=1/tps=0until restart, which is why I thought it was worth posting here.Two things worth checking on a wedged run, both cheap:
2.14.0is the first Windows release whereexpandable_segmentsis actually compiled in;v2.13.0andrelease/2.13still gate itbehind
if(NOT WIN32)inc10/cuda/CMakeLists.txt.PYTORCH_ALLOC_CONF/PYTORCH_CUDA_ALLOC_CONF. If itdoes,
_ensure_expandable_segments()returns early and logs nothing, so the default issilently lost.
Setting
expandable_segments:Trueis the entire difference between the two columns above.Caveat: I cannot claim your wedge is this -- the two checks above would tell you in a minute.
Happy to run any test you'd like; these numbers come from a small sampler I keep outside the
engine.