Laguna server batch prefill fails at >=7k tokens: routed MoE intermediate quantize launch invalid argument

[nexus-cw]

Environment

• Hardware: NVIDIA GB10 (DGX Spark class, sm_121a), CUDA 13, Linux aarch64
• Branch: laguna-s2.1 @ 448d569 ("Tune Laguna sampling defaults")
• Model: Laguna-S-2.1 Q4_K_M GGUF (~65 GiB), fully resident, --cuda
• Server: ds4-server -m laguna-s-2.1-Q4_K_M.gguf --cuda --ctx 32768

Symptom

Any chat/messages request whose prompt is >= ~7,069 tokens fails server-side prefill immediately after the first layer:

ds4: CUDA Laguna routed MoE intermediate quantize launch failed: invalid argument
ds4: Laguna batch prefill failed in routed experts after 1/48 layers

The client gets a 500; every retry fails identically (deterministic, not transient).

Bisection (server chat path, ctx 32768)

prompt tokens	result
<= 6,049	prefill OK (e.g. 6049/6049 avg=558.56 t/s 10.830s, generation fine)
>= ~7,069	fails as above, always after layer 1/48
24,438 (real workload)	fails, every retry

Why this looks like launch geometry, not memory

• Plenty of free memory at failure time (~58+ GiB available); the same server handles short prompts fine at the same ctx.
• The one-shot CLI path (ds4 binary, same model, same build) prefilled a ~16k-token prompt without error at ~2,198 t/s — the failure is specific to the server batch prefill path.
• invalid argument from a CUDA launch at a token-count threshold in the low-7k range is the classic signature of a grid-dimension limit being exceeded (e.g. tokens x top-k experts overflowing a 65,535-capped grid dimension) in the batched routed-MoE intermediate-quantize launch. Offered as a hypothesis — we have not bisected the kernel itself.

No mitigation available

--prefill-chunk is a hard startup reject for Laguna, so prompts cannot be chunked below the threshold.

Minimal repro

1. Build laguna-s2.1 @ 448d569 with CUDA; start ds4-server -m <laguna-s-2.1 Q4_K_M> --cuda --ctx 32768 --port 8010.
2. Send one OpenAI chat completion with a ~7,500-token prompt:

python3 -c 'import json,urllib.request,random; random.seed(1); w=["alpha","bravo","charlie","delta","echo","foxtrot","golf","hotel","india","juliet"]; p="Summarize this list briefly: "+" ".join(random.choice(w) for _ in range(7500)); body=json.dumps({"model":"laguna-s-2.1-chat","messages":[{"role":"user","content":p}],"max_tokens":20}).encode(); print(urllib.request.urlopen(urllib.request.Request("http://127.0.0.1:8010/v1/chat/completions",data=body,headers={"Content-Type":"application/json"})).read())'

Expected: completion. Actual: HTTP 500 with the two log lines above; a short prompt on the same running server returns 200.

Real-workload impact

We hit this running the claude CLI against the server's Anthropic-compatible surface: its ~24.4k-token first request can never complete (8 retries, clean API error), so any long-system-prompt agent client is hard-blocked on Laguna.

Re-verified today (2026-08-05) at the same branch head before filing.

🤖 Generated with Claude Code