CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists

[alfredodeza]

Summary

Unlike most GPU-ineligible architectures (e.g. Phi's LayerNorm gap, #3075), which fall
back to CPU inference with a warning, Qwen3.5/Qwen3.8-architecture GGUFs (hybrid Gated
DeltaNet / SSM layers, general.architecture = qwen35) are refused completely —
there is no CPU inference path at all today. Confirmed with two different downloads:

apr chat ~/Downloads/Qwen3.8-27B-UD-IQ2_XXS.gguf
apr chat ~/Downloads/Qwen3.5-4B-UD-Q4_K_XL.gguf

Both fail identically, even without --gpu:

[GGUF model parse failed: Format error: Architecture 'qwen35' uses SSM/Gated Delta Net
layers which are not yet supported for inference. Use a standard transformer model
(e.g., Qwen2.5, LLaMA, Mistral) or wait for SSM support in a future release., will use CPU]
You: hey
Assistant: [Error: Failed to create GGUF model: Format error: Architecture 'qwen35' ...]

The chat loop even prints "will use CPU" in the CUDA-init warning line, but then the
CPU load of the same GGUF fails with the exact same error — because the refusal is in
the shared GGUF transformer constructor (QuantizedGGUFTransformer::from_gguf), which
both the CPU and GPU code paths call. There is no degraded-but-working CPU mode; the
model simply cannot be loaded by this binary in any form.

Root cause

crates/aprender-serve/src/gguf/transformer.rs::unsupported_architecture_reason is
called unconditionally inside QuantizedGGUFTransformer::from_gguf (GH-704), before
backend selection, and returns Err(FormatError) whenever any tensor name contains
ssm_/ssm.. This is upstream of the CPU/GPU fork — it blocks both.

Ask

1. Implement a scalar/SIMD CPU Gated DeltaNet (linear attention) forward pass for
   GGUF-loaded hybrid layers — recurrent state + short causal conv1d + gated delta rule,
   per Qwen3.5/Qwen3-Next's architecture — analogous to what
   crates/aprender-serve/src/gpu/scheduler/linear_attn.rs already sketches for the GPU
   scheduler (SafeTensors path), but for the GGUF QuantizedGGUFTransformer CPU path.
2. Until CPU inference exists, consider narrowing the current blanket refusal message so
   it doesn't imply CPU already works ("will use CPU" in the CLI's CUDA-fallback log line
   is actively misleading for this architecture — see below) — or explicitly state in
   the error that neither backend supports it yet, to save users a round trip.
3. Once a real CPU path lands, extend apr qa/apr validate to report this
   architecture's status accurately (load-time capability, not just a runtime crash).

Secondary bug: misleading CLI message

The chat CLI's own log line claims a fallback that doesn't exist:

[GGUF model parse failed: ... will use CPU]

printed by crates/apr-cli/src/commands/chat_load_tokenizers.rs:166
(format!("[GGUF CUDA init failed: {}, will use CPU]", e)). That code path assumes any
GGUF parse failure on the GPU attempt is recoverable by retrying on CPU — true for the
Phi LayerNorm case (#3075), false here, since the same parse failure happens on the
CPU retry too. Worth a small fix regardless of when the real SSM CPU path lands: don't
print "will use CPU" when the failure reason is architecture-unsupported rather than
GPU-capability-specific, since the retry is guaranteed to fail with the identical message.

Evidence

• crates/aprender-serve/src/gguf/transformer.rs:73-86,158-167 (refusal call site,
  upstream of backend selection)
• crates/apr-cli/src/commands/chat_load_tokenizers.rs:166 (misleading "will use CPU" log line)
• Repro commands above, no --gpu flag, same host

Related

• GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 (GPU support for the same architecture — companion ticket)
• Phi-2/Phi-3 GGUF never GPU-eligible: LayerNorm kernel exists but isn't wired into forward_gpu_resident #3075 (Phi LayerNorm — contrast case where CPU fallback genuinely works)
• Docs: publish a GPU-vs-CPU model architecture support table (no user-facing doc exists today) #3077 (GPU/CPU model support table doc) — needs a row/footnote for qwen35

[noahgift]

Triage (0.67.0 train, PMAT-1098 — decided plan, not a question):

Confirmed on lambda with a local Qwen3.5-0.8B-Q4_K_M.gguf (arch qwen35, 24 layers = 18 ssm_* Gated-DeltaNet + 6 gated full-attention): the published 0.66.0 apr run refuses it from unsupported_architecture_reason before backend selection, exactly as described. Qwen3.8-* files in the wild carry general.architecture=qwen35; this repo treats them as one architecture.

Split into three deliverables:

1. Ships in 0.67.0 — the honest refusal: the message says neither CPU nor GPU implements Gated DeltaNet yet and names GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090/CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091; the CLI stops printing "will use CPU" for a format-level refusal and exits non-zero on the first attempt; the qwen3_5 row in arch_constraints_fallback.rs describes the real hybrid; apr qa/apr validate report it as a structured load-time capability. In progress on branch PMAT-1098-qwen35-honest-refusal.
2. CPU Gated DeltaNet in the GGUF path (CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091 ask 1) — in progress on PMAT-1098-qwen35-cpu-deltanet with greedy-token parity against llama.cpp master on the 0.8B fixture as the gate. It ships in 0.67.0 only if parity is green before the 2026-09-12T18:00Z cut; otherwise it moves to 0.68.0 with the measured gap recorded here.
3. GPU path (GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090) — after (2) exists, because the parity oracle for the GPU scheduler is the CPU implementation; scheduled for 0.68.0 unless (2) lands early.

Dissent recorded: adding two P1s two days before a train's cut is exactly what the schedule's "slip first" rule is for; the honest refusal is the part that cannot slip because today's message actively misleads.

[noahgift]

Ask 2 (honest refusal, no phantom CPU fallback) is in PR #3099 with the measured acceptance on the 0.8B fixture — the message now says neither backend implements Gated DeltaNet yet, the CLI exits non-zero on the first attempt, and apr qa reports it as a structured capability failure. Ask 1 (CPU Gated DeltaNet) is in progress on branch PMAT-1098-qwen35-cpu-deltanet with llama.cpp-master greedy parity as the gate; ships in 0.67.0 only if green before the 2026-09-12T18:00Z cut.

[noahgift]

Measured status of ask 1 (CPU Gated DeltaNet), 2026-09-10 16:42Z on lambda: the first implementation pass did not reach a working path. The tree it left builds and its one new qwen35 unit test passes, but apr run ~/models/Qwen3.5-0.8B-Q4_K_M.gguf --prompt "The capital of France is" --max-tokens 16 still exits 8 with the new honest refusal ("NEITHER the CPU nor the GPU backend implements Gated DeltaNet/SSM layers yet"), i.e. the loader never routes qwen35 layers to a DeltaNet forward. llama.cpp master runs the same file, so the oracle for the parity gate is in place. The honest refusal (ask 2) is PR #3099 and ships in 0.67.0 either way; the CPU forward path gets a second pass now and ships in 0.67.0 only if greedy-token parity with llama.cpp is green before the 2026-09-12T18:00Z cut, otherwise in 0.68.0 — the measurement will be posted here.

[shifulegend]

If anyone needs a working CPU implementation of the Qwen3.5/3.8 Gated DeltaNet/SSM forward pass as a reference for the Rust port, I recently implemented this in pure C99 for Project Zero (we parse the qwen35 hybrid layers natively from GGUF).

The scalar and SIMD state-gating math might be useful to look at when sketching out the QuantizedGGUFTransformer CPU path. Happy to share notes on the short causal conv1d handling if it helps get this unblocked!

[alfredodeza]

@shifulegend if you have anything that you have that could help here that would be incredible. Thank you so much for being aware on the progress we are making!

[noahgift]

Ask 1 status, measured 2026-09-11 ~05:10Z on lambda. The CPU path works end to end in draft #3114 (PMAT-1098). Stacked on #3099 (refusal) and #3110 (Q5_K).

• apr run ~/models/Qwen3.5-0.8B-Q4_K_M.gguf --prompt "What is 2+2? Answer in one word." → Four.. For "The capital of France is" it gives The capital of France is **Paris**.. apr chat answers coherently. The dense Qwen2.5 path is unchanged.
• Root causes. The failed first pass had nine defects, all located by comparing each op against llama-eval-callback's per-node sums:
  • DeltaNet: q/k L2 norm was global and applied twice, instead of per head.
  • DeltaNet: RoPE was applied, but llama.cpp applies none there.
  • DeltaNet: a stray L2 norm on the output.
  • DeltaNet: beta went through sigmoid twice.
  • DeltaNet: conv output went through SiLU twice.
  • Attention: full-width GPT-J RoPE instead of partial NEOX (n_rot 64 of 256).
  • Attention: the output gate was never applied.
  • Attention: head width taken as 128 instead of 256.
  • Q5_K weights decoded with an invented layout (attn_qkv and ssm_out are Q5_K). Fixed in fix(quantize): Q5_K readers decoded an invented layout — every Q5_K tensor on the CPU path and in apr import was wrong (PMAT-1101) #3110.
• Unit tests. New tests cover per-head L2, the sigmoid gate and partial NEOX RoPE. Reverting either fix turns them red.

The 0.67 gate (greedy 32-token parity with llama.cpp master): NOT green as written. From examples/qwen35_parity.rs, recorded in evidence/parity/qwen35/lambda/parity.json:

prompt	free-running greedy	teacher-forced	disagreement
0 (5 tokens)	diverges at step 0	27/28	llama.cpp's token is realizar's rank 2, gap 0.145 logits
1 (10 tokens)	identical, 23/23	23/23	none
2 (16 tokens)	diverges at step 13	16/17	rank 2, gap 0.185

Both divergences are near-ties. As a check, I made realizar emulate llama.cpp's CPU precision: Q8_K/Q8_0 activation rounding before each matmul, plus f16 K, V and q. That does not remove the near-ties; it moves them. Teacher-forced agreement drops to 63/68, with new rank-2 misses at other steps (gaps 0.016 to 0.24). They are noise-level near-ties, not a defect (the same pattern as #2359).

Consequence under the decided plan. At the 0.67 freeze this moves to 0.68.0, with the gap above recorded, unless one of two things happens by then:

• strict greedy goes green, or
• the operator accepts the margin-aware gate the example implements, which PASSES: llama.cpp's token must be realizar's rank ≤ 2 within 0.25 logits.

Recommendation: accept the margin-aware gate. Exact greedy equality across two backends with different accumulation precision is a coin flip at a near-tie. The teacher-forced table shows realizar tracks llama.cpp everywhere else.

Dissent recorded: the plan fixed the gate in advance precisely so it could not be relaxed after the fact.

[noahgift]

slipped_from: 0.67.0 → 0.68.0 (freeze 2026-09-12T06:00Z, 06x-release-schedule §4.1).

Gate of record was "ships in 0.67 only if greedy 32-token parity with llama.cpp is green". Measured on draft #3114 (branch PMAT-1098-qwen35-cpu, fixture Qwen3.5-0.8B-Q4_K_M.gguf): strict greedy diverges on 2 of 3 prompts at near-ties — llama.cpp's token is realizar's rank 2 with logit gaps 0.145 / 0.185; teacher-forced agreement 66/68; emulating llama.cpp's activation precision moved the near-ties (63/68) rather than removing them. The margin-aware gate in examples/qwen35_parity.rs (rank ≤ 2, gap ≤ 0.25) passes. apr run and apr chat work on the CPU path.

Not shipped in 0.67.0 because the strict gate is red and no one has accepted the margin-aware gate as the contract. If the margin gate is accepted (comment here), #3114 is the PR to un-draft for 0.68.0; the GPU path is #3090.