feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints

[jason-fxz]

Summary

Serves llama.cpp qwen2 (Qwen2 / Qwen2.5), qwen3 (Qwen3 dense) and qwen3moe (Qwen3 MoE, e.g. Qwen3-30B-A3B, Qwen3-235B-A22B, Qwen3-Coder-30B-A3B) GGUF files on the existing qwen2, qwen3 and qwen3_moe model classes, YaRN long-context variants included.

Changes

• models/{qwen2,qwen3,qwen3_moe}/gguf.py (new): one per family.
  • Builds the config from the GGUF metadata, YaRN included.
  • Maps checkpoint names to GGUF tensors. llama.cpp stores them unchanged (NeoX rope on both sides, plain RMSNorm), so nothing is undone.
  • qwen2 loads the qkv bias; qwen3 and qwen3moe load the q/k norms; qwen3moe loads the router and its experts as GGUF expert banks, the fused ffn_gate_up_exps included.
  • Reads the single attn_qkv matrix (and bias) of files converted with --fuse-qkv as qkv_proj.
  • Refuses prism (Hadamard-rotated) files and NVFP4 GGUF files with per-tensor .scale tensors.
• model.py of the three families: embed_tokens takes the quant config, so a quantized token_embd is served packed.
• gguf/config.py, register.py, kernel/aot_models.py: register the three archs and claim them in the AOT model table. Also claims the two Qwen3.5 GGUF archs from feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints #701, which were missing, so test_every_registry_architecture_is_claimed_by_an_aot_entry failed on main.

Tested models

RTX PRO 6000 Blackwell (sm_120), ft serve --model-path <file>. AIME25 problems 1–8 × 2 samples (temperature 0.6, top_p 0.95, top_k 20). Also 48 staggered requests at concurrency 16.

Model	Quant	bs=1 tok/s	AIME
unsloth/Qwen3-30B-A3B-GGUF	Q8_0, fused / offload	258 / 247	15/16
unsloth/Qwen3-4B-GGUF	Q8_0	247	14/16
unsloth/Qwen3-4B-128K-GGUF	Q8_0, YaRN ×4	–	–
bartowski/Qwen2.5-1.5B-Instruct-GGUF	Q8_0	500	– (6 reasoning / coding prompts correct)

Staggered requests: no corrupted output on any file.

Results

• Weights: after ggml.dequantize, every float tensor is bf16-exact against the HF checkpoint, and Q8_0 tensors reach cos 0.99997, the 30B's experts included. Dry load against the meta-built model matches 339/339 params (30B-A3B), 290/290 (4B) and 198/198 (Qwen2.5-1.5B).
• vs llama.cpp f2cc728: same file, greedy, 6 prompts × 128 tokens. Identical outputs: Qwen2.5-1.5B 2/6, 4B 3/6, 4B-128K 2/6, 30B 1/6. Every divergence is a near-tie in llama.cpp's own probabilities: both engines' picks are its top-2.
• Coverage: header check (dry load, kernel coverage, tokenizer, chat template) of 730 files from 48 HF repos: 695 pass. Every dense size passes, and so does every Q2_K–Q8_0, IQ2–IQ4 and BF16 quant. By arch: qwen2 187/188, qwen3 203/213, qwen3moe 305/320.
• --fuse-qkv: HF Qwen2.5-1.5B, Qwen3-4B and Qwen3-30B-A3B converted to Q8_0 with and without --fuse-qkv load to byte-identical state dicts (198, 290 and 339 tensors), with qkv_proj read from attn_qkv.
• Unit tests: tests/models, tests/moe and tests/kernels/test_ggml.py: 578 pass. The 5 failures are test_quant_config cases that fail identically on main.

Not covered

• IQ1_M weights (no MMQ kernel).
• The ARM types Q4_0_4_4 / 4_8 / 8_8, which llama.cpp has removed (9 old Qwen2.5 files).
• TP > 1.