feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next

[void-mckenzie]

Part of #671 (Model Support: glm5-next).

Follows the per-family GGUF adapter pattern established for qwen4exp in #690 (config from KV metadata through the family's own parse_config, checkpoint-leaf to_gguf map, fused() state-dict emission, GGUF expert banks, text-only TP=1).

What

Serves llama.cpp glm5-next GGUFs — unsloth GLM-5.3-Flash-GGUF — on the GGUF quant kind from #690, reusing the existing glm5_next model classes:

• config: parse_gguf_config builds the text config from the GGUF KV metadata. The new llama.cpp writer carries attention.head_count_kv as a per-block list (0 = KDA, 1 = MLA), which encodes the 34/11 hybrid layout directly; unsloth's earlier glm5next spelling predates it and falls back to a tensor-name census. The trailing MTP block (nextn_predict_layers) is dropped exactly like the HF loader drops the MTP layer. MLA head widths come from the new *_length_mla keys with a q_b/v_b-rows fallback. The served-model facts the KV cannot express (NoPE, interleaved indexer rope) are pinned to the HF release values.
• weights: iter_gguf_weights maps every tensor into the glm5_next state dict and undoes the converter transforms: A_log is stored pre-transformed as -exp(A_log) and inverted at load; the MLA kv_b_proj is split per-head as attn_k_b/attn_v_b (k transposed) and re-fused into the dense [H*(nope+v), lora] layout the module's absorption reads; the q|k|v conv kernels concatenate on the channel axis. The mixed-type fused KDA in_proj (Q8_0 q|k|v + BF16 b|f_a|g_a, as shipped) rides the feat(gguf): run experts whose gate_up and down mix float and quantized types #702 segments; kv_b, the indexer projections and the router stay dense where the file stores floats.
• experts: the routed experts load as GGUF expert banks at their shipped per-layer types — unsloth dynamic quants (IQ3_S gate/up + IQ4_XS down with Q8_0 outlier layers) pass through natively. load_gguf_expert_banks gains the dense-MLP prefix offset (first_k_dense_replace): glm5-next's first MoE layer sits mid-stack, and the banks walked blk.0 before.
• registration: Glm5NextGGUFForCausalLM registered and claimed in the AOT table; also claims the two qwen3.5 GGUF archs feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints #701 left unclaimed (that omission breaks test_every_registry_architecture_is_claimed_by_an_aot_entry on current main). embed_tokens takes quant_config so the Q8_0 token_embd serves packed.

Verified

• Against unsloth GLM-5.3-Flash-IQ4_XS (5 split shards, 159 GB): the full 45-layer engine (RTX 5090 32GB, 40k context, 6 GiB expert slot cache) loads 43 layers of IQ3_S/IQ4_XS experts as host banks and serves coherent, correct output over /v1/chat/completions; the tokenizer, eos set and chat template come from the GGUF via the shared loader.
• Measured, bs=1 on that setup: prefill ≈ 1.6k tok/s steady-state on 8k-token chunks (29k-token prompt in 4 chunks; first-ever chunk pays a one-time Triton autotune), decode ≈ 14–16 tok/s — expert-fetch bound (top-8 of 288 × 43 layers streaming from pinned host banks over PCIe).
• Config parse field-for-field matches the zai-org/GLM-5.3-Flash config.json through glm5_next.config.parse_config; the state dict matches the constructed model 1042/1042 tensors (names + shapes).
• Unit tests build a synthetic miniature glm5-next GGUF (both arch spellings, head_count_kv present and stripped, trailing MTP block) and cover the config parse, the full state-dict mapping and each converter transform.

Text-only, TP=1 — same status as the other GGUF families. Vision (mmproj sidecar) is a natural follow-up.

[void-mckenzie]

Test stats & setup

Workload results (bs=1, single RTX 5090):

Test	Result
Prompt processing, 29,207-token prompt (4× 8,192 chunks)	steady-state ~1.6–1.7k tok/s per chunk (log input throughput: 1599.6 / 1678.1 on the middle chunks); first-ever chunk pays a one-time Triton autotune at the first big prefill shape; whole request 40.6 s incl. tokenizing 103 KB
Decode, coding task (write a SQLite-backed LRU cache module), model-default sampling, 600 tokens	14.3–15.6 tok/s steady-state (log gen throughput, 40-step interval); 44.2 s end-to-end — expert-fetch bound: top-8 of 288 experts × 43 MoE layers streaming from pinned host banks over PCIe into the 6 GiB slot cache
Functional	coherent, factually correct chat + code generation via /v1/chat/completions

Model: unsloth GLM-5.3-Flash-IQ4_XS (5 split shards, ~159 GB), arch glm5-next. Geometry: 45 decoder layers (34 KDA linear + 11 DSA NoPE-MLA with kpool-4 indexer) + 1 MTP block (dropped at load), hidden 4096, 288 experts top-8 + 1 shared, mHC×4, vocab 154,880. Quant mix as shipped: experts gate/up IQ3_S + down IQ4_XS with one all-Q8_0 outlier layer, embedding & lm_head Q8_0, kv_b / indexer projections BF16, norms / routers / mHC F32.

Launch: ft serve --gpu 0 --max-seq-len-override 40000 --moe-cache-mem 6GiB --memory-ratio 0.82 → MoE offload (552 expert slots × 11.12 MiB unified cache), radix cache, bf16, TP=1. One serving note: at the default --memory-ratio 0.9 with a 40k window the KV pool leaves the GPU with ~50 MiB free and the first-use Triton autotune (float-expert grouped GEMM) OOMs the worker — 0.82 leaves ~5.5 GiB headroom and is what the numbers above ran with.

Host: Ryzen 9 7950X3D (32 threads), 187 GiB RAM (~130 GiB pinned expert banks), RTX 5090 32 GB (sm_120), driver 580.95.05, CUDA 13.0, Linux 6.8.0-136-generic, shards on a Crucial T500 2 TB NVMe. Stack: Python 3.10.19, torch 2.14.1, triton 3.8, transformers 5.18.0.

@jason-fxz Upstreaming my private branch that I was using for GLM 5.3 Flash GGUF. Rebased to current main to align this branch with the gguf direction chosen by the Repo. I also have a branch up for deepseek v4 privately, if you'd like me to help you finish these gguf ports, let me know :)