fix(gemma4): load mixed-quant GGUF checkpoints

[Poyo-Yon]

Summary

• read embedding, dense projection, and per-layer expert GGML types from the GGUF tensor table instead of assuming Q6_K/Q4_0
• store heterogeneous routed experts in fixed-width byte slots and pass the real expert stride through every vendored MMVQ quant type
• expose all 19 MMVQ block layouts to the native GGUF dense-layer dispatch
• keep the existing uniform Q4_0 path compatible and reject unsupported mixed-type CPU/hybrid or FTW conversion paths explicitly

Fixes #188.

The fixed-stride expert-bank approach is consistent with the design explored in the closed #102.

Tested on

• FreeToken base: cac247a (v0.1.3)
• GPU: NVIDIA GeForce RTX 5060 Ti 16 GB
• Driver: 616.92
• CPU: Intel Core i7-12700F
• RAM: 64 GB
• OS: WSL2 Ubuntu 24.04, Linux 6.18.33.2
• Python 3.12.3, PyTorch 2.11.0+cu130, CUDA 13.0

Checkpoints:

• unsloth/gemma-4-26B-A4B-it-GGUF
  • gemma-4-26B-A4B-it-UD-IQ4_XS.gguf
  • gemma-4-26B-A4B-it-UD-Q3_K_XL.gguf
• mradermacher/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-v2-i1-GGUF
  • gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-v2.i1-IQ4_XS.gguf

Exact serve command (repeated with each checkpoint path):

ft serve \
  --model <checkpoint.gguf> \
  --host 127.0.0.1 \
  --port 1919 \
  --attention-backend triton \
  --moe-backend offload \
  --moe-cache-rate 0.2 \
  --memory-ratio 0.8 \
  --max-running-requests 1 \
  --max-seq-len-override 4096 \
  --max-prefill-length 512 \
  --num-tokens 4096 \
  --kv-reserve-tokens 4096 \
  --cuda-graph-max-bs 1

All three checkpoints loaded, captured the CUDA graph, completed prefill warm-up, and served an OpenAI-compatible chat completion. The deterministic prompt 17x23 returned 391 on each checkpoint.

Focused tests:

pytest -q tests/models/test_gemma4_gguf_quant.py tests/models/test_gemma4_gguf_rope.py
6 passed, 1 skipped

pytest -q \
  tests/moe/test_offload.py::test_dummy_expert_banks_follow_the_kernel_layout \
  tests/moe/test_offload.py::test_offload_cache_rebuild_resizes_and_preserves_sources \
  tests/moe/test_offload.py::test_set_bank_sources_locked_layer_requires_cpu_layer_ids \
  tests/moe/test_offload.py::test_copy_plan_skips_locked_layers_and_keeps_fused_path
4 passed

The full tests/moe/test_offload.py result was 23 passed and 1 environment-only failure: the isolated POC environment does not have FlashInfer installed, while that existing test explicitly selects the fi attention backend.

[jason-fxz]

Thanks for the detailed work and testing. #690 replaces the GGUF path this PR builds on: kernel/csrc/gguf/, layers/gguf.py, moe/fused_q4_0.py and models/gguf/dequant.py are removed, and models/gemma4/gguf.py is rewritten. Each tensor's ggml type is read from the file, and with offload, expert layers of different types go to separate slot pools. The Gemma-4 26B-A4B checkpoints from google (QAT Q4_0), ggml-org (BF16) and unsloth (UD-Q8_K_XL: Q8_0, plus one BF16 layer) run end to end with fused and offload. We did not run the three checkpoints listed here; the headers of the two unsloth files (UD-IQ4_XS, UD-Q3_K_XL) only list tensor types #690 supports.

#690 is the new GGUF infrastructure. Since it covers this and replaces the old path this PR builds on, we are not planning to merge this PR, and it can be closed.