Skip to content

feat(models): add nvidia/Qwen3.8-Flash-Next-FP4-Local checkpoint support - #708

Open
chrisqianz wants to merge 14 commits into
FlashML-org:mainfrom
chrisqianz:qwen4_exp-fp4-local
Open

chrisqianz wants to merge 14 commits into
FlashML-org:mainfrom
chrisqianz:qwen4_exp-fp4-local

Conversation

@chrisqianz

Copy link
Copy Markdown

Adds support for serving nvidia/Qwen3.8-Flash-Next-FP4-Local on top of the existing qwen4_exp (Qwen3.8-Flash-Next NVFP4) framework. The two checkpoints share the architecture; FP4-Local differs only in byte format and quant coverage:

  • Routed experts: per-16 NVFP4 scales stored in the CSF slab encoding (weight_scale.nvfp4_csf_fixed U8 slabs + weight_scale.nvfp4_csf_exceptions U32 records). decode_csf_scales expands them on CPU to plain per-16 E4M3 so the existing expert reader and triton kernel pipeline are unchanged. The expert spec is selected automatically from the weight_map (presence of .nvfp4_csf_fixed).
  • PLE table (layers.1.ple): 128 shards of packed e2m1 U8 rows + per-shard E4M3 scale rows + one global F32 scale. Loaded as packed e2m1 (no BF16 materialization, 25.5 GB pinned instead of ~102 GB) and dequantized by the triton PLE gather kernel (IS_FP4 path). --ple-backend disk falls back to the pinned backend automatically for packed tables (the disk row store streams native rows only).
  • Dense projections: FP4-Local quantizes the text projections (GDN in_proj_qkv/z, self_attn q/k/v/o, index_qk) to MXFP8 per-32 and the vision MLP to W4A16 NVFP4, while the non-Local release leaves them bf16. The reader maps checkpoint scale names to the engine's buffer names per quant scheme (weight_scale -> weight_scale_inv, weight_scale_2 -> weight_global) and expands the scalar W4A16 global to the model's [out] fp16 buffer.
  • Checkpoint layout: FP4-Local keeps its 21 shards in a tensors/ subdirectory referenced by the index; iter_weight_files falls back to the index's weight_map when the top-level glob finds nothing (also fixes any future index-only HF layout).
  • The parallel expert loader's dtype table gains U16/U32/U64 (the CSF exceptions stream is U32).

Committer ran the model end to end with ft serve and it is serving real requests (radix cache hits, ~5k tok/s prefill, ~80 tok/s decode).

Hardware

  • NVIDIA RTX 5090 D, 32 GB
  • Driver / CUDA: torch 2.11.0+cu130, transformers 5.16.1
  • FreeToken: this branch rebased onto main (9b68e00)

Checkpoint

  • nvidia/Qwen3.8-Flash-Next-FP4-Local (21 shards in tensors/, ~96 GB)

Exact command

ft serve --model <path> --ple-backend pinned   --moe-strategy offload --moe-cache-auto   --max-running-requests 1 --kv-reserve-tokens 131072   --moe-prefill-hit-d2d --host 0.0.0.0   --served-model-name Qwen3.8-Flash-Next-FP4-Local --enable-cache-report

Tests

uv run pytest tests/models/qwen4_exp/ tests/models/test_models_loader.py

New coverage: CSF decode round-trip, expert spec auto-selection, PLE packed-table load, e2m1 kernel vs LUT, dense scale renames, W4A16 global expansion, indexed-subdirectory weight discovery, PLE backend fallback.

chrisqian added 14 commits October 16, 2026 18:34
The official FP4-Local checkpoint stores routed-expert block scales as lossless CSF slab pairs (weight_scale.nvfp4_csf_fixed / _exceptions) instead of the plain E4M3 matrix. Add decode_csf_scales and a scale_pair spec mechanism so the expert reader expands each pair into the 9 plain roles, and pick the CSF spec from the checkpoint's tensor names.
KarrAcaRn pushed a commit to KarrAcaRn/FreeToken-ByAI that referenced this pull request Oct 20, 2026
FlashML-org#764 FlashML-org#763 FlashML-org#708 FlashML-org#687 deferred, FlashML-org#688 kept as a feature; FlashML-org#169 kept after its upstream close,
FlashML-org#494/FlashML-org#154/FlashML-org#315 replaced upstream by FlashML-org#690/FlashML-org#698; FlashML-org#278 dropped with upstream's auto->hybrid pick (FlashML-org#734).

Assisted-by: Claude Opus 5.5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant