Repository navigation
feat(models): add nvidia/Qwen3.8-Flash-Next-FP4-Local checkpoint support - #708
Open
chrisqianz wants to merge 14 commits into
Open
chrisqianz wants to merge 14 commits into
chrisqianz wants to merge 14 commits into
Conversation
added 14 commits
October 16, 2026 18:34
The official FP4-Local checkpoint stores routed-expert block scales as lossless CSF slab pairs (weight_scale.nvfp4_csf_fixed / _exceptions) instead of the plain E4M3 matrix. Add decode_csf_scales and a scale_pair spec mechanism so the expert reader expands each pair into the 9 plain roles, and pick the CSF spec from the checkpoint's tensor names.
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 20, 2026
FlashML-org#764 FlashML-org#763 FlashML-org#708 FlashML-org#687 deferred, FlashML-org#688 kept as a feature; FlashML-org#169 kept after its upstream close, FlashML-org#494/FlashML-org#154/FlashML-org#315 replaced upstream by FlashML-org#690/FlashML-org#698; FlashML-org#278 dropped with upstream's auto->hybrid pick (FlashML-org#734). Assisted-by: Claude Opus 5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for serving
nvidia/Qwen3.8-Flash-Next-FP4-Localon top of the existingqwen4_exp(Qwen3.8-Flash-Next NVFP4) framework. The two checkpoints share the architecture; FP4-Local differs only in byte format and quant coverage:weight_scale.nvfp4_csf_fixedU8 slabs +weight_scale.nvfp4_csf_exceptionsU32 records).decode_csf_scalesexpands them on CPU to plain per-16 E4M3 so the existing expert reader and triton kernel pipeline are unchanged. The expert spec is selected automatically from theweight_map(presence of.nvfp4_csf_fixed).layers.1.ple): 128 shards of packed e2m1 U8 rows + per-shard E4M3 scale rows + one global F32 scale. Loaded as packed e2m1 (no BF16 materialization, 25.5 GB pinned instead of ~102 GB) and dequantized by the triton PLE gather kernel (IS_FP4path).--ple-backend diskfalls back to the pinned backend automatically for packed tables (the disk row store streams native rows only).in_proj_qkv/z,self_attnq/k/v/o,index_qk) to MXFP8 per-32 and the vision MLP to W4A16 NVFP4, while the non-Local release leaves them bf16. The reader maps checkpoint scale names to the engine's buffer names per quant scheme (weight_scale->weight_scale_inv,weight_scale_2->weight_global) and expands the scalar W4A16 global to the model's[out]fp16 buffer.tensors/subdirectory referenced by the index;iter_weight_filesfalls back to the index'sweight_mapwhen the top-level glob finds nothing (also fixes any future index-only HF layout).Committer ran the model end to end with
ft serveand it is serving real requests (radix cache hits, ~5k tok/s prefill, ~80 tok/s decode).Hardware
main(9b68e00)Checkpoint
nvidia/Qwen3.8-Flash-Next-FP4-Local(21 shards intensors/, ~96 GB)Exact command
Tests
New coverage: CSF decode round-trip, expert spec auto-selection, PLE packed-table load, e2m1 kernel vs LUT, dense scale renames, W4A16 global expansion, indexed-subdirectory weight discovery, PLE backend fallback.