Switch Qwen 3.8 NVFP4 base to `nvidia/Qwen3.8-27B-NVFP4`

[philpax]

The current NVFP4 quant for Qwen 3.8 27B is based on unsloth/Qwen3.8-27B-NVFP4. Nvidia has just released their own quant, which is smaller and is likely to have been better calibrated for NVFP4.

[Gevil]

Pretty sure this QUASAR one is still the better one on size / accuracy ratio if the benches can be trusted on the model card:
https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4

Been using it for quite a while with ninfer and now with also the DFLASH2 grafted on top of it.

[ndizazzo]

I did a pass by converting the Nvidia model and benchmarking with Ninfer off the latest main... here's the initial findings:

Interleaved Benchmark

Engine: NInfer @ b88c0f6f · GPU: RTX 5090 (exclusive) · CUDA: 13.3
Protocol: 2 alternating passes × (3 reps + 1 warmup), single thermal window (29 °C → 42 °C)

Results

Mean of 2 alternating passes ± half-range.

Test	unsloth	nvidia hybrid	Δ
pp2048 prefill (tok/s)	11,106.9 ± 1.2	11,111.4 ± 3.9	+0.04%
pp2048+tg128 decode (tok/s)	425.10 ± 0.02	424.80 ± 0.02	−0.07%
pp512 prefill (tok/s)	9,882.1 ± 6.1	9,958.6 ± 30.2	+0.77%
tg128 decode (tok/s)	127.80	89.61	−29.9%

The tg128 gap is a DFlash2 draft-head artifact (n_prompt=0, so the draft head is unconditioned), not a quantization result.

Reproducibility

• Pass-to-pass half-range ≤ 0.12% on every cell except nvidia pp512 (0.61%).
• DFlash2 acceptance reproduced bit-for-bit across passes (greedy → deterministic):
  • pp2048+tg128, both artifacts: 0.8800 rate / 7.111 length / [51,51,51,51,45,45,36]
  • tg128 unsloth: 0.1588 / 2.098 / [87,63,27,15,6,3,0]
  • tg128 nvidia: 0.0697 / 1.477 / [84,33,6,0,0,0,0]

[Xtravaganz]

Pretty sure this QUASAR one is still the better one on size / accuracy ratio if the benches can be trusted on the model card: https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4

Been using it for quite a while with ninfer and now with also the DFLASH2 grafted on top of i

Pretty sure this QUASAR one is still the better one on size / accuracy ratio if the benches can be trusted on the model card: https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4

Been using it for quite a while with ninfer and now with also the DFLASH2 grafted on top of it.

But you've already tried that with your fork, just not with this one, right? Should I check to see if it works on Linux with your fork?

Right now, I'm just glad to have this info, since I'm looking for a smaller model for my 24 GB that works seamlessly with Ninfer.

[Neroued]

:) I hope I can deliver a better engine architecture this weekend so you will be able to customize your weights.

[cometkim]

:) I hope I can deliver a better engine architecture this weekend so you will be able to customize your weights.

Without hardcoding the ID registration, right? Not sure how can it handle whole mappings, but that will be awesome if it's viable somehow.

^ Please also consider the DFlash2 weight as well. I've refactored it on my side to use NVFP4 instead of W8G32_F16S

[cometkim]

I just launched a new evaluation campaign for updating my nvfp4full profile.

While some of the identified differences are worth checking, others are redundant or offer no real value. Since the nvfp4full profile is already compact enough, I'll focus the evaluation on other quality factors such as precision, speed, and draft acceptance rates, and share the results here.

[cometkim]

Quick check before time-consuming benchmarks:

NVFP4 weight-space rel-L2 comparison

Note: rel-L2 (‖W − Ŵ‖_F / ‖W‖_F vs the official BF16 reference) is a fast weight-space screen: for PTQ candidates lower is better and the ranking is meaningful, but it neither predicts end-to-end scores nor applies to QAT — for a co-adapted QAT checkpoint it measures training drift from the teacher, not quantization error. Final quality calls stay with the end-to-end lanes (GPQA/LBV2, deterministic MTP3/DFlash2 acceptance).

1. MLP codes, 192 tensors (64 layers × gate/up/down)

Source	Count	Min	Median	Mean	Max
NVIDIA ModelOpt PTQ (Local-Hessian/MSE, 2,048 Nemotron samples)	192	0.0833	0.0861	0.0864	0.1008
QUASAR QAT (measured from the nvfp4qat artifact)	192	0.0838	0.0850	0.0854	0.1052
unsloth (NVFP4 0–55 + FP8 row-scaled 56–63; registered-nvfp4 style)	192	0.0266	0.1070	0.0980	0.1535
nvfp4full local encoder (all 64 layers)	192	0.0942	0.0949	0.0949	0.0951
nvfp4full payload (unsloth 0–55 + local 56–63)	192	0.0942	0.1070	0.1065	0.1535

The unsloth row mixes formats by necessity: its export has no NVFP4 for layers 56–63 (FP8 row-scaled there), so the row is completed as the registered nvfp4 profile carries it; the accurate FP8 tail pulls min/mean down and costs 2× the bytes on those eight layers (MLP total ≈11.27 GB vs ≈9.63 GB for every all-NVFP4 row).

Per-band detail:

Band	Source	Min	Median	Mean	Max
layers 0–55	NVIDIA	0.0833	0.0863	0.0865	0.0995
layers 0–55	local	0.0946	0.0949	0.0949	0.0951
layers 0–55	unsloth	0.1040	0.1073	0.1082	0.1535
layers 56–63	NVIDIA	0.0835	0.0839	0.0856	0.1008
layers 56–63	local	0.0942	0.0950	0.0949	0.0951
layers 56–63	unsloth FP8	0.0266	0.0266	0.0266	0.0266

Head-to-head counts:

• NVIDIA beats unsloth on 168/168 NVFP4-comparable tensors (the FP8 tail stays more accurate per tensor, at 2× the bytes).
• NVIDIA beats the local encoder on 190/192 (exceptions: L0 and L63 down_proj, worst +6.8% relative).
• NVIDIA beats the actual nvfp4full payload on 191/192 (sole exception: L63 down_proj 0.1008 vs 0.0942).
• QUASAR QAT beats NVIDIA on 117/192, same band, not separable in weight space.

2. lm_head

Encoding	rel-L2	Size
NVIDIA FP4 head (Hessian-calibrated)	0.0848	0.72 GB
nvfp4full W8G32_F16S head (MAXABS_F16_RECIP_RNE_V1)	0.0055	1.35 GB

The FP4 head is a 15× weight-space error increase traded for −0.59 GiB device weights.

Well-calibrated for FP4 (slightly better than their MLP median), but gated on end-to-end lanes (GPQA/LBV2, deterministic MTP3/DFlash2 acceptance) before adoption.

Since I have already modified and tested the lm_head in the nvfp4qat profile, I'd say it's worth a try. The changes were not really noticeable until the context grew sufficiently large.

3. Conclusion

In weight space, NVIDIA's official PTQ codes are the strongest FP4 encoding of this checkpoint's MLP available to us: they beat unsloth on all 168 NVFP4-comparable tensors.

QUASAR QAT lands in the same weight-space band (117/192 ahead of NVIDIA) but cannot be ranked by this metric and must never be mixed with PTQ codes.

The FP4 lm_head is the only size lever this checkpoint offers (−0.59 GiB), but it buys that with 15× head weight error, so it stays a separate variant gated on the end-to-end lanes rather than on rel-L2.

[cometkim]

Measurements using the ninfer fixed corpus show that while PPL does indeed improve, the MTP/DFlash2 acceptance rate drops significantly.

If you are satisfied with the quality of the current profile, it might be better to stay on it until well-tuned MTP/DFlash2 weights are developed by the community.

[cometkim]

End-to-end decoding perf measurement on my side:

(v2 is the current nvfp4full, v3 is NVIDIA sourced)

	v2 (tok/s)	v3 (tok/s)	Pair gain
MTP-0 (Control)	88.03 / 88.07	88.02 / 88.21	−0.00% / +0.15%
MTP-3	140.44 / 141.17	127.94 / 127.51	−8.90% / −9.67%
DFlash2-K7	126.10	120.97 / 120.75	−4.06% / −4.24%

[Avalonec]

Hi guys,

I've successfully ported and compiled the QUASAR-QAT Full-NVFP4 layout engine (4-bit MLP + 4-bit Attention) under a native Windows/vcpkg MSVC runtime.

To make it work, I integrated a Windows compatibility layer across ~36 files (replacing POSIX mmap with Win32 CreateFileMapping, adding MSVC __int128 polyfills, and fixing TMA descriptor memory alignments).

The monolithic artifact (Full NVFP4 text core + embedded W8 DFlash2 speculative drafter) runs fully stable on an RTX 5090 without any logit scale noise or token-loop degradation.

If you are interested in integrating the Full-NVFP4 layout bindings or need the Windows compatibility patch-set, let me know and I'll provide the source code.

hf

[cometkim]

@Avalonec Yeah, we are all settling on each's own fork; you can check out my version on HF as well.

The issue here isn't so much Windows support. You can't share profiles with others without a patch that hardcodes the artifact mapping, so.. I'm still waiting for the compatibility layer on upstream.

My two cents on Windows environment, try using the DirectStorage API; this allows you to fully utilize NVMe throughput by completely bypassing OS-level overhead. Also NVIDIA supports this and Vulkan via an abstraction layer called RTX IO, but it seems no one has tried this on Linux yet.

[cometkim]

it might be better to stay on it until well-tuned MTP/DFlash2 weights are developed by the community.

By the way, while waiting it, I ran a training campaign to tune the MTP head with various datasets and approaches, but I haven't seen positive results yet. This is a tough one.

[Avalonec]

@cometkim Reviewed the implementation - excellent work! The density gains and the HyperQuant integration are incredibly impressive.
One specific thing that caught my eye is the degradation of the DFlash2 acceptance rate when the draft module weights are quantized to NVFP4 (dropping to 21.4% / 2.5 tokens per round, compared to the 64.3% / 5.5 tokens in the unquantized/requant profile). It makes sense given the extreme sensitivity of block-diffusion drafters to quantization noise, but it's definitely an interesting trade-off to keep in mind for future QAT iterations of the drafter itself.
By the way, have you run any dedicated throughput benchmarks for prefill and decode speeds using the hq-e8-2b KV codec? The memory density is phenomenal (~7x over bf16), but I'm curious if the E8-lattice decoding and Rice coding overhead noticeably impacts the attention kernels (tok/s) compared to standard nvfp4 or rk2v4-e8.
Also, a quick thought looking ahead: leveraging DirectStorage API could be highly effective for the upcoming Qwen4 family. Bypassing the CPU for weight streaming and managing massive KV-cache state swaps could be a massive win for next-gen multi-modal or long-context workloads.

[cometkim]

Since this is a bit off-topic, it would be better if you opened an issue directly in my fork :)