Consider fuller-NVFP4 Qwen3.8-27B weight profile

[cometkim]

I read this in the nvfp4 profile description.

NVFP4 MLP weights in Text layers 0–55 and row-scaled FP8 for the token embedding, attention input/output projections, GDN Q/K/V/Z and output projections, output head, and remaining MLP weights

I guess it is came from Unsloth's checkpoint, but I wondered that is really useful. Because if we use nvfp4 here more aggressively, we could get more space for KV cache.

I built and validated a fuller-NVFP4 weight profile for qwen3.8-27b that extends NVFP4 from the MLP (layers 0–55) to nearly the whole Text backbone, using the exception pattern already registered for qwen3.6-27b-nvfp4

• every Text MLP parent NVFP4 (0–55 copied bit-exact from unsloth, 56–63 quantized locally)
• every GDN query_key_value_z + GDN output on 47/48 layers,
• full-attention query_key_gate_value on the 10 deepest layers, attention/output on 14/16,
• nine BF16 exception parents (layers 3,7,11,15,19,23 input; 3,7 attention output; 4 GDN output),
• W8G32_F16S embedding/head instead of row-scaled FP8.

Result: 17.07 GiB vs the registered profile's 20.02 GiB, with no significant quality loss observed (GPQA 89.39% vs 90.40%, a two-question difference on single-sample runs at n=198)

Why this fits the existing engine

No new kernels are needed.

Every NVFP4 op class at these exact shapes already runs for qwen3.6-27b-nvfp4. The engine-side change is one WeightsProfile value whose binder reuses the qwen3.6 NVFP4 allocation pattern (with this target's fused a_b_projection naming) plus W8 endpoints — the same shape as the existing Qwen38Nvfp4 case.

Evidence

Quality (registered serving profile: thinking, MTP3, INT8 KV, 262,144 ctx; EvalScope 1.9.0, 0-shot rule scoring, temp 0.6, seed 42, single sample):

Benchmark	nvfp4full	registered nvfp4 (published)
GPQA-Diamond	89.39% (177/198)	90.40% (179/198)

Per-tensor fidelity vs the BF16 source (relative Frobenius error, same metric both rows):

	locally quantized parents	unsloth-copied parents
max / typical	0.0951 / ~0.0948	0.126 / ~0.11

Speed (RTX 5090, same engine binary, greedy):

Regime	nvfp4full	registered nvfp4
Prefill 8,032 tok (MTP0)	9,536 tok/s	7,749 tok/s (+23%)
Decode @8k ctx (MTP0)	72.9 tok/s	63.7 tok/s (+14.5%)
Decode @8k ctx (MTP3)	166.4 tok/s	159.9 tok/s (+4%)

Memory at 262,144-token INT8 KV: 4.91 GiB free after startup (registered profile: 2.22 GiB; its published eval ceiling was 252,928 tokens). Full native context now fits with room to spare.

Links

• Artifact (HF): https://huggingface.co/cometkim/Qwen3.8-27B-nvfp4full-NInfer
• Branch: https://github.com/cometkim/ninfer/tree/feat/qwen3.8-nvfp4full

[cometkim]

https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 Looks interesting.

Let me try this one too. If QAT is actually working here, we can cut another 1GB.

[jepjoo]

If QAT is actually working here, we can cut another 1GB.

And retain more intelligence than the current default NVFP4 model which is based on Unsloth's quant.

This is super interesting, please keep us posted if you manage to support this model.

edit: btw QUASAR is the work of Together AI, so seemingly professional work.

[Apoze]

I independently reproduced the fuller-NVFP4 approach on a different Qwen3.8-27B derivative,
starting only from a pinned HuiHui BF16 checkpoint. I did not copy any pre-quantized NVFP4 parent
from Unsloth or another checkpoint.

These numbers are not a direct A/B against the artifact in this issue: the source checkpoint,
calibration corpus, sampling settings and runtime revision differ. I am sharing them as independent
evidence that the profile is viable and reproducible.

Artifact and provenance

• BF16 source: huihui-ai/Huihui-Qwen3.8-27B-abliterated
• pinned revision: d42ca8978c5a66e92c3446d46e8adfe03ef692ff
• source set: 32 files, 55,586,059,478 bytes, checked against the resolved Hugging Face metadata
• artifact: 18,324,059,648 bytes (17.07 GiB)
• artifact SHA-256: b3dba4dc33c73f7d0378c3e9fdb7d4421af38c6ba071c7ce629c811f460cf2d0
• inventory: 1,253 tensors + 6 frontend/vision/MTP resources
• 247 NVFP4 parents derived from the pinned BF16 source, 9 BF16 exception parents and 2 W8 endpoints

Calibration used a deterministic 14,986-token corpus plus a distinct 8,490-token held-out set.
With 2x activation headroom, the held-out pass saturated 0/247 sites (maximum ratio 0.63757).
Without that margin, 60/247 sites saturated, with a maximum of 1.8147x.

An independent verifier linked 1,253/1,253 tensors back to the pinned source, checked all 247 NVFP4
parents word-for-word, covered 111,093,984 row-split groups, and measured a maximum relative error of
0.0950963511.

Quality

EvalScope 1.10.0, GPQA-Diamond, official 198-question split, seed 42, 0-shot
SINGLE_ANSWER_COT, official acc scorer, temperature 1.0, top-p 0.95, top-k 20, min-p 0,
thinking enabled:

• 175/198 = 88.38%
• parsing: 198/198
• runtime: 5 h 58 min

I did not complete a full run of the previous HuiHui NVFP4 artifact, so this result must not be used
to claim a paired quality improvement or equivalence over that artifact.

RTX 5090 / Windows runtime observations

Environment: Windows 11, RTX 5090, MSVC 19.39, CUDA 13.3, sm_120a. The retained profile was
INT8 KV, MTP4, prefill chunk 4096, concurrency 1.

• deterministic five-workload A-B-A: MTP4 median decode 222.45 and 223.38 tok/s
• MTP5 median decode 193.48 tok/s; a repeat measured 192.43 tok/s
• MTP5 lost all five paired workloads, so MTP4 was retained
• 109,435-token prefill: 5,436.64 / 5,434.38 tok/s on the two INT8 legs; all three needles recalled
• 15/15 cold starts passed across 64k, 128k, 150k, 200k and 262,144-token contexts
• full 262,144-token INT8 context started three times and left 1,815 MiB free at readiness

The 262k memory figure includes the current master resource policy (context cache, device/host state
and CUDA Graph allowance), so it should not be compared directly with a weights-only memory number.

The artifact also passed the OpenAI Chat Completions, Responses and Anthropic routes, streaming,
reasoning on/off, tools, vision, multi-image, multimodal tool results, cache reuse and multi-turn
smoke coverage. The Windows C++ suite passed 94/94 relevant tests; three artifact-dependent 35B
tests were skipped.

The performance runs used a local master-based Windows build with the fuller profile port and a
temporary port of PR #55 for cache-usage reporting. That port changes reporting, not the model
weights or decode kernels. No experimental TMA or compressed-KV patch was included.

If useful, I can clean up and share the conversion/calibration manifests and the independent
verification summary. I would wait for the maintainer's preferred scope before preparing any code
contribution, since the implementation overlaps the existing branch linked in this issue.

[cometkim]

I'm currently running evaluations using QUASAR's weights.

Memory savings are not significant as expected. Device weights differ by only 0.3 GiB, since I've already applied nvfp4 to most layers of the existing profile, and exceptions are very small.

Both are getting equivalent scores in GPQA-Diamond and AIME26. New evals with QUASAR's:

Benchmark	Value
GPQA-Diamond	89.22 ± 2.49 %
AIME26	91.11 ± 3.85 %

Since these are no longer sufficient for differentiation, I'm now running additional evaluations (LongBenchV2, Terminal Bench 2.1, SWE Bench Pro). It will likely take a few more days or a week. In benchmarks involving long contexts (~= 524k), the differences may become more apparent as KV compression and RoPE scaling are additionally applied.

[s1lverkin]

@cometkim have youu finished your benchmarks at long context?

[cometkim]

@s1lverkin The LBv2 was finished for the nvfp4+qat profile. (3 rounds per each)

Cell	Seeds	Mean ± std
GPQA-Diamond (INT8)	90.91 / 86.36 / 90.40	89.22 ± 2.49
AIME26 (INT8)	93.33 / 93.33 / 86.67	91.11 ± 3.85
LBv2 short (INT8, native)	66.11 / 65.56 / 67.22	66.30 ± 0.86
LBv2 medium (hq, 524k YaRN-2)	53.95 / 58.60 / 58.14	56.90 ± 2.62
LBv2 long (hq, 786k YaRN-3)	39.81 / 38.89 / 39.81	39.50 ± 0.53

The hq, and YaRN mean here; I applied HyperQuant 2bps KV and RoPE scaling to run on a single 5090. Based on my previous observations, the scaling factor has a bigger impact on accuracy than the KV compression.

For reference, the human baseline for LBv2 is around 53%, and the quality to be acceptable up to 786K (3x scaling) to me.

To verify the impact of QUASAR's weights, benchmarks are also underway on the existing "nvfp4full" profile, and I will have to wait a few more days for the results.

Once all scores are confirmed, I plan to run the coding benchmarks one round at a time. If it does not significantly compromise coding capabilities, I intend to upload it to HF as a profile named something like nvfp4qat.

[cometkim]

Progress Update: nvfp4full re-measure and the HyperQuant comparisons for LongBench are complete, and re-confirmed there is no difference in quality compared to int8 KV. It is just a difference in speed.

nvfp4full profile results re-measured:

Benchmark	KV cache	Context / RoPE	Score
GPQA-Diamond (3-round mean)	INT8 G64	262,144	87.88 ± 2.62 %
GPQA-Diamond (hq A/B)	hq-e8-2b	262,144	89.90 %
AIME26 (3-round mean)	INT8 G64	262,144	93.33 ± 3.34 %
AIME26 (hq A/B)	hq-e8-2b	262,144	96.67 %
LBv2 short (3-round mean)	INT8 G64	262,144	67.41 ± 1.94 %
LBv2 short (hq A/B)	hq-e8-2b	262,144	66.67 %
LBv2 medium (3-round mean)	hq-e8-2b	524,288 · YaRN 2	59.07 ± 2.27 %
LBv2 long (3-round mean)	hq-e8-2b	786,432 · YaRN 3	38.89 ± 2.33 %

I'm still running TB2.1 benchmarks for both profiles. I use my own DFlash2 implementation there. However, since DFlash2 support has now been added upstream, I need to sync my implementation first for artifact compatibility.

It will likely be next week once all the work is finished, but it might be faster to release additional improvements upstream.

Please post further discussions regarding the artifact on the HuggingFace discussion.

[cometkim]

One important observation is that when using QAT weights, the DFlash2 token acceptance rate drops significantly compared to the existing nvfp4full for the same prompt.

Not always, but it is frequently observed. This is a major issue for inference speed.

[cometkim]

QUASAR QAT variant profile is ready.

https://huggingface.co/cometkim/Qwen3.8-27B-nvfp4qat-NInfer

It allows replacing all 9 existing exception BF16 tensors with NVFP4 using a quantization-aware training approach.

While this results in a smaller and faster model without a loss in precision, the difference is not as significant as expected: 16.02 GiB vs. 15.31 GiB. Since speeds may decrease due to changes in acceptance rates when using MTP or DFlash2 speculative decoding, its speed needs to be re-evaluated for each specific environment and actual workloads.