DFlash2 support is now available for Qwen3.8-27B

[Neroued]

DFlash2 is now supported for both Qwen3.8-27B weight profiles!

To try it, update and rebuild NInfer from the latest master, then redownload .ninfer artifact from Hugging Face:

• NVFP4
• Groupwise INT

Older artifacts still load as before, but they do not contain the DFlash2 companion weights. A fresh download is required to enable DFlash2.

Enable it with:

--spec dflash2 --draft-tokens 7 --lm-head-draft

Measurements were collected with --draft-tokens 7, int8 kv.

Workload	NVFP4	Groupwise INT
AIME 2026 #1	321.1 ± 15.6	224.2 ± 10.8
AIME 2026 #15	183.4 ± 8.0	133.5 ± 4.9
AIME 2026 #30	199.6 ± 10.2	175.2 ± 71.5
Code	265.5 ± 21.7	191.4 ± 12.9
Story	121.3 ± 30.6	84.0 ± 20.9
Translation	255.9 ± 50.6	181.9 ± 34.8
Structured output	356.8 ± 40.8	267.3 ± 30.7

Please give it a try! Feedback, performance reports, and reproducible bug reports are welcome.

[Neroued]

really interesting

[yadiel-tien]

On RTX 5090D (32GB), DFlash2 fails to start with OOM, and the MTP config that previously worked also fails now. However, after changing --kv-dtype from fp8 to nvfp4, both DFlash2 and MTP can run successfully. MTP + nvfp4 achieves ~220 tok/s without --vision. It seems the memory footprint has increased in recent builds, making nvfp4 necessary for both modes on 32GB cards.

[knoopx]

i can fit 100k context @ int8 + mmproj, really nice single-request speed bumps @ 4096 prefill batch, decode is now sitting within 150-250tok/s range, i feel like multi-concurrent agentic performance degraded tho, looks like requests stall for longer despite lowering prefill batch, i need to investigate

[K0IN]

for docker users you can also build the image locally directly from master like this:

docker build -t ninfer:dflash2 https://github.com/Neroued/ninfer.git#master

for podman users that dont default to docker.io as registry:

podman pull docker.io/nvidia/cuda:13.1.2-runtime-ubuntu24.04

podman image tag
docker.io/nvidia/cuda:13.1.2-runtime-ubuntu24.04
nvidia/cuda:13.1.2-runtime-ubuntu24.04

podman build -t ninfer:dflash2
https://github.com/Neroued/ninfer.git#master

[MichaelDementii]

really interesting

I really want details

[Neroued]

I really want details

nvfp4, draft tokens 15, concurrency 8, with a extremely simple prompt.

[knoopx]

btw z-lab 3.8 dflash2 weights seem to work on 3.6 too, acceptance rates are not great but still outperforming mtp3:

3.8 dlash2 on ThinkingCap-Qwen3.6-27B @ 11 draft tokens

[cometkim]

This made the model larger, should there be a separate one for this?

Draft weights make the artifact size larger for sure, but they are only loaded when the server actually uses it. While having separate weights may useful in experiments, it will be more convenient to fuse them into the .ninfer artifact for both users and engine itself.

[potatohog]

I really want details

nvfp4, draft tokens 15, concurrency 8, with a extremely simple prompt.

Do you recommend using draft tokens 7 (in first post) or 15? Or something in between? I ought to try this myself, but I'm not going to be home in the next 12 hours.

As far as my experience goes on this, everything looks solid; I haven't been carefully looking so I don't have hard numbers, but I used to get 200 t/s very rarely and now it isn't nearly as uncommon. This is on pl 450W too.

[Neroued]

Do you recommend using draft tokens 7 (in first post) or 15? Or something in between? I ought to try this myself, but I'm not going to be home in the next 12 hours.

7, 15 is only for fun.

[young-developer]

Dflash2 - Throughput Analysis for Coding Agent

Source: logs — parsed only the req#N done per-request summary lines - 104 completed requests.
Model: qwen3.8-27b/nvfp4 | spec: dflash2 (7 draft tokens) | prefill-chunk 1024 | kv-dtype fp8 | single-request decode (batch 1).

Throughput per request

Metric	Average	Min	Max	Median
Prefill	1,969 tok/s	54.6	8,330	1,620
Decode	197.2 tok/s	150.0	297.1	190.6

Notes

• Prefill spans a wide range (55 → 8,330) because it depends on the prompt cache hit
  rate: ~55–900 tok/s are near-100% cached prompts; 5–8k tok/s are cold (0% cache) prompts.
• Decode is much tighter (150–297), typical for single-request generation. Lower end
  (~150–170) tends to be long-output requests; highest (~285–297) short ones.

[cometkim]

As far as my experience goes on this, everything looks solid; I haven't been carefully looking so I don't have hard numbers, but I used to get 200 t/s very rarely and now it isn't nearly as uncommon. This is on pl 450W too.

Monitor actual power consumption during decoding using nvidia-smi dmon -s p -i 1. DFlash2 consumes more power than MTP, so it may not achieve full performance due to power limits.

[Skylux70]

I have tested the latest v2 NVFP4 model (assuming it's based on Unsloth NVFP4) using IFbench and my own coding benchmarks. The new model completely took a massive dump on my tests, where the output quality has effectively degraded back to Qwen 3.5 levels. Even when disabling DFlash and falling back to MTP, the intelligence drop persists.

Is the new .ninfer v2 artifact using a more aggressive quantization compared to the previous version, rather than simply adding a DFlash layer to it?

[Macasacker]

Can we separate the draft model?

The checkpoints are too large for me to have max context now when using mtp

[cometkim]

Can we separate the draft model?

The checkpoints are too large for me to have max context now when using mtp

#188 (comment)

And I believe the maintainer is experimenting with a new architecture that allows weight mappings to be loaded without hardcoding them. To enable dynamic mapping, all weights including those for the drafter must be included in the ninfer artifact along with the metadata describing them.