Offer to help validate NInfer on RTX PRO 4000 and RTX PRO 4000 SSF

[Xtravaganz]

Hi Neroued!

First of all, thank you for all the work you've put into NInfer. It has genuinely doubled my productivity and has become an essential part of my local LLM setup.

I noticed the repository already includes a comprehensive benchmark suite. I happen to have access to both an RTX PRO 4000 Blackwell and an RTX PRO 4000 SSF Blackwell, which I imagine aren't particularly common among contributors.

If it would be useful, I'd be happy to help by:

Running the existing benchmark suite on both GPUs.
Verifying performance changes between commits.
Testing Blackwell-specific optimizations or fixes.
Helping identify regressions before releases.

I'm already using NInfer regularly, so running tests whenever needed wouldn't be a problem.

If there's a preferred workflow, benchmark configuration, or reporting format you'd like me to use, I'm happy to follow it.

Thanks again for creating and maintaining this project!

Hardware Setup

• ASRock X870E Taichi Lite
• Ryzen 9 9900X
• 48 GB RAM
• GPU0: RTX PRO 4000 Blackwell SFF, 70 W
• GPU1: RTX PRO 4000 Blackwell, 145 W
• CPU-Root-Ports, PCIe Gen5 x8/x8

Software Setup

• Ubuntu 26.04
• Kernel 7.0.0-31-generic
• NVIDIA Open Kernel Module
• 595.84
• NInfer Container CUDA 13.1.2

[Xtravaganz]

My longtime test on a SFF card:

Model

https://huggingface.co/Schestex/Qwen3.8-27B-ColdFusion-NVFP4-NInfer

Next week i can tell more about the non-SFF card.

Datasets for the SFF/nonSFF card

https://huggingface.co/datasets/Schestex/ninfer-lab

[Xtravaganz]

Quick update from my side: the RTX PRO 4000 Blackwell is now running flawlessly with the current master and the new v3 artifacts.

Verified upstream state:

Neroued/ninfer
master
9e163eee4b8acec21ab0ac765107b6a3f287b217
perf(ops): route the q4/q5 a16 input projections by column band

Both current v3 variants are working very well for me:

• NoVision v3
• MTP Hybrid-A1 v3

Model/artifacts:
https://huggingface.co/Schestex/Qwen3.8-27B-ColdFusion-NVFP4-NInfer

I also tested the same upstream runtime with the Qwen3.8 Sharp v22.5.0 chat template. My test branch is intentionally only two commits ahead of your master, one adds the template and one includes it in the Docker image; there are no additional runtime or kernel changes:

master...Xtravaganz:ninfer:neroued-v3-sharp

So at this point I no longer need a separate compatibility/runtime fork. Your current master handles the v3 artifacts directly, and the RTX PRO 4000 Blackwell has been rock-solid in my setup.

A huge thank you for all the work you put into NInfer. The amount of performance work, testing, and continuous improvement in this project is seriously impressive. NInfer has become an essential part of my local setup and makes these Blackwell PRO cards incredibly useful for day-to-day local LLM work.

🚀🙏 Really appreciate it thank you! 🙏 🚀

[urec56]

Hi! Could you please provide your ninfer arguments?
I'm trying to run it on rtx 4000 pro, and it always failing because of lack of memory

[Xtravaganz]

Hi! Could you please provide your ninfer arguments? I'm trying to run it on rtx 4000 pro, and it always failing because of lack of memory

Since I'm already further along in development and am just putting the finishing touches on a 16 GB model, I no longer have the builds from back then. But please let me know which version you're currently using, and I'll be happy to help you get a working version.

[urec56]

I tried the latest master commit of ninfer and https://huggingface.co/Schestex/Qwen3.8-27B-ColdFusion-NVFP4-NInfer model
Since it is near 24gb size, it's always overflowing

If you have 16gb model, could you share it?

[Xtravaganz]

I tried the latest master commit of ninfer and https://huggingface.co/Schestex/Qwen3.8-27B-ColdFusion-NVFP4-NInfer model Since it is near 24gb size, it's always overflowing

If you have 16gb model, could you share it?

All right, but I just chose a different development strategy than the rest because I like to see the differences in size, so you'll probably end up with the larger image and not the one with MTP or the “no-vision” one.

Since I'm in the middle of calibrating right now, I think I'll do it tonight, but this is just the first draft. I haven't touched the large layers yet, I stopped at layer 48 with NVFP4 for now to see how high the PPL is, and so that I can then see how the prefill gain compares to the PPL loss in the higher layers.

The new MTP head will follow next, or rather DFlash, since N7 really does allow for faster output there (code+text). But that still needs to be verified as well.

[Xtravaganz]

You don't need to wait for the 16 GB artifact. The current large v3 artifact should fit on the RTX PRO 4000 24 GB if you let NInfer size the KV cache from the remaining VRAM.

Try this first:

./build/apps/ninfer-serve \
  /path/to/qwen3_8_27b_coldfusion_nvfp4_direct-s256-seed38027.ninfer \
  --host 0.0.0.0 \
  --port 8080 \
  --max-context 32768 \
  --kv-capacity auto \
  --max-concurrency 1 \
  --kv-dtype nvfp4

./build/apps/ninfer-serve \
  /path/to/qwen3_8_27b_coldfusion_nvfp4_direct-s256-seed38027.ninfer \
  --host 0.0.0.0 \
  --port 8080 \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 1 \
  --kv-dtype nvfp4

Take a look at nvidia-smi, there you can see the available VRAM.

For the first test, don't enable --vision, MTP or DFlash. --kv-capacity auto calculates the KV capacity from the VRAM remaining after the model/runtime allocations and keeps additional sizing headroom, so it should avoid allocating a KV cache that pushes the card over its VRAM limit.

If that works, you can increase --max-context while keeping --kv-capacity auto.

If you want to maximize context capacity even further, --kv-dtype nvfp4 uses substantially less KV memory than INT8.

[urec56]

Thanks! I'll test it later

The problem with 24gb models is that I need big context window and in llama.cpp with q4 context it's about 8gb for 250k tokens. Adding ~16gb model and 1.1gb dflash2 model give the whole 24gb usage
So 16gb model should be quite similar to mine, but with different runtime which I want to test

[Xtravaganz]

Thanks! I'll test it later

The problem with 24gb models is that I need big context window and in llama.cpp with q4 context it's about 8gb for 250k tokens. Adding ~16gb model and 1.1gb dflash2 model give the whole 24gb usage So 16gb model should be quite similar to mine, but with different runtime which I want to test

You can follow me on HF, than you get a notification about updates!