NInfer

Selected checkpoints. Maximum single-GPU inference performance.

NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is deliberately specialized: one GPU, one resident model, and a startup-fixed capacity of one to eight active requests.

NInfer supports five artifact identities. The quick-start commands use Qwen3.8-27B NVFP4.

Model	Weights	Artifact	Download and model card
Qwen3.6-27B	groupwise-int	qwen3_6_27b.ninfer	Qwen3.6-27B
Qwen3.6-27B	nvfp4	qwen3_6_27b_nvfp4.ninfer	Qwen3.6-27B NVFP4
Qwen3.8-27B	groupwise-int	qwen3_8_27b.ninfer	Qwen3.8-27B
Qwen3.8-27B	nvfp4	qwen3_8_27b_nvfp4.ninfer	Qwen3.8-27B NVFP4
Qwen3.6-35B-A3B	groupwise-int	qwen3_6_35b_a3b.ninfer	Qwen3.6-35B-A3B

The artifact identity fixes the exact model and weight profile. Every artifact also embeds the tokenizer, chat template, and media frontend resources required by its registered target.

Quick start

NInfer requires 64-bit Linux, an NVIDIA GeForce RTX 5090, CUDA Toolkit 13.1 or newer, CMake 3.28 or newer, a C++20 host compiler, Ninja, pkg-config, FFmpeg development libraries (libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7), and libcurl >= 7.85. The build rejects CUDA architectures other than sm_120a.

Build the two product binaries:

git clone https://github.com/Neroued/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Tests, benchmarks, and maintainer tools are excluded from the default build. There is no install target or packaged binary distribution; run NInfer from its source build tree.

Download the artifact used by this example with the Hugging Face CLI:

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

Start a long-running text/agent server with two active-request lanes and explicit Device/Host checkpoint capacity:

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Each request has a 240,000-token logical ceiling. A shared 240,000-token Device KV pool serves admitted requests; two requests run concurrently when their combined reservations fit. The cache tiers provide two Device checkpoint slots, eight pinned Host State slots, and 8 GiB of pinned Host KV beyond the two active StateImages.

Send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

Run a one-shot CLI request with a 32,768-token allocation:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode, then give a concise conclusion." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Answer content is written to stdout. Loading progress, reasoning, timings, throughput, memory, and speculative-decoding statistics are written to stderr. Use --messages FILE and --vision for structured image/video input; see the CLI guide and committed examples.

Resource-aware long-context reuse

A reusable prefix checkpoint contains KV and the complete continuation state for its exact prompt frontier. A Device-resident checkpoint resumes directly. Under pressure, the planner weighs Device retention, pinned Host State/KV, and eviction by immediate restore work and later reuse cost. Active requests retain their completion reservations.

See Resource scheduling and context cache for the algorithm and Serve TTFT benchmark for public-HTTP coverage of hot reuse, Host resume, eviction, shared prefixes, scheduling boundaries, and multimodal load.

Performance

Published measurements use an RTX 5090. Performance records the exact benchmark profiles and methodology.

Concurrent MTP3 decode

Saturated decode used INT8 group-64 KV, CUDA Graphs, MTP3, and one 8,192-token generation per active request. Values are aggregate committed decode throughput and MTP acceptance from complete intervals whose actual decode batch equaled the configured concurrency.

Model profile	C=1 tok/s / accept	C=2 tok/s / accept	C=4 tok/s / accept	C=8 tok/s / accept	C8 / C1
Qwen3.6-27B groupwise-int	185.8 / 68.2%	247.0 / 69.0%	309.5 / 68.4%	535.0 / 68.3%	2.88×
Qwen3.6-27B nvfp4	202.4 / 69.3%	399.7 / 71.4%	699.7 / 69.3%	1,146.9 / 68.6%	5.67×
Qwen3.6-35B-A3B groupwise-int	593.0 / 67.2%	877.7 / 68.2%	1,166.0 / 69.8%	1,313.8 / 67.3%	2.22×
Qwen3.8-27B nvfp4	143.8 / 48.9%	267.6 / 48.1%	461.1 / 45.8%	766.6 / 46.0%	5.33×

Single-request serving

The serial serving corpus used INT8 group-64 KV, CUDA Graphs, a 1,024-token prefill chunk, and five fixed seeds after warm-up. The table keeps one short-prefill, one extreme-prefill, and one structured-output MTP3 point for each published profile; the full context and scenario matrices are in the performance document.

Model profile	7,680-token prefill	260,096-token prefill	Structured MTP3 decode
Qwen3.6-35B-A3B groupwise-int	15,544.3 tok/s	5,157.1 tok/s	770.9 tok/s
Qwen3.6-27B groupwise-int	3,218.1 tok/s	1,614.8 tok/s	193.0 tok/s
Qwen3.6-27B nvfp4	11,191.5 tok/s	2,510.6 tok/s	252.2 tok/s
Qwen3.8-27B nvfp4	8,340.4 tok/s	2,203.1 tok/s	219.8 tok/s

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

Model profile	AIME 2025	AIME 2026	GPQA-Diamond	ERQA	RealWorldQA
Qwen3.6-27B groupwise-int	86.67%	93.33%	86.87%	—	—
Qwen3.6-27B NVFP4	93.33%	93.33%	84.34%	—	—
Qwen3.6-35B-A3B groupwise-int	90.00%	90.00%	85.35%	—	—
Qwen3.8-27B groupwise-int	96.67%	96.67%	87.37%	66.25%	82.22%
Qwen3.8-27B NVFP4	96.67%	96.67%	90.40%	66.25%	83.53%

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8 rows used temperature 1.0 and presence penalty 0.0. Multimodal evaluation used --vision and an 81,920-token context limit. Text evaluation used 262,144 tokens except Qwen3.8-27B NVFP4, which used 252,928 tokens to fit the RTX 5090 after weights. Each score is one sample per problem; model cards contain the correct/total counts and evaluation notes.

Artifact and startup notes

Current builds accept only version-2 .ninfer containers. All five published downloads are version 2. Migration is needed only for Qwen3.6 artifacts downloaded before their version-2 publication:

python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer

Use the same command with the exact older Qwen3.6 NVFP4 or 35B-A3B file. Migration updates container metadata without rewriting the weight payload.

GPU residency is fixed at process startup. --spec selects speculative decoding residency, and --vision selects Vision residency. DFlash is available for text-only Qwen3.6-35B-A3B execution.

Docker

Build the runtime image on a host with the NVIDIA Container Toolkit:

docker build --tag ninfer:local .

Mount the downloaded model and run the same example server profile:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer \
  --host 0.0.0.0 \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Capabilities and limits

All registered model IDs support:

• text generation with thinking and non-thinking prompt modes;
• image, multi-image, video, and mixed multimodal messages;
• chunked prefill, exact-batch CUDA Graph decode, and startup-bounded batched decode;
• MTP speculative decoding with draft windows from one to five;
• BF16, INT8 group-64, and row-scaled FP8 E4M3 KV storage;
• private and shared exact-prefix reuse with Device/Host State and KV retention;
• model-aware sampling defaults and explicit sampler overrides;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming, tools, local response state, token counting, and usage accounting.

The 35B-A3B target additionally supports text-only DFlash with draft windows from one to fifteen.

The product boundary remains intentionally small:

• one RTX 5090 and one resident model per Engine;
• a startup-fixed capacity of one to eight active requests with bounded FIFO ingress;
• no request preemption, priority/QoS, active-request swapping, weight offload, multi-GPU, or distributed serving;
• one shared startup-fixed KV pool across active requests and retained prefixes;
• no runtime model discovery or unregistered checkpoint fallback;
• parsed tool calls are returned to the client; NInfer does not execute tools;
• the in-tree C++ headers are not distributed as an installed SDK.

--max-context is each sequence's logical limit. --kv-capacity sizes the shared Main Text KV pool used by active requests and retained prefixes; auto resolves the largest legal capacity at startup from the memory remaining after weights while keeping 1 GiB of sizing headroom. Explicit capacities remain fixed for the process lifetime.

Documentation

• Documentation index
• CLI
• HTTP serving
• Performance
• Resource scheduling and context cache
• Serve TTFT benchmark
• CLI examples
• Contributing

Run ./build/apps/ninfer --help or ./build/apps/ninfer-serve --help for the exact current option contract.

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.