Start with the project README to build NInfer, download a published artifact, and run the CLI or HTTP server. Windows 11 users should also read the Windows guide.
| Document | Purpose |
|---|---|
| CLI | text, chat-history, image/video input, output streams, sampling, MTP, and common runtime options |
| HTTP serving | OpenAI Responses/Chat Completions, Anthropic Messages, state, streaming, token counting, authentication, and tool calls |
| Performance | RTX 5090 single-request and concurrent-decode results, MTP/DFlash measurements, and reproduction commands |
| Windows | native Windows 11 x64 requirements, vcpkg setup, build commands, and run notes |
| Perplexity | fixed-corpus and custom-text causal perplexity, comparison rules, progress, and reports |
| CLI examples | committed text, multimodal, thinking, long-decode, and long-context inputs |
The executable --help output is the exact source for command-line option spelling and defaults.
| Model | Weights | Download | Versioned model card source |
|---|---|---|---|
| Qwen3.6-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.6-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.8-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.8-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.6-35B-A3B | groupwise-int |
Hugging Face | model card |
The active references under maintainer/ record current architecture, model,
artifact, and maintenance contracts. These files are not additional user workflows or installed
API documentation.
Runtime and Op references:
- Engine architecture, execution ownership, scheduling, and request lifecycles
- Resource scheduling, continuation/checkpoint, and Device/Host context-cache contracts
- Paged KV context storage, ownership, and capacity model
- Operational logging channels, ownership, format, levels, and data policy
- Op admission, contracts, ownership, qualification, and performance rules
- ReplaySSM GDN technical reference
- Linear benchmark contract and registered suites
engine-architecture.md is the sole top-level Engine architecture reference.
resource-scheduling-and-context-cache.md is its narrower authority for resource selection,
materialization, checkpoint ownership, and replica policy. The remaining files define physical
storage, model, artifact, Op, or measurement contracts rather than parallel architecture variants.
Artifact and model references: