NInfer now supports custom weight recipes and chat templates

[Neroued]

NInfer now supports custom weight recipes. You can convert your own post-trained or fine-tuned checkpoints for supported model architectures and configurations, and choose weight formats across layers and projections.

Start from a built-in recipe or write your own. Recipes can stay local and do not need to be submitted upstream.

See the [weight conversion guide](https://github.com/Neroued/ninfer/blob/master/docs/weight-conversion.md) for commands and examples.

What is supported?

Custom checkpoints and format combinations no longer require checkpoint-specific engine registration. However, a particular source may still need conversion code, and the resulting artifact needs Op support for its formats, layouts, shapes and combinations.

Op support and performance tuning

Community contributions are welcome for missing Op support and performance tuning of existing implementations.

Keep each PR focused on one concrete change—for example, supporting one format/shape combination or tuning one existing shape—with the relevant verification and report.

• [Contribution guide and PR requirements](https://github.com/Neroued/ninfer/blob/master/CONTRIBUTING.md)
• [Op development guide](https://github.com/Neroued/ninfer/blob/master/docs/maintainer/op-development.md)
• [Linear tuning and report format](https://github.com/Neroued/ninfer/blob/master/docs/maintainer/linear-tuning.md)
• [Q4 tuning report example](https://github.com/Neroued/ninfer/blob/master/docs/maintainer/examples/q4-linear.md)

Updating

The changes are available on master, and the official Hugging Face artifacts have been updated to the v3 container format. Update and rebuild NInfer to use them.

Existing official v2 artifacts can be [upgraded locally](https://github.com/Neroued/ninfer/blob/master/docs/weight-conversion.md#upgrade-an-existing-v2-artifact) without downloading the weights again.

Related requests: #231, #192, #81, #214 and #70. These cover custom post-trained weights, alternative quantization sources and mixed-format choices; individual conversion and Op requirements still apply.

[Neroued]

Custom Jinja chat templates are now supported on master. Update and rebuild NInfer to use them.

You can embed a custom template when creating an artifact:

python3 -m tools.convert \
  --model /path/to/Qwen3.8-27B \
  --recipe qwen3_8_27b \
  --resource chat_template.jinja=/path/to/custom.jinja \
  --out models/custom.ninfer

You can also override the embedded template at startup without rebuilding the artifact:

./build/apps/ninfer-serve models/qwen3_8_27b.ninfer \
  --chat-template /path/to/custom.jinja

The CLI supports the same --chat-template option.

The official Hugging Face artifacts have also been refreshed with our maintained default templates, including support for system/developer messages at their original positions. The container version remains v3, but the embedded templates have changed.

If you previously upgraded or downloaded an earlier artifact:

• Still have the original official v2 file: update your checkout and rerun the [updated upgrade tool](https://github.com/Neroued/ninfer/blob/master/docs/weight-conversion.md#upgrade-an-existing-v2-artifact), using that original v2 file as input and a new output path.
• Only have an earlier v3 file: download the latest artifact from the same Hugging Face repository. The upgrade script accepts v2 input.
• Prefer to keep your existing v3 file: use --chat-template tools/chat_templates/qwen3_8.jinja, or qwen3_6.jinja for Qwen3.6 Dense/MoE. This avoids downloading the weights again.

See the [conversion guide](https://github.com/Neroued/ninfer/blob/master/docs/weight-conversion.md) and [CLI guide](https://github.com/Neroued/ninfer/blob/master/docs/cli.md#text-input).

Related requests and PRs: [#182](#182), [#183](#183), [#41](#41), [#42](#42), [#43](#43), and [#78](#78).

[cometkim]

Hey @Neroued,

I tested my nvfp4full profile conversion and it's working fine for plane usage with MTP. However, since the profile use calibrated NVFP4 DFlash2, it requires a coherent set of changes, not just registering more matrix shapes:

• DFlash2-specific NVFP4 linear shapes and three-output QKV projection.
• Aligned NVFP4 subviews for context K/V weights.
• NVFP4 context materialization and grouped-convolution routes.
• A16-only SwiGLU support beyond 16 tokens.
• Quantized selector-codebook binding and execution.

I extracted it from my fork engine, but as you see this could not be done with minor changes.

master...cometkim:ninfer:feat/nvfp4-dflash2

[DuncanBetts]

I've just been doing this, and thought I'd put this here for anyone - worked for me on 3.6 35B A3B, 3.6 27B, 3.8 27B (I haven't tried it on any models aside Neroued's releases).

To upgrade supported v2 .ninfer artifacts without installing Python, Torch etc on the host, build a small Podman image from a current NInfer checkout:

FROM python:3.11-slim
WORKDIR /opt/ninfer
COPY tools/upgrade_ninfer_v2_to_v3.py tools/upgrade_ninfer_v2_to_v3.py
COPY tools/chat_templates/qwen3_6.jinja tools/chat_templates/qwen3_6.jinja
COPY tools/chat_templates/qwen3_8.jinja tools/chat_templates/qwen3_8.jinja
ENTRYPOINT ["python3", "tools/upgrade_ninfer_v2_to_v3.py"]

podman build -f Containerfile -t localhost/ninfer-v3-upgrade:local .

MODEL_DIR=/path/to/your/ninfer/models
podman run --rm --userns=keep-id \
  -v "$MODEL_DIR:/models:rw" \
  localhost/ninfer-v3-upgrade:local \
  /models/input-v2.ninfer /models/output-v3.ninfer

The output path must differ from the input. The converter preserves stored weight values/formats and installs the maintained matching chat template. Ensure space for both files; when space is limited, convert and validate one model at a time, then replace the original:

mv "$MODEL_DIR/output-v3.ninfer" "$MODEL_DIR/input-v2.ninfer"

--userns=keep-id keeps outputs owned by the invoking user.