Skip to content

README.md

Latest commit

 

History

History
95 lines (71 loc) · 3.53 KB

README.md

File metadata and controls

95 lines (71 loc) · 3.53 KB
Raw

NInfer maintainer tools

tools/ contains the project-owner workflows for artifact conversion and inspection, benchmark orchestration, and serving smoke checks. These tools are not part of the public download-and-run path; normal users should start with the project README.

Run commands from the repository root with a Python 3.11 environment containing the dependencies for the selected tool.

Task index

Task Location
Build the 27B artifact convert/qwen3_6_27b/
Build the Qwen3.8-27B artifact convert/qwen3_8_27b/
Build the 35B-A3B artifact convert/qwen3_6_35b_a3b/
Inspect artifact metadata and objects artifact/inspect.py
Run benchmark matrices bench/
Measure external Serve TTFT bench/ttft/
Exercise a resident HTTP server smoke/serve_contract.py
Exercise thinking preservation through a managed server smoke/serve_thinking_preservation.py

Artifact workflow

The converters consume an official local BF16 checkpoint and write one complete .ninfer artifact. The paths below are placeholders for the maintainer's local checkpoint checkouts:

python3 -m tools.convert.qwen3_6_27b.convert \
  --model /path/to/Qwen3.6-27B \
  --out out/qwen3_6_27b.ninfer

python3 -m tools.convert.qwen3_8_27b.convert \
  --model /path/to/Qwen3.8-27B \
  --out out/qwen3_8_27b.ninfer

python3 -m tools.convert.qwen3_8_27b.convert_nvfp4 \
  --model /path/to/Qwen3.8-27B \
  --quantized-model /path/to/Qwen3.8-27B-NVFP4 \
  --out out/qwen3_8_27b_nvfp4.ninfer \
  --device cuda

python3 -m tools.convert.qwen3_6_35b_a3b.convert \
  --model /path/to/Qwen3.6-35B-A3B-base \
  --dflash-model /path/to/Qwen3.6-35B-A3B-DFlash \
  --out out/qwen3_6_35b_a3b.ninfer

Inspect either result:

python3 -m tools.artifact.inspect out/qwen3_6_27b.ninfer --objects

The exact source revisions, inventories, formats, and conversion recipes are recorded in docs/maintainer/. Published users download the completed artifacts from Hugging Face instead of running these workflows.

Benchmark orchestration

tools/bench/run_ninfer_bench_matrix.py builds and runs the public-Engine benchmark matrix and writes ignored local reports below profiles/bench/:

python3 tools/bench/run_ninfer_bench_matrix.py --preset core --dry-run
python3 tools/bench/run_ninfer_bench_matrix.py --preset core

See tools/bench/README.md and bench/README.md for the orchestrator and executable contracts.

For request-arrival latency, use the managed Qwen3.8-27B NVFP4/FP8 TTFT campaign. Its measurement runner remains an external-only HTTP client; the separate controller owns Serve lifecycle and artifacts. See tools/bench/ttft/README.md.

Serving smoke

After starting ninfer-serve in another terminal:

python3 -m tools.smoke.serve_contract \
  --base-url http://127.0.0.1:18080 \
  --model qwen3.6-27b

The client exercises OpenAI, Anthropic, streaming, usage, multimodal, and tool-call response surfaces against the resident process.

For typed rewrite-checkpoint and thinking-history behavior, the managed smoke script launches a real server and consumes the repository fixture:

python3 tools/smoke/serve_thinking_preservation.py \
  --artifact out/qwen3_6_27b.ninfer --backend mtp