Skip to content

fp16 decoder tags, the model lifecycle, and numbers from results/ (0.14.0) - #82

Merged
cobanov merged 1 commit into
mainfrom
feat/fp16-decoders
Oct 19, 2026
Merged

cobanov merged 1 commit into
mainfrom
feat/fp16-decoders

Conversation

@cobanov

@cobanov cobanov commented Oct 19, 2026

Copy link
Copy Markdown
Member

fp16 tags for the Qwen3.5 decoders, a written model lifecycle, and every site number from results/.

Runtime and models (ADR 0008)

  • Every Qwen3.5 graph now uses Split instead of SplitToSequence, which ONNX Runtime ran on the CPU. The values are unchanged: all 14 fp32 tags re-ran parity with the same numbers (results/runs/2026-10-19-parity-rtx4090-cuda.json).
  • Six new -fp16 tags: kev:4b-fp16, kev:9b-fp16, nimble:9b-fp16, clef:flash-fp16, pplx-decider:flash-fp16 and strands-decider:2b-fp16. Each has its own gate: the encodings exact and at least 99.5 % of decisions the same as the fp32 reference. All pass, at 99.58–100 %. kev:0.8b-fp16 passed too, but it is not shipped because the fp32 graph is as fast.
  • ollaya run --ask asks your own questions from the command line.
  • The ONNX Runtime arena shrinks after each run.
  • decision's upstream org was renamed to vllm-sr.

Process (ADR 0009, docs/model-lifecycle.md)

  • One runbook covers every stage from intake to live: what each stage needs, its commands, the machine and a time budget.
  • A rolling bench edition: a release measures only what it changed, and bench_plan decides what that is from the manifests and the parity records.
  • New tools: parity_run, bench_plan, bench_run (resumable), bench_edition --update and scripts/bump-version.sh.
  • The site, the library pages, the docs and the README read their numbers from results/ ({{result <tag> <path> <digits>}}, result(), generated README blocks). The build fails on a missing result or a stale README.
  • Ollama 0.40.6's runs are result files. /results has a table with both servers.

Measured (edition 2026.10, updated by 0.14.0)

Measured for 0.14.0 (results/runs/2026-10-19-*): parity on the RTX 4090; typed-decisions, the median five-question
request over HTTP and the GPU memory on the RTX 5090 of Ollaya Bench 2026.10. The fp32 typed-decisions numbers are the
edition's; the fp32 graphs' outputs did not change.

Tag Decisions the same as fp32 (parity, RTX 4090) Typed-decisions, fp16 / fp32 Five questions, RTX 5090: fp16 / fp32 GPU memory, fp16 / fp32
kev:4b-fp16 478 / 480 (99.58 %) 0.669 / 0.669 98 / 171 ms 10.8 / 10.9 GB
kev:9b-fp16 480 / 480 (100.00 %) 0.723 / 0.722 125 / 248 ms 17.6 / 17.9 GB
nimble:9b-fp16 492 / 492 (100.00 %) 0.665 / 0.665 428 / 1015 ms 19.7 / 20.1 GB
clef:flash-fp16 570 / 571 (99.82 %) 0.703 / 0.703 150 / 268 ms 19.4 / 19.6 GB
pplx-decider:flash-fp16 695 / 696 (99.86 %) 0.726 / 0.726 90 / 126 ms 10.3 / 10.4 GB
strands-decider:2b-fp16 649 / 650 (99.85 %) 0.615 / 0.616 72 / 98 ms 5.8 / 8.5 GB
kev:0.8b-fp16 (not shipped) 478 / 480 (99.58 %) 0.461 / 0.460 67 / 62 ms 3.6 / 4.5 GB

Typed-decisions accuracy is within 0.002 of the fp32 tag's for every fp16 tag. kev:0.8b-fp16 passed its gate but is not
shipped: once Split replaced SplitToSequence, the fp32 graph of the 0.8B model was as fast.

Split alone made every fp32 graph faster, with the same parity numbers (median five-question request on the RTX 5090):

Tag 0.13.0 0.14.0
decider:0.8b 189 ms 60 ms
decider:2b 193 ms 91 ms
decider:4b 393 ms 207 ms
decision:eos 209 ms 69 ms
kev:0.8b 142 ms 62 ms
kev:4b 319 ms 171 ms
kev:9b 375 ms 248 ms
clef:flash 406 ms 268 ms
jeeves:9b 641 ms 380 ms
nimble:9b 1688 ms 1015 ms

Checks

  • cargo fmt --check, clippy -D warnings and cargo test --workspace pass on Linux.
  • python3 -m unittest discover -s convert/tests passes.
  • npm run build passes.

The registry manifests are not in this PR: they go in as "Registry: …" commits at release time.

🤖 Generated with Claude Code

… number from results/

- Split instead of SplitToSequence in every Qwen3.5 graph: the same parity numbers, 1.5 to 3 times faster fp32 (ADR 0008).
- Six -fp16 tags (kev:4b, kev:9b, nimble:9b, clef:flash, pplx-decider:flash, strands-decider:2b), each through its
  own gate; kev:0.8b-fp16 passed but is not shipped, its fp32 graph being as fast.
- docs/model-lifecycle.md and ADR 0009: a rolling edition, bench_plan decides what a release measures; parity_run,
  bench_plan, bench_run, bench_edition --update and scripts/bump-version.sh.
- Pages, docs and the README take their numbers from results/; Ollama 0.40.6's runs are result files.
- Edition 2026.10 updated by 0.14.0: strands-decider:2b, pplx-decider:flash, nimble:9b-v3, the fp16 tags and the
  new graphs' latency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cobanov
cobanov merged commit ca10013 into main Oct 19, 2026
7 checks passed
@cobanov
cobanov deleted the feat/fp16-decoders branch October 19, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant