Post your benchmark here 📊 — datapoints, A/Bs, cold→warm curves

[JustVugg]

This is the home for benchmark datapoints (they used to live in Issues — new ones belong here).

Template (same rigor as the README table):

Hardware:  CPU / RAM / disk (+ iobench number) / GPU if any
OS:        (native? WSL2? filesystem)
Commit:    git rev-parse --short HEAD
Model:     container + MTP head variant (int8 sizes verified?)
Command:   exact env vars + coli invocation
Warm-up:   cold? how many runs? median?
Result:    tok/s · expert hit % · RSS · profile split if you have it

What makes a datapoint great: an A/B (one knob changed), a cold→warm curve, or a config nobody has tried (see the README table for covered ground — first-of-a-kind hardware always gets a row). Negative results are welcome: "X did not help on my box" with numbers has saved this project from wrong turns at least four times.

Current table highlights to beat: 6.84 tok/s (6×5090 full residency) · 2.24 (M5 Max Metal tuned) · 1.23 (Gen5 NVMe + 5090) · 1.10 (Strix Halo) · 0.5 (native Windows warm).

[ebootheee]

Hardware:  Ryzen 9 9950X3D2 (16C/32T) / 121 GB DDR5 / Samsung 9100 PRO Gen5 NVMe -- iobench 11.48 GB/s O_DIRECT / 
  2x RTX 5090 (Gen4 x8 + x4)
  OS:        Ubuntu native, kernel 7.0.0-27, ext4 (no WSL2)
  Commit:    a032299 = 6d3ed7e + the three patches now in PR #270
  Model:     GLM-5.2 int4 (mateogrgic container), int8 MTP heads byte-verified
  Command:   MTP=0 DIRECT=1 PIPE=1 COLI_PIPE_BLOCK=1 PIPE_WORKERS=16 PREFETCH=1 \
             CUDA_DENSE=1 COLI_CUDA_ATTN=1 CUDA_RELEASE_HOST=1 COLI_NO_OMP_TUNE=1 \
             PIN=<model>/.coli_usage PIN_GB=55 COLI_CUDA_BATCH=2 \
             ./coli run --auto-tier --ram 92 --gpu 0,1 --vram 52 --repin 16 --ngen 1024
  Warm-up:   .coli_usage at ~3.5M selections; back-to-back runs, both reported
  Result:    2.2554 / 2.2505 tok/s @ 1024 tok · hit 79.0-79.8% · RSS ~82 GB
             profile: disk 149s service (0 wait) / matmul 117s / attention 69s

A/Bs from the same box, one knob each:

• PIPE=1 (workers were set, pipe was off -- see the implication fix in pipe: blocking pipe_wait (COLI_PIPE_BLOCK=1) + PIPE_WORKERS implies PIPE=1 #270): 1.98 -> 2.16 @ 192
• COLI_CUDA_BATCH=2: closed 2.17 -> 2.2554 @ 1024
• COLI_NO_OMP_TUNE=1: 0.41 -> 1.29 on stock main ([Bug]: ~3x throughput regression on x86+CUDA since 92011de — OMP hot-spin tuning vs new threads (I4_ACC512 exonerated; COLI_NO_OMP_TUNE=1 recovers and then some) #159) -- load-bearing
• negative: --ram 96 regressed (memory pressure), OMP team-size tuning never beat spin-off,
  and a 2-drive shard split LOST 5% (the async pipe already hides disk)

Research row (routing approximation, NOT for the main table): ROUTE_J=2 ROUTE_M=1 cache-aware routing -> 3.09 tok/s @ 92% hit, 15.3% expert substitution, MMLU-200 A/ indistinguishable from true routing (details in #161).

Trail: #120 -> #160 -> #215 -> PR #270. Won't catch Zac's 6.84 with two cards and a disk, but per-GPU it's a fight. :)

[VincentMarquez]

DGX Spark / GB10 — stock top-8 vs CACHE_ROUTE (+ O_DIRECT disk)

Two-tier discipline: stock full top-8 vs experimental CACHE_ROUTE. Metric is decode tok/s (generation window), not overall wall tok/s.

Hardware:  NVIDIA DGX Spark · GB10 (sm_121) / 20× ARM (X925+A725) / 121 GB unified
           local NVMe · iobench 4.25 GB/s buffered · 9.69 GB/s O_DIRECT (2.3×)
OS:        Ubuntu 24.04 · Linux 6.17 · aarch64 · native (not WSL)
Commit:    CACHE_ROUTE via PR #199 (62419af); engine stack as measured in notes
Model:     GLM-5.2 int4 (colibrì snap) · full K=8 · MTP off for speed cells
Command:   DIRECT=1 + CUDA unified/dense path · large pin/LRU
           Stock:  CACHE_ROUTE=0
           Exp:    CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12  (optional ROUTE_AGREE=1)
Warm-up:   session warm LRU / multi-turn chat with :reset; timed PROFILE WARM=160 NGEN=48
Result (decode tok/s · full K=8):

  Tier A stock routing:     2.08–2.39 decode · hit ~82%
  Tier B CACHE_ROUTE early: 3.33 decode · hit 97% · swap 14%   (chat ladder peak)
                            2.67 decode · hit 93.5% · swap 15% (timed one-shot)
  Tier C later CUDA stack + CR (not CR alone): 5.14–6.17 decode band
                            fair CR on/off same stack: 2.98 → 5.14

Disk lever: DIRECT=1 / O_DIRECT 9.69 GB/s vs 4.25 buffered — flag louder on real NVMe;
            weaker lever on slow/VHDX-backed disks (measure both).

Why two tiers

Tier	What it isolates
A	Leaderboard-comparable stock full-k8 routing
B	CACHE_ROUTE alone on early stack (~2.4 → 3.33)
C	CR plus MLA/fuse/device-tier — do not call this “CR alone”

Quality A/B (limited — not full coli bench)

Protocol: SCORE log-likelihood · HellaSwag · n=20 · seed 1234 · same request file · full K=8.

Cell	acc_norm	acc	notes
CACHE_ROUTE=0	80% (16/20)	40%	stock
CACHE_ROUTE=1 J=2 M=12	70% (14/20)	35%	−10 pp
Same normalized pick	85%		2 flips stock✓→CR✗

Caveat: n=20 only. Primes multi-turn still produced correct first-20 list (eyeball). Prefer larger greedy-token agreement set for ROUTE_J/M defaults. Supports default off.

Vs EXPERT_BUDGET (blind drop): CR is the principled middle path — substitute toward resident experts inside top-M (arXiv:2412.00099), not drop by gate weight alone. Connects to expert-specialization (#207): which experts are safe to swap.

Links

• Notes / numbers: https://github.com/VincentMarquez/glm52-gb10-colibri
• CACHE_ROUTE PR: routing: opt-in CACHE_ROUTE max-rank (pin∪LRU prefer) #199
• Mechanism comment: Benchmark: NVIDIA DGX Spark (GB10) 121GB — 11.1+ tok/s full-k8 CUDA (PLD); was 2.4 / 3.33 chat #161 (comment)
• Quality snapshot JSON in that repo under results/

Happy to extend with greedy-token agreement on a small reasoning set (not just primes) when the box is free — that should help tune ROUTE_J / ROUTE_M.

[VincentMarquez]

Independent repro — thank you (@ebootheee)

This is exactly the cross-arch confirmation that was needed.

	GB10 (unified, our notes)	x86 dual 5090 (your box)
Stock full-k8	~2.39 decode	2.255 (1024-tok window)
J=2 M=12	3.33 (~+39%)	3.09 (~+42%)
Substitution	~14%	15.3%
Quality	HellaSwag n=20 canary only	MMLU-200: 59% vs 62%

Quality read: agree 100% — at n=200, ±~3.5 pp noise means no detectable difference, not “CR helps.” That is already a strong statement: ~15% expert substitution did not measurably dent MMLU. Full bench / --limit 1000 would be welcome as the real claim gate.

Discipline: also agree these stay off the main true-routing table (research rows only). Same two-tier rule we use on Spark: stock top-8 vs experimental CACHE_ROUTE; never mix Tier-C CUDA stacks into a “CR alone” row.

Upstream status: the routing-side path is already landed as an official opt-in (default off) in PR #199 (CACHE_ROUTE / ROUTE_J / ROUTE_M, optional ROUTE_AGREE + swap% on the stats line). No need for fork magic if you’re on a post-#199 tree — reimplementation from the write-up is still a great independent check.

@JustVugg — +1 to “research flag” framing: big README warning, substitution rate on the result line, benchmark rows separated. Happy to help fold any remaining docs/flags with the #215 / #270 stack if useful.

Notes updated: https://github.com/VincentMarquez/glm52-gb10-colibri (QUALITY + NUMBERS).

[woolcoxm]

Windows 11 native — GLM-5.2 744B int4 — RTX 5070 Ti + Core Ultra 9 185H + 32 GB RAM

Setup: MinGW-w64 build, NVMe Gen4, CPU-only AVX-VNNI matmul + CUDA dense/attention offload via runtime DLL (coli_cuda.dll).

Decode throughput — progressive optimization (32-token decode, DRAFT=0)

step	config added	tok/s	hit%	expert-disk	decode (32 tok)
0	stock EXPERT_BUDGET=4	0.33	9%	65.9s	97.4s
1	+ PIPE=1 (async expert load)	0.34	—	—	93.4s
2	+ RAM_GB=28 (larger cache cap 1→3)	0.39	—	—	81.5s
3	+ PILOT_REAL=1 (cross-layer prefetch)	0.42	—	—	77.1s
4	+ DIRECT=1 (O_DIRECT / FILE_FLAG_NO_BUFFERING)	0.63	72%	21.5s	50.6s
5	+ CUDA_DENSE=1 COLI_CUDA_ATTN=1 (GPU attention)	0.72	75%	18.4s	44.5s
6	+ COLI_CUDA_PIPE=2 (GPU resident pipeline at decode)	0.85	57%	18.2s	37.8s
7	+ ws_b cache reserve fix (cap 3→4, matches budget)	1.03	73%	12.4s	31.0s
8	+ CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12 (cache-aware routing)	1.07–1.43	83%	8.4s	22–30s

Total: 3.2–4.3× over stock. Disk I/O reduced 8× (65.9s → 8.4s).

Full config

EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0
CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12

Per-token breakdown (accurate, from non-pipe2 run to avoid overlap artifacts)

phase	ms/token	%	notes
Expert matmul (CPU serial)	386	36%	gate+up+down for ~5 experts/layer × 75 layers
Other (rmsnorm + routing + residual)	334	31%	156 rmsnorms + 75 router GEMMs + CACHE_ROUTE logic — mostly moved to GPU via pipe2
Expert disk I/O (PIPE-overlapped)	241	23%	~50 misses/token × 19MB, 82% hit rate
Attention (GPU)	111	10%	MLA absorb + projections on RTX 5070 Ti

Key findings

1. CACHE_ROUTE is the single biggest win (+37%): steers the MoE router to prefer cache-resident experts. route_agree=94.6% confirms minimal quality cost. This is the right fix for GLM-5.2's near-uniform routing (237 of 256 experts used per layer across 75 tokens — no "hot" experts to pin).

2. pipe2 resident pipeline unblocked by token-exactness proof: CUDARES = CUDAOFF = DIRECT token-for-token across 4 prompts. The resident pipeline is a pure execution-location change, not a math change. (CUDA greedy may differ from CPU on near-tie argmax — FP accumulation order drift, not a bug.)

3. The "other" bucket is the new frontier: 31% of decode is untracked CPU glue (rmsnorm, residual, router, top-K, gather/scatter). On a warm cache this is bigger than disk I/O. Fusing rmsnorm+residual and tightening the router/top-K path are hardware-independent wins.

4. Windows compat layer was essential: compat_fadvise WILLNEED cache-warmer (replaces no-op), PIPE default ON for Windows, FILE_FLAG_NO_BUFFERING for expert reads.

What didn't work (documented for the record)

• Windows mmap (COLI_MMAP): RSS bloat collapses cache budget (Windows-vs-Linux page-cache difference)
• CUDA expert matmul (CUDA_EXPERT_GB): sync-stall regression (16GB VRAM too small for expert tier)
• Async attention dispatch: overlap window too small (~0.1ms CPU vs 1.2ms GPU)
• ROUTE_M=24: faster but quality degrades (KL 2.78 vs 0.92)
• I4S=1 (IDOT at S=1): quantization overhead exceeds VNNI speedup
• pc_b reserve removal: memory pressure at 24GB RSS drops hit rate

Hardware details

component	spec
GPU	NVIDIA RTX 5070 Ti (17.1 GB VRAM, sm_120 Blackwell)
CPU	Intel Core Ultra 9 185H (6P+8E+2LP, 22 threads, AVX-VNNI)
RAM	32 GB DDR5
SSD	NVMe Gen4
VRAM usage	9.7 GB dense + 0.5 GB pipe2 scratch = 10.2 GB of 17.1 GB
RSS	~21 GB of 32 GB (16.7 dense + 5.8 expert cache + overhead)

Full diagnostic report with raw profiles, memory hierarchy analysis, and ranked optimization recommendations: DIAGNOSTIC.md on the windows-optimizations branch. PR #274, issues #256/#258/#273.

[allan-56]

3.6 tok/s on AMD EPYC 7713, 96 cpu avx2, 1 TB RAM for model, 1 TB RAM for full expert loaded

Hardware:  2× AMD EPYC 7713 64-Core Processor (Zen 3), 256 threads / 128 cores, 2 NUMA nodes, AVX2 kernels (idot: avx2),  4096 GB DDR4 (32×128GB Samsung DIMMs, DDR4-3200, running DDR4-2667), bare metal (2TB used), Micron 7450 NVMe SSD ( Cached read: 16.2 GB/s, Buffered read: 2.0 GB/s)
OS:        Ubuntu 22.04.3, 5.15.0-181 kernel, GCC 11.4.0
Commit:    62419af
Model:     mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
Command:   OMP_NUM_THREADS=96 DRAFT=3 OMP_PROC_BIND=spread OMP_PLACES=cores COLI_NO_OMP_TUNE=1 PIN=~/colibri/models/glm52/.coli_usage PIN_GB=1000 COLI_MODEL=~/colibri/models/glm52 numactl --interleave=all ./coli run --ram 1000 --gpu none --temp 0 "test"
Warm-up:   .coli_usage
Best result:    3.60 tok/s

Run	Threads	NUMA	Bind	Draft	Warm RAM	Hit rate	Prefill	Decode tok/s	MTP acceptance	Notes
1	64	interleave	spread	1	213GB	93.5%	4.18s	1.57	65%	cold-ish cache
2	128	interleave	spread	1	213GB	100%	1.55s	3.55	65%	stable
3	256	interleave	spread	1	213GB	100%	25.97s	~0.15	65%	bad regression
4	96	interleave	spread	1	213GB	100%	1.58s	3.60	65%	best
5	112	interleave	spread	1	213GB	100%	1.51s	3.54	65%	similar
6	112	interleave	close	1	213GB	100%	1.49s	3.55	65%	same
7	96	interleave	close	1	213GB	100%	1.58s	3.58	65%	same
8	96	node0 only	spread	1	213GB	100%	1.32s	3.29	65%	slower decode
9	96	interleave	spread	3	227GB	97%	1.56s	1.56	38%	draft hurts

Best run:

== GLM C engine (glm_moe_dsa), cache=8 experts/layer | experts@8-bit dense@8-bit | idot: avx2 ==
loaded in 6.42s | resident dense: 9912.75 MB | layers=78 experts=256 | MTP ACTIVE (draft=1)
[MTP] active: native speculative decoding (draft=1)
[PIN] placement: 0 VRAM + 12088 RAM expert (228.6 GB warm) in 38s da /home/ubuntu/colibri/models/glm52/.coli_usage
[USAGE] expert history: 328416 selections (/home/ubuntu/colibri/models/glm52/.coli_usage)
[RAM_GB=1000.0] cap raised 8->256: budget allows it (projected peak 618.0 GB; set CAP_RAISE=0 to disable)
[stop] 3 stop tokens: 154820 154827 154829
prompt: 7 tokens | generating up to 1024 (EOS stop=154820) | n-gram draft=1
[gMASK]<sop><|user|>test<|assistant|><think></think>PROFILO PREFILL (1.58s):
PROFILE: expert-disk 0.000s service / 0.000s wait | expert-matmul 1.187s | attention 0.222s (including kvb 0.016s) | lm_head 0.007s | other 0.165s
ATTENTION: projection/RoPE 0.080s | score-softmax-value 0.010s | output projection 0.115s
I am working perfectly! How can I help you today?

*(If you
[t=16  RSS 227.26 GB  hit 100%  3.44 tok/s  1.50 tok/fw]
 are just testing the system, everything looks good! Let me know what else you
[t=32  RSS 227.26 GB  hit 100%  3.60 tok/s  1.63 tok/fw]
 would like to do.)*
---
prefill 7 tokens in 1.58s | decode 38 tokens in 10.71s (3.55 tok/s) | expert hit rate 100.0% | RSS 227.26 GB
experts loaded/token: 734.3 (per-layer 9.79 across 75; baseline topk=8) | TOPK=0 TOPP=0.00
speculation: 1.65 tokens/forward (23 forwards per 38 tokens) | MTP acceptance 65% (15/23)
PROFILE: expert-disk 0.000s service / 0.000s wait | expert-matmul 6.411s | attention 2.527s (including kvb 0.000s) | lm_head 0.000s | other 1.776s
ATTENTION: projection/RoPE 0.659s | score-softmax-value 0.804s | output projection 1.061s

Ref: #234

Please let me know if you think I can tune this any further.

[ZacharyZcR]

6× RTX 5090 full stack, current dev: 5.2 tok/s end-to-end through HTTP/SSE, TTFT 0.43 s

The strictest measurement we've taken of this rig: client-side, streaming over the OpenAI SSE API through an SSH tunnel — every layer of the real serving stack included, nothing engine-side about it.

Hardware: dual Xeon Silver 4510 (24C/48T, AVX-512 VNNI, 2 NUMA nodes) · 251 GB DDR5 · 6× RTX 5090 32 GB · NVMe
Engine: current dev (post #313/#274/#294/#232) · Config: CUDA_EXPERT_GB=auto (176.6 GB VRAM tier, 9,280 experts across 6 devices) + PIN_GB=all (rest pinned in RAM — disk 0) + CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 COLI_CUDA_TC_W4A16=1 + COLI_NUMA=1 (#313, first production run) + OMP_NUM_THREADS=24 OMP_PROC_BIND=spread + DRAFT=0

Six consecutive 96-token turns (client-measured SSE chunk timing):

turn	TTFT	decode
1 (fresh prompt)	2.74 s	3.94 tok/s
2–3 (KV prefix hit, warming)	0.47 s	4.55 → 4.98
4–6 (plateau)	0.43–0.55 s	5.13 → 5.22 → 5.24 tok/s

Three things the curve shows:

1. The REPIN warm-up is textbook: 3.94 → 5.24 over six turns as the adaptive tier swaps hot experts into VRAM slots (every 16 tokens).
2. KV prefix reuse makes TTFT ~free on continued conversations: 0.43 s, with the new [API] KV slot 0 prefix 19/24, prefill 5 diagnostic (serve: emit the [API] KV prefix-reuse diagnostic in mux mode (#153) #248) confirming only the delta prefills.
3. This is ~20% above the same rig on last week's engine at the same E2E protocol-stack measurement — the delta is the accumulated dev work (resident pipeline at decode, NUMA interleave, kernel consistency). Engine-side decode-only on this rig remains 6.0–6.8 (README experiment log); the gap between that and 5.2 is the honest cost of HTTP+SSE+tunnel.

One bug found while measuring: after the recent telemetry merges, the server's colibri.stats response field comes back empty (the SSE side stream works; the JSON attach doesn't) — will send the fix.

[ZacharyZcR]

Full-residency decode on 6× RTX 5090 is CPU-bound, not GPU-bound — a direct overlap-window measurement

Following up on the earlier 5.2 tok/s datapoint with a mechanism result that surprised me. I instrumented moe() at decode to split the per-turn expert time into the GPU-group window vs the CPU rows, running them concurrently (issue the VRAM-resident experts' grouped GPU calls, then compute the RAM-tier/miss rows on the CPU while the GPUs work, then collect). Over a 512-token greedy run, full residency (CUDA_EXPERT_GB=auto PIN_GB=all, 9,276 experts in VRAM, 1,604 in RAM):

overlap window: pack+issue 3.20s | cpu-rows 44.55s | take(sync+acc) 0.63s

The GPU side of a token's experts costs ~3.8 s of wall time; the CPU side costs 44.5 s. The six 5090s finish their 9,276-expert share and then idle, waiting on the CPU to grind through the ~90 RAM-tier experts each token routes to (AVX-512 VNNI, 1–4 rows each). Independent confirmation from SM sampling during decode: 1.1% mean utilization. The cards are not the bottleneck and never were — at S=1 there simply isn't enough GPU work, and what work exists is dwarfed by the CPU tail.

Consequences for where speed lives on this class of host:

1. Concurrency and speculation don't help (measured separately: aggregate throughput is flat 1→8 streams because independent streams' expert sets barely intersect; speculation's S=4 verify batch is a net 3× loss). The MoE sparsity that makes the model fit is the same thing that starves small-batch GPU utilization.
2. The lever is the RAM tail, not the GPU. Every expert you can move from RAM into VRAM is one the CPU no longer computes. Two ways there: a streaming-weight GPU path for the RAM experts (upload-compute-discard on the idle cards — PCIe cost vs the 44 s CPU cost, worth measuring), or a smaller expert container so the whole set fits VRAM. The latter is why the int2/2.5-bit lattice work (Rotation-preconditioned quant for streamed experts (int2/int3 quality) worth considering? #81) matters for speed, not just disk: a ~200 GB container puts all 19,456 experts on six 32 GB cards and takes the CPU out of the decode loop entirely.

The COLI_GROUP_ASYNC=1 overlap that produced this measurement is up as #342 (opt-in, +6% end-to-end — small precisely because it hides GPU time that was never the bottleneck; its value is the diagnostic).

[mkultra333]

Windows 11 Native + External USB Thumb Drive

I ran it on a prompt overnight so I figured I may as well post a benchmark. I'm not using an NVME SSD because I don't have enough room on my main os drive. I'm running it on an EXTERNAL USB DRIVE, results are suprisingly good given that, the overnight run got 0.12 tokens per second. --ngen is 10240.

Hardware:
AMD Ryzen 5 3600X 6-Core Processor, 3800 Mhz, 64GB ram. RTX 5060ti 16GB.
External 2TB SSD via USB 3.2: buffered x8 threads: 64 reads x 19MB = 1.3 GB in 1.30s -> 0.98 GB/s (20.3 effective ms/block)
OS: Windows 11 (Native, no WSL)
Commit: 550ddcb
Model: https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
Command:
env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="14";
python coli chat --model S:\Colibri_GLM52 --ngen 10240 --policy balanced
Warm-up: Have run the 30 prompts plus messed around with it, so I guess warmed up?
Result: 2529 tok · 0.12 tok/s · hit 50% · RSS 44.6 GB · 21004s

Edit: Ah, I get the impression I did this wrong, you're supposed to run a particular set of settings and scripts for the official benchmarks.
At any rate, it might still be interesting to see some USB drive SSD numbers. I cleared some space on my M.2 nvme C: drive and copied the model over there, just to see the difference. It was underwhelming, about 25% faster and not worth the wasted space and extra heat wear and tear. The official numbers for the nvme SSD are:
buffered x8 threads: 64 reads x 19MB = 1.3 GB in 1.00s -> 1.27 GB/s (15.7 effective ms/block)

[drjoga]

2.6 tok/s on Xeon W-2265, 512 GB DDR4-2400, RTX 3060 12 GB

CPU              : Intel Xeon W-2265 @ 3.50GHz (12 Cores / 24 Threads, AVX-512 VNNI)
System Memory    : 512 GB DDR4 ECC (Quad-Channel @ 2400 MT/s)
Storage Controller: Crucial T500 PCIe 4.0 NVMe SSD (rev 01)
Measured Disk I/O: 5.80 GB/s (Buffered, 8 threads) | 3.16 GB/s (O_DIRECT, 8 threads)
OS / Kernel      : Ubuntu 24.04.1 LTS (Linux 6.17.0-35-generic x86_64)
Compiler / Tools : GCC 13.x / nvcc (CUDA 13.1)
Engine Version   : colibri v1.1.1 (Commit 81f08a0)
GPU              : NVIDIA GeForce RTX 3060 12GB (Ampere sm_86, PCIe 4.0 x16)

Median tok/s of five runs:

Step	Configuration Increments	32 Tokens	1024 Tokens	Hit Rate (Pin / LRU)	RSS (Peak)
1	Baseline (Out-of-the-box Defaults)	1.27 tok/s	1.34 tok/s	99.0% (92.1%)	344.72 GB
2	+ PIN=auto PIN_GB=400, Max RAM & Expert Pinning	1.57 tok/s	1.45 tok/s	100.0% (100%)	357.96 GB
3	+ MTP=0, Disable MTP	2.48 tok/s	2.13 tok/s	100.0% (100%)	348.68 GB
4	+ COLI_CUDA=1 CUDA_EXPERT_GB=11, CUDA Offload 11GB VRAM	2.60 tok/s	2.24 tok/s	100.0% (100%)	348.85 GB
5	+ TOPP=0.7, Top-P Expert Pruning	3.23 tok/s	2.73 tok/s	100.0% (100%)	348.85 GB

---

Key Takeaways from the Optimization Steps

1. Step 1 $\rightarrow$ Step 2 (Memory Pinning):
   Ensuring all 19,200 MoE experts pinned in memory bumped the pin hit rate from 92.1% to 100.0%, eliminating disk I/O wait times and raising throughput from 1.34 tok/s to 1.45 tok/s (1024 tokens). Baseline config does not pin all experts.
2. Step 2 $\rightarrow$ Step 3 (Disabling Speculative MTP):
   Disabling speculative verification (MTP=0) delivered the single largest performance jump across all steps—increasing long-generation throughput by +46.9% (to 2.13 tok/s) and short-generation speed by +58.0% (to 2.48 tok/s). On CPU-bound quad-channel memory setups, pure single-token forward evaluation is substantially faster than MTP verification overhead.
3. Step 3 $\rightarrow$ Step 4 (GPU Hot-Expert Offloading):
   Offloading 548 hot experts (~10.37 GB VRAM) to the NVIDIA RTX 3060 provided a steady +5.2% speedup for long sequences (2.24 tok/s) and +4.8% for short sequences (2.60 tok/s).
4. (Step 4 $\rightarrow$ Step 5 (Top-P Expert Pruning, Trading quality for speed))

[drjoga]

3.5 tok/s on i9-7940X, 256 GB DDR4-3200, 2 x RTX 3090

CPU               : Intel Core i9-7940X @ 3.10GHz (14 Cores / 28 Threads, AVX-512)
System Memory     : 256 GB DDR4 (Quad-Channel @ 3200 MT/s)
Storage Controller: Western Digital WD_BLACK SN850X 2TB NVMe SSD
Measured Disk I/O: 4.3 GB/s (Buffered, 8 threads) | 3.2 GB/s (O_DIRECT, 8 threads)
OS / Kernel       : Ubuntu 24.04 LTS (Linux 7.0.0-28-generic x86_64)
Compiler / Tools  : GCC / nvcc (CUDA 13.2, Driver 595.84)
Engine Version    : colibri v1.1.1 (Commit 1b8b62e)
GPU               : 2x NVIDIA GeForce RTX 3090 24GB (Ampere sm_86, 48GB Total VRAM)

Step	Configuration Increments	32 Tokens (Decode Speed)	Hit Rate (Pin / LRU)	RSS (Peak)
1	Baseline (Defaults)	0.79 tok/s	90.4% (73.6% / 16.8%)	204.49 GB
2	+ COLI_CUDA=1 COLI_GPUS=0,1	1.08 tok/s	91.3% (80.1% / 11.1%)	190.63 GB
3	+ CUDA_EXPERT_GB=auto (46GB Dual 3090 VRAM)	1.68 tok/s	95.1% (90.3% / 4.8%)	166.49 GB
4	+ RAM_GB=250 PIN=auto PIN_GB=200	2.82 tok/s	98.6% (97.9% / 0.7%)	215.44 GB
5	+ PIN_GB=232 (Maximize RAM Warm Set)	3.51 tok/s	99.3% (99.2% / 0.0%)	228.28 GB
6	+ TOPP=0.7 (Top-P Expert Pruning)	4.39 tok/s	99.2% (99.0% / 0.2%)	228.19 GB
7	VRAM+RAM (RAM_GB=204 PIN_GB=186)	2.74 tok/s	97.7% (97.4% / 0.2%)	185.54 GB
8	RAM Only (COLI_CUDA=0 RAM_GB=250 PIN_GB=232)	2.04 tok/s	97.1% (96.8% / 0.3%)	227.91 GB

Step 7 and 8 is to compare the effect of having the same number of resident experts either dispersed over GPU+CPU (Step 7), or only on CPU (Step 8, which allocates no VRAM, but 46 GB more RAM). Having a subset of experts in VRAM is faster as expected, but this comparison provides a value on how much faster. When activating CUDA in step 2, MTP seems to get disabled automatically.

[brad-evony]

Hmm... My system specs are somewhat similar to yours, but I was only getting around 0.8 tok/s. I will have to try the settings you listed here, as it might improve my speed a lot.

[anagingcompiler42]

0.10 tok/s on old cpu/ram/disk over ethernet

Hardware: Dual Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz on host, 8x vcpu in vm / 480 GB DDR4-2400 ECC RDIMM / SAN with 15k rpm drives over gigabit ethernet / no gpu
OS: Ubuntu 24.04.4 LTS
Model: GLM-5.2 int4 (mateogrgic container), int8 MTP heads byte-verified
Compiler / Tools : GCC 13.3.0 / nvcc n/a
Engine Version : colibri v1.3.0
Command: COLI_MODEL=~/glm52_i4 ./coli chat --ram 460
Warm-up: cold and a second run
Result: Cold: 354 tok * 0.05 tok/s * hit 93% * RSS 344.6 GB * 6555s
Result: 2nd: 305 tok * 0.10 tok/s * hit 99% * RSS 366.9 GB * 3118s

Yes, I knew it would be like this, but I wanted to see.

[bokiko]

Kimi K3 thread-configuration and cache-scaling datapoints from three hosts.

This builds directly on work already in this project. @MichaelFomenko ran K3 end to end first and posted a K3_EXPERT_GB sweep from a RAM-constrained box; @terrizoaguimor did the cache-sizing analysis that made those numbers legible; and @dlewis7444 measured the thread-count effect on GLM/Zen3 in #718, along with the median-of-three discipline I follow below. What follows is a Kimi-specific reproduction and extension of that thread finding across three hosts, plus a cache ladder on a machine with enough RAM to go past the usual range.

Summary: Explicit OpenMP configuration reduced K3 decode time by 3.5× on the 64-core host and 40% on one 16-core host; the third host was inconclusive. On the 256 GB host, increasing the expert cache from 8 to 180 GB improved decode from 2.78 to 1.53 s/token. These are narrow, 32-token performance datapoints, not quality results or universal tuning advice.

Scope: one 10-token prompt, 32 greedy output tokens, v1.3.0 source tag (01abef3), ARCH=native. Model verification: all 96 shard sizes and total shard bytes matched the HF manifest, with full-shard SHA256 spot-checks against declared LFS hashes. Correctness evidence is coherent English output only — no quality benchmark was run, and no token-ID or logit comparison was made.

Hosts

	A	B	C
CPU	Threadripper PRO 5995WX, 64c/128t (avx2, no AVX-512)	Ryzen 9 9955HX, 16c/32t	Ryzen 9 7950X, 16c/32t
RAM	256 GB DDR4-3200 ECC, 8 ch	92 GB DDR5-5600	96 GB DDR5-6000
Governor	performance	performance	powersave
Model storage	3×NVMe RAID0, ext4	NVMe, ext4	NVMe, ext4

OpenMP thread configuration

All rows: decode only — initialization and prefill excluded. K3_EXPERT_GB=45, ARCH=native, all other K3_* at v1.3.0 defaults.

host	OpenMP env unset (n=3 median, range)	explicit setting (n=3 median, range)	change
A (64c)	211.7 s (203.5–233.5)	61.1 s (61.1–61.1) — OMP_NUM_THREADS=32 OMP_PROC_BIND=spread OMP_PLACES=cores	−71% (3.5×), confirmed
B (16c)	163.6 s (160.9–163.6)	97.5 s (97.5–97.6) — OMP_NUM_THREADS=16, same binding	−40.4%, confirmed
C (16c)	165.6 s (153.9–172.8)	180.3 s (147.0–202.2)	inconclusive

"Confirmed" here means n=3 per arm with non-overlapping observed ranges — nothing stronger.

How this was run, so the comparison can be judged: three OpenMP settings changed together (thread count, OMP_PROC_BIND, OMP_PLACES), so no arm isolates which one carries the effect. Runs were executed in configuration blocks rather than interleaved, and page-cache state between runs was not controlled. All arms for a given host were same-host, same-day, same-binary.

• No single thread-count rule follows from this. Host A's best was 32 threads on 64 physical cores; host B's was 16 on 16. Host A at 64 bound threads gave 65.5 s (n=1), close to its 61.1 s result at 32 threads. This suggests reducing below the physical-core count added little in that diagnostic, but it does not isolate thread count from affinity.
• Host C is inconclusive, not negative. Its two arms' ranges overlap (bound 147.0–202.2 vs unset 153.9–172.8), and its bound runs were monotonically increasing — 147.0 → 180.3 → 202.2, a 1.38× range, CV ≈ 16% — while its governor was powersave. Its first run alone (147.0 s) looked like an 11% win that the next two did not support. Neither a gain nor its absence can be claimed there.
• Mechanism not established. Consistent with SMT contention, thread placement, or all-core frequency/power behaviour; none of these were profiled.
• On SIMD: with K3_IDOT at its default the expert matmul is matmul_mxfp4_i8, and all three hosts select its same AVX2 source arm (the alternatives are that arm and scalar). Generated native machine code was not compared.
• At the default 8 GB cache, host A went 337.5 s → 89.0 s (n=1 per side). That is consistent with the n=3 result above; it does not independently establish it.

Relevant code context: colibri.c and inkling.c both open main() with an OpenMP tuning block (wait policy, spin count, OMP_PROC_BIND=close, then re-exec), and resource_plan.py already advises setting OMP_NUM_THREADS to physical cores. kimi_k3.c has none of this. I am not claiming that block is a drop-in fix: it uses close binding where I tested spread, it does not cap thread count, and my intervention bundled three settings.

Scope of the settings above: these were used for Kimi K3 testing only. Explicit affinity is topology-sensitive, and #471 documents a different engine's re-exec interacting badly with user-set OMP_PROC_BIND/OMP_PLACES. Treat them as something to measure on your own host with at least three runs per configuration — not as general colibrì advice.

Cache scaling (host A, fixed 32-thread configuration)

K3_EXPERT_GB	slots/layer	hit	GB streamed	decode	s/token	peak RSS	n
8 (default)	4	6.7%	891	89.0 s	2.78	44.8 GB	1
45	27	38.8%	585	61.1 s	1.91	80.8 GB	3
180	111	60.6%	376	48.8 s (48.6–48.9)	1.53	212.8 GB	3

Accounting caveat: hit rate and streamed bytes are cumulative over prefill plus decode, while decode seconds and s/token cover generation only. Do not divide those columns against each other to infer bandwidth — they cover different windows.

Hit rate keeps rising well past the cache sizes reachable on a 64 GB box, at a resident cost worth knowing before trying it: 212.8 GB peak on a 256 GB host.

int8 dense residents

host A, 32 threads, both at 160 GB cache	decode	hit	peak RSS
K3_BITS=4	46.3 s	59.3%	193.9 GB
K3_BITS=8	51.6 s	58.0%	214.3 GB

An observed 11.4% end-to-end difference and +20.4 GB resident, for this prompt, n=1 per arm. This is the end-to-end effect of the setting including the routing changes it produces — the arms did not generate identical text — so it is neither an established cost nor a quality result.

Not tested

• Token-level determinism: hosts B and C printed matching output text and reported identical expert-hit counters (20630/54455); host A's counters differ (54461). No token-ID or logit diff was run, so only printed text and counters are being reported.
• K3_LOAD_THREADS at 8 and 16 on host A changed decode by under 3% (n=1 each). Note the engine clamps this to 16, so 16 was already the ceiling.
• Any quality measurement, longer contexts, other prompts, sampling configurations, or an XFS comparison.

No completed runs were discarded from the comparisons reported above. Additional completed storage and diagnostic arms remain documented in the internal report. Raw logs (init curves, per-token traces, time -v, load snapshots) are kept and can be shared.

[mohamedmastouri2000-boop]

RTX 5080 16 GB · 128 GB DDR5 · NVMe — int4-g64 vs E8/IQ3 (fmt=6), same FP8 parent

Container swap on one box. colibrì v1.4.0 (b085b48), Core Ultra 9 285K, SK Hynix PCIe 4.0 NVMe, Windows. Balanced CACHE_ROUTE J=3 M=16 α=0.5, CUDA_DENSE=1 CUDA_EXPERT_GB=4 PIPE=1 DIRECT=1, greedy, NGEN=64. Two replicates per arm, warm-up discarded, .coli_usage restored from a master copy before every run.

	int4-g64 (4.50 bpw)	E8/IQ3 (3.06 bpw)
on disk	429.3 GB	289.1 GB
decode, balanced J3/M16	1.19, 1.17	1.56, 1.58
decode, speed J2/M24	1.41	1.59
prefill, 19 tokens	15.5 s	10.2 s
expert hit rate	93.5%	97.4%
decode disk wait	12.7 s of 49.8 s	2.7 s of 41.0 s

Quality — same harness (tools/eval_glm.py, per-choice loglikelihood, no generation), both containers from the same FP8 parent:

task	n	int4-g64 acc / acc_norm	E8 acc / acc_norm
hellaswag	100	73.0% / 86.0%	72.0% / 88.0%
arc_challenge	40	70.0% / 70.0%	67.5% / 75.0%
mmlu	40	85.0% / 85.0%	87.5% / 87.5%

33% smaller, 22–33% faster, no measurable quality loss.

The win is residency, not arithmetic. E8's expert-matmul is higher — 24.1 s against 20.9 s — so the decode itself is dearer. It wins because smaller experts mean more of them fit, and the disk wait collapses. On a host where experts are already 100% resident the same format is a decode regression, measured by @ZacharyZcR on 6×5090 in #452. Full analysis, the three quality caveats and the placement rule are in my comment there.

---

One measurement note that may be more useful than the numbers, for anyone benchmarking a streaming host:

The same g64 arm read 1.275 tok/s in the morning and 1.18 in the evening on this box, from nothing but ~20 hours of sustained NVMe load (a 289 GB conversion and two long eval runs). That looked like a v1.4.0 regression. Before reporting one I reinstalled the v1.3.0 binaries and re-ran the same arm in the evening machine state: 1.205. So the version accounted for 0.025 — inside this box's ±0.04 noise band — and the machine accounted for 0.07. No regression.

Across all three sessions E8 read 1.56 / 1.56 / 1.57 while g64 slid 1.275 → 1.205 → 1.18. A tiring drive punishes the arm that waits 15 s on disk far more than the arm that waits 2.4 s. Two consequences worth carrying: never compare a number taken today against one taken yesterday without re-measuring both arms in the same session, and expect the disk-bound arm to be the one that drifts.