[datapoint] Qwen3.6-35B-A3B on Tesla M10

[ggeorgovassilis]

Summary

The CUDA backend works on Maxwell (Tesla M10, sm_50, compute capability 5.0). This is the first validation below the sm_80 compile-verified floor, and as far as I can tell the first Maxwell-era datapoint in the repo. The portable quant_matmul kernels are correct on sm_50 (the full cuda-test suite passes on actual hardware) and reasonably fast where they are used. All four GPUs together (note: the M10 is a PCI card which combines 4 GPUs) hold 100 % of the 10,240 experts of Qwen3.6-35B-A3B in VRAM (zero CPU misses). End-to-end decode is 2.45 tok/s, but the bottleneck is not the GPUs: it is the dense/recurrent trunk, which qwen36 keeps on a Sandy Bridge CPU with no AVX2/FMA. For reference, the same hardware does 7.3 tok/s under llama.cpp (-ngl 99 -ts 1,1,1,1) when the whole model is offloaded.

Environment

• GPU: 4x Tesla M10 (Maxwell, sm_50, no tensor cores), 8 GB VRAM each (32 GB combined), driver 580.173.02 (host CUDA 13.0)
• CPU: 2x Intel Xeon E5-2640 @ 2.50 GHz (Sandy Bridge), 12 physical cores / 24 threads; avx, sse4_1, sse4_2 present, avx2, fma, avx-512 absent
• RAM: 128 GB (rebalanced 64 GB / 64 GB across the two NUMA nodes)
• Storage: NVMe SSD
• OS: Ubuntu 24.04.2 LTS (Noble Numbat), kernel 6.8.0-138-generic
• Docker: 29.7.2 (client and server), with the nvidia container runtime (nvidia-container-toolkit 1.20.0). All runs were made in containers; the GPU runs used --gpus "device=..." to expose the devices, the CPU run used none.
• Toolchain: CUDA 11.8 (the last toolkit that still targets sm_50; Maxwell was dropped in CUDA 12.0), inside nvidia/cuda:11.8.0-devel-ubuntu22.04
• Commit: 9d5d05d (v1.12.0-4-g9d5d05d, main)
• Model: Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 (Qwen3.6-35B-A3B, int4 / gs=64, colibri-native container, 22 GB)

Build

make -C c qwen36 CUDA=1 CUDA_ARCH=sm_50 ARCH=native

ARCH=native is required on this CPU (not x86-64-v3): the host has no AVX2, and a -march=x86-64-v3 binary SIGILLs. With -march=native the AVX2 dense-int8 kernels compile out and the engine falls back to per-row/scalar paths. Correctness is unaffected; only CPU speed is.

The WMMA/tensor-core path is correctly skipped: it is gated compute_major >= 7 and the M10 reports sm_50, so the portable path is what runs.

Benchmark command (4-GPU warm run shown; other rows vary only COLI_GPUS, OMP_NUM_THREADS, and HEAT_FILE):

docker run --rm --gpus '"device=0,1,2,3"' -v ~/colibri:/src \
  -v /mnt/ssd2/models/colibri:/models -w /src/c colibri-cuda:11.8 bash -lc \
  "COLI_CUDA=1 COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto OMP_NUM_THREADS=12 \
   OMP_WAIT_POLICY=ACTIVE OMP_PROC_BIND=close \
   SNAP=/models/qwen36_i4_gs64 TOK=/models/qwen36_i4_gs64/tokenizer.json \
   N_NEW=200 COLI_TIMERS=1 HEAT_FILE=/models/qwen36_i4_gs64/heat.bin \
   ./qwen36 256 4 prompt.txt"

Correctness

make -C c cuda-test CUDA=1 CUDA_ARCH=sm_50 on the real M10, exit 0, all pass:

• backend_cuda_test q8/q4/q2/f32/e8 correctness on 1 device
• ragged_attention_test rms=0 (exact)
• fp8_cuda_test / mxfp4_cuda_test 0 mismatches (including the exp-255 cases)
• absorb_determinism_test, weights_owned_test, cuda_fmt_trap_test, alloc_footprint_test all OK

So the portable kernels produce correct results on Maxwell, with no garbage and no drift. Real-model generation is coherent (Qwen3.6 is a reasoning model, so output is a "thinking process" followed by the answer).

CPU-vs-GPU token parity (greedy). Direct argv mode is pure greedy argmax (no temperature/top-p on this path), so I ran the same prompt through both tiers and byte-compared the decoded output:

Run	Decode tok/s	TTFT	Experts in VRAM	VRAM hit %	CPU-fallback misses
CPU-only (COLI_CUDA unset)	0.09	171.0 s	n/a	n/a	n/a
GPU mixed tier (COLI_GPUS=0, CUDA_EXPERT_GB=auto + heat.bin)	2.00	5.5 s	3245 / 10240	86.3	3409

Both produced byte-identical 64-token greedy output (258 bytes, md5 89d059e3…). The GPU run did exercise the mixed tier (3245 experts VRAM-resident, 86.3 % hit, 3409 CPU-fallback misses), yet the argmax path stayed token-exact end to end. This is a positive contrast to #510, where the HIP/RDNA2 (gfx1030) non-WMMA fallback showed a one-token greedy divergence on a mixed tier; on Maxwell/sm_50 the CUDA portable kernel and the CPU path agree on this model. As a side note, the GPU tier turns a 12.5-minute pure-CPU generation into 32 s (about 22x) on this no-AVX2 box.

Results

Prompt was the repo's canonical c/prompt.txt ("Explain the difference between a stack and a queue in three sentences."); each run decoded 200 new tokens (N_NEW=200), greedy (no temperature/top-p on the argv path). Each configuration was a single warm run (n=1), so medians and spreads do not apply.

Config	Decode tok/s	VRAM hit %	Experts resident	CPU misses	Peak RSS
1x GPU, cold	1.38	32.0	3245	>0	42.1 GB
1x GPU, warm (heat.bin)	1.81	85.7	3245	>0	42.0 GB
4x GPU, warm, 6 thr	2.01	100	10240/10240	0	22.8 GB
4x GPU, warm, 12 thr (2nd CPU)	2.45	100	10240/10240	0	22.3 GB

The 4-GPU spread is the main viability result: COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto makes all 10,240 experts VRAM-resident (100 % hit, zero CPU misses, zero LFRU swaps), halves the MoE time (161 to 91 ms/token) and halves peak RSS (42 to 23 GB) because experts are no longer duplicated in system RAM.

Where the bottleneck actually is

Per-token (4-GPU warm, 12 threads):

Phase	Cost	Runs on
DeltaNet trunk	169 ms	CPU
attention	55 ms	CPU
MoE	91 ms	all 4 GPUs
lm_head	69 ms	GPU 0

qwen36.c only calls qt_trunk_offer("lmhead") and qt_trunk_offer("dnproj"); the dense DeltaNet/attention path stays on the CPU. On this no-AVX2 Sandy Bridge that dense trunk is the ceiling, not the GPUs. Two controls confirm it:

1. Second CPU (6 to 12 cores): 2.01 to 2.45 tok/s (+22 %), with only the CPU phases improving (attention 99 to 55 ms, DeltaNet 210 to 169 ms); the GPU phases were unchanged.
2. NUMA rebalance (126/16 to 64/64 GB): no change; decode is ISA/latency-bound, not memory-bandwidth-bound.

The external comparison: llama.cpp does 7.3 tok/s on this exact hardware (Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf, -ngl 99 -ts 1,1,1,1) because it offloads the dense trunk to the GPUs too. So the M10s are not the limit; the un-offloaded dense path in qwen36 is.

[JustVugg]

Thank you, this is a valuable datapoint: the first Maxwell run in the repo, with the whole cuda-test suite passing on sm_50 and the CPU-vs-GPU greedy output byte-identical through a mixed tier. The 4-GPU spread halving both the MoE time and the peak RSS is exactly what the auto expert budget is meant to do.

Your bottleneck table points at the right place. Two notes:

1. dev took perf(sse41): add SSE4.1 fallback tier for olmoe and qwen36 int8 GEMV #1239 this morning: an SSE4.1 tier for the olmoe and qwen36 int8 GEMV, for hosts with SSE4.1 but no AVX2, which is your Sandy Bridge. The dense trunk kernels that today fall back to per-row/scalar paths on your build get a 128-bit SIMD body. If you can rebuild from dev (make -C c qwen36 CUDA=1 CUDA_ARCH=sm_50 ARCH=native) and rerun the 4-GPU warm row, the DeltaNet and attention numbers are the ones to watch.

2. The DeltaNet trunk and attention are not offloaded on qwen36 today (only lmhead and dnproj go through qt_trunk_offer), which your two controls confirm is the ceiling on this CPU. That is the open item this datapoint makes concrete, and it will be tracked as such.

I am keeping the issue open as a datapoint.

[JustVugg]

The dense trunk you measured as the ceiling is now offered to the VRAM placer in #1657: DeltaNet out_proj (dnout), the attention q/k/v/o (attnproj) and the shared expert (shexp), on top of lmhead and dnproj. About 1.9 GB of int8 in total on the 35B, which your four cards hold with every expert still resident.

I have no GPU here, so the engine side is verified on a fake CUDA backend (same bytes as the CPU GEMV) and the CPU path is unchanged; the measurement is yours if you are willing. Same command as your 4-GPU warm row, on the PR branch:

git fetch origin pull/1657/head:trunk-vram && git checkout trunk-vram
make -C c qwen36 CUDA=1 CUDA_ARCH=sm_50 ARCH=native
COLI_CUDA=1 COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto OMP_NUM_THREADS=12 COLI_TIMERS=1 \
  SNAP=... TOK=... N_NEW=200 HEAT_FILE=... ./qwen36 256 4 prompt.txt

What to look at: the [place] / [dense] lines at startup (what landed where), the per-phase table you already produce (DeltaNet and attention are the two numbers that should move), and COLI_PLACE=off as the control on the same build, with byte-identical output between the two. If the placer leaves something on the CPU for lack of room, COLI_PLACE=dnout=1,attnproj=2,shexp=3 pins the components to the cards you choose.

[ggeorgovassilis]

Thank you for the quick response and the guidance. I'd be grateful for any advice about improving throughput, especially prefill, and parallelism on the M10.

Results from the Tesla M10 box (issue #1652)

I ran the two tests you asked for on the same hardware. Everything below is on 4x Tesla M10
(compute 5.0, Maxwell, no tensor cores, 8 GB VRAM each) with a 2x Xeon E5-2640
(Sandy Bridge, no AVX2/FMA). Build line for all binaries:

make -C c qwen36 CUDA=1 CUDA_ARCH=sm_50 ARCH=native

Model is Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64, warm (persisted heat.bin),
200 tokens, same prompt throughout. All four runs produce byte-identical output
(md5 3f3d8e16dcfd6f84ceba5983c809fa33, 793 bytes).

PR #1657 (trunk-vram)

Same binary, only COLI_PLACE differs. Startup placement with auto:

[place] auto: dev 0 holds 485.9 MB of trunk (lmhead yes, 0 dnproj layers)
[place] auto: dev 1 holds 448.8 MB of trunk (lmhead no, 10 dnproj layers)
[dense] 190 trunk matrices on GPU (dnout/attnproj/shexp, 0.61 GB VRAM)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)

Experts are 10240/10240 resident with miss(CPU) 0 in both configs.

Phase (ms/token)	auto	off
DeltaNet	152.7	104.0
dn-sub: proj	106.0	65.7
dn-sub: conv	8.2	8.4
dn-sub: l2n+rec	9.7	8.4
dn-sub: norm+out	28.6	21.3
attention	38.8	26.1
moe_total	93.8	92.8
lm_head	68.8	41.7
Speed	2.68 tok/s	3.56 tok/s

dev branch (SSE4.1, #1239)

Metric	main	dev (auto)	dev (off)
Speed	2.45 tok/s	2.87 tok/s	3.54 tok/s
attention (ms/token)	55	25.9	26.2
DeltaNet total (ms/token)	169	145.7	105.2
dn-sub: norm+out (ms/token)	~49	21.9	21.5
moe_total (ms/token)	91	90.6	92.9

The main figure is from the earlier 12-core report on the same box and setup.

Summary

• With COLI_PLACE=auto the trunk timings are higher than with COLI_PLACE=off,
  so overall decode is slower: 2.68 tok/s against 3.56 tok/s, while MoE stays
  about the same. On this hardware the trunk offload makes single-stream decode
  slower.
• The SSE4.1 build measures faster than main on this box: 2.87 tok/s at auto and
  3.54 tok/s with COLI_PLACE=off, against 2.45 tok/s on main.
• The fastest config measured here is the SSE4.1 build with COLI_PLACE=off, at
  about 3.5 tok/s.

[JustVugg]

Thank you, this is the measurement that mattered, and it came back the wrong way round for the PR: on your box every placed component is slower on the GPU than on the CPU (lm_head 68.8 vs 41.7 ms, the 30 DeltaNet projections 106 vs 66, attention 38.8 vs 26.1), byte-identical output, decode 2.68 vs 3.56 tok/s. The placer prices a dense component by the bytes it saves on the CPU's memory bus, which presumes the GPU answers a GEMV faster than the CPU does. On the Quadro RTX 4000 and the 3070 of the calibration table it did (+34% and +49% warm); on four sm_50 Maxwell GPUs behind one shared PCIe board it does not, and a single-row int8 GEMV there is dominated by launch and transfer latency, not bandwidth. A prediction that is right on one card and wrong on another has to be measured on the card in front of it.

Pushed to #1657: before any trunk upload the engine now times one DeltaNet input projection both ways on the device that would host it (ten GEMVs, best of three, after a warm-up) and prints one line:

[place] probe: one [12288 x 2048] int8 GEMV takes 3.4xx ms on CUDA dev 1, 2.2xx ms on the CPU -> trunk stays on the CPU
[place] trunk stays on the CPU (measured slower than the CPU): 1.9x GB of VRAM back to the experts

If the GPU loses, the whole automatic placement is withdrawn (lm_head included) and its VRAM goes back to the experts before the warmstart. So on your box auto should now equal off, and on a card where the GPU wins the trunk still goes up. If you can run the same warm 200-token decode once more on the updated branch (git pull on trunk-vram), the two lines above plus the tok/s are what I need; COLI_TRUNK_PROBE=0 reproduces the previous behaviour if you want the A/B on the same build.

On your other questions, only what your numbers already say:

• Decode: with experts at 100% in VRAM and the trunk correctly on the CPU, the ceiling is the dense path on a CPU without AVX2. The SSE4.1 tier gave you 2.45 to 3.54 tok/s; the next lever there is the kernel itself (the SSE4.1 body is a first cut), not the GPU.
• Prefill: the prompt path is CPU-bound dense work here too (QWEN_DENSE_BATCH is on by default and batches rows across the dense-int8 weights). Your second-socket control (+22% from 6 to 12 threads, CPU phases only) says threads still pay; OMP_PLACES=cores OMP_PROC_BIND=close with all 24 logical CPUs is worth one run, and a prefill-phase breakdown (COLI_TIMERS=1 prints [timers] prefill:) would say where the rest goes.
• Parallelism across the four cards: the experts already spread over all four; the trunk is one sequential chain per token, so it cannot be split across cards without paying the transfer at every layer, which is exactly what your numbers show it cannot afford.

[rofl0r]

fyi i'm running qwen 3.6 35b q4_k_m on a gpu-less ryzen 5 (probably DDR4 ram) server with 6 cpu cores at 10-15 decode t/s using ik_llama.cpp. on high-end hardware like OP's i'd expect A LOT faster inference...

[ggeorgovassilis]

Tested the updated trunk-vram (the GEMV probe) on the same M10 box. Same warm 200-token decode as before, COLI_PLACE left at auto, byte-identical output throughout.

HEAD: 0e84b37 (qwen36: measure the trunk placement at startup, withdraw it when the GPU loses)

Probe ON (default — auto):

[place] probe: one [8192 x 2048] int8 GEMV takes 2.303 ms on CUDA dev 1, 0.860 ms on the CPU -> trunk stays on the CPU
[place] trunk stays on the CPU (measured slower than the CPU): 1.79 GB of VRAM back to the experts

Speed: 3.54 tok/s (56.5 s for 200 tokens).

Control (COLI_TRUNK_PROBE=0 — trusts the placer):

[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
[dense] 190 trunk matrices on GPU (dnout/attnproj/shexp, 0.61 GB VRAM)

Speed: 2.67 tok/s (74.9 s for 200 tokens).

So auto now equals off on this box, exactly as predicted: the probe measures the GPU GEMV slower than the CPU (2.303 vs 0.860 ms), withdraws the whole placement (1.79 GB back to the experts), and decode lands at 3.54 tok/s — the same as the earlier COLI_PLACE=off result (3.56). The withdrawn run is byte-identical to the placed run (md5 f8c12ea5…, 1316 bytes).

Per-phase (ms/token):

Phase	auto (probe ON)	probe OFF (placed)
deltanet	104.8	153.3
attention	26.2	39.2
moe_total	93.1	94.2
lm_head	41.9	69.1
TOTAL	266.1	355.9

The probe prices this card correctly — thank you for the fix.

[JustVugg]

@rofl0r thanks for the datapoint, and it is consistent with ours. On a CPU with AVX2 and the whole model resident in RAM, colibri decodes Qwen3.6-35B-A3B at 15.7 tok/s on 12 cores (measured in #1477, the shared expert kernel that shipped in 1.12.0), which is the same range as your 10 to 15 on 6 cores with a Q4_K_M. When the model fits in RAM the launcher keeps it there (coli plan on a 61 GB box picks full residency, cap 256/layer); streaming from disk is for models that do not fit.

This box is a different case, and that is why the numbers here look low: the two Xeons have no AVX2 or FMA, so the dense part of the token (DeltaNet, attention, lm_head), which runs on the CPU, takes scalar or SSE4.1 paths. That is where 3.5 tok/s comes from, not from the experts, which are all in VRAM here.

On the 7.3 tok/s of the other engine on the same box, two things make it not a like-for-like comparison, and both are ours to close:

1. Bytes. Its file is IQ3_XXS, about 3 bits per weight; our container is int4 with a group scale, about 4.5 bits. We read roughly 50% more bytes per token.
2. Where the dense part runs. There, on the GPUs; here, on the CPU without AVX2. And when we tried moving it to the GPUs on this box (the trunk placement above), our int8 GEMV ran at about 7 GB/s on Maxwell against an 80 GB/s card, so it lost to the CPU. The kernel is the problem, not the card.

What follows from this, in order: a bandwidth microbenchmark for GPU owners so the kernel gap is measured on Maxwell and on Ampere before any rewrite; a dense GEMV kernel that uses the bandwidth without dp4a; and reading GGUF files one to one, same quantized blocks, no requantization, which also makes the comparison above exact: same weights, same bytes, two engines.

[JustVugg]

Thank you, that closes the loop: the probe measured what your first run showed (2.303 ms on the M10 against 0.860 on the CPU for the same GEMV), withdrew the placement, and auto lands where off was, byte-identical. Merging #1657 with your two runs as the evidence; on a card where the GEMV wins the trunk still goes up, and the probe line will say so.

The 1.79 GB it hands back matters on smaller cards more than here. Next on this box, in the order I wrote above: the bandwidth microbenchmark, so the 7 GB/s of that kernel becomes a number per card rather than an inference from lm_head. I will ping you when it is ready to run; it will be a single binary and one line of output per format.