[Performance]: Vulkan (#418) vs ROCm/HIP on RDNA4 (RX 9070 XT) — CORRECTED: Vulkan is 19–24% faster (original comparison was confounded)

[noobdev-ph]

Important

CORRECTED — the conclusion below is wrong. My original comparison was confounded: COLI_CUDA=1 tripped the g_draft guard and disabled MTP on the HIP arms while the Vulkan arms drafted at 2.00 tokens/forward, so the two backends were not running the same decode configuration.

With speculation controlled on both, Vulkan is 19–24% faster than ROCm/HIP, not 32–35% slower. @steve-m's ~35% faster than ROCm on RDNA4 note is directionally correct and my challenge to it is withdrawn.

See the corrected 80-run results. The ReBAR finding and the correctness results below still stand; the performance comparison does not.

---

Independent measurements of the Vulkan backend from #418 on RDNA4, since @JustVugg mentioned not having a Vulkan GPU to verify against. Filing as an issue rather than a PR comment so the numbers are citable on their own and the review thread stays focused.

Summary up front: the backend works correctly on RDNA4 with no ROCm installed, and produces exact output on a real fmt=4/gs=64 container. That's the headline and it's a good one — this is the only path that gets the engine onto a lot of hardware.

Two things I ran into that I can't explain on my own, and would value your read on: it needs Resizable BAR (and fails silently without it), and on this particular box it comes out 23–35% slower than ROCm/HIP at matched VRAM — which runs opposite to the ~35% faster than ROCm on RDNA4 note in c/Makefile:260. I'm assuming I've got something configured differently rather than that the note is wrong, so I've documented my full setup below and would happily re-run anything.

Commit

550df5c (branch of #418, rebased onto dce7012 / v1.1.0). One binary built with both backends — they are independent in the Makefile — so every arm below is the same executable and only environment variables differ.

Hardware and storage

• CPU: AMD Ryzen 7 5700G (8C/16T, Zen 3)
• RAM: 64 GB DDR4
• GPU: AMD Radeon RX 9070 XT (gfx1201, RDNA4), 16 GB VRAM, Resizable BAR enabled (16 GB host-visible)
• Storage: NVMe SSD, link capped at Gen3 x4 (~3.5 GB/s) — the 5700G's PCIe limitation, not the drive's
• Model: GLM-5.2 744B, fmt=4 grouped int4, gs=64, full container on the NVMe

This box is storage-bound. All backends here are gated by expert streaming rather than compute, so absolute throughput is low and these ratios may not transfer to a Gen4/Gen5 machine.

Software environment

• Linux 7.0.14-arch1-1
• Mesa 26.1.3 / RADV, Vulkan 1.4.348
• ROCm 6.x (hipcc, HIP_ARCH=gfx1201)
• glslc from shaderc

Build and benchmark commands

make -C c colibri HIP=1 HIP_ARCH=gfx1201 VK=1

# Vulkan arm (320 or 640)
env COLI_VULKAN=1 COLI_VK_EXPERTS=320 COLI_VK_SHADERS=$PWD/c/shaders/qmatmul.spv \
    SNAP=model_gs64 NGEN=40 TOPP=0.7 TEMP=0 \
    PROMPT='[gMASK]<sop><|user|>Explain in two sentences why the sky is blue.<|assistant|><think></think>' \
    ./c/colibri 320

# HIP arm (6 or 12 GB)
env HIP_VISIBLE_DEVICES=0 COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=6 CUDA_RELEASE_HOST=1 \
    SNAP=model_gs64 NGEN=40 TOPP=0.7 TEMP=0 \
    PROMPT='[gMASK]<sop><|user|>Explain in two sentences why the sky is blue.<|assistant|><think></think>' \
    ./c/colibri 320

Methodology

• 50 runs total, 10 per arm.
• Arms interleaved (VK6, HIP6, VK12, HIP12, repeat) so page-cache warmth and thermal drift affect all arms equally.
• .coli_usage frozen: the same snapshot is restored before every run, so the "top-N of history" hot-expert ranking is identical in all 50 runs. Without this the learning cache drifts and silently changes what each arm holds.
• Fresh process per run.
• Warm-up policy: the frozen snapshot carries prior chat history; no run in the series is cold-start, and every run sees the identical history.
• Machine quiesced: llama-server stopped (it auto-restarts as a user service and holds ~15 GB VRAM), Firefox killed, and a MemAvailable >= 48 GB gate before each run — the engine sizes its RAM tier from MemAvailable at startup, so an unsettled machine silently produces a smaller tier.
• Throughput computed from decode_s, not the printed tok/s. The printed value rounds to 2 decimals; at these speeds that turns a true +2.35% into an apparent +4.2%.

Results

1. Resizable BAR is required, and its absence fails silently

pick_memtype() requires HOST_VISIBLE, and the tier wants HOST_VISIBLE|DEVICE_LOCAL. With ReBAR off, that type only exists in a 256 MB window, so the allocation falls back to plain host-visible system RAM — while still reporting success.

	ReBAR off (256 MB BAR)	ReBAR on (16 GB BAR)
tok/s	0.11	0.24
expert hit rate	22.3% (vk 5.7%)	38.8% (vk 13.2%)
GPU VRAM in use	82 MB	6,724 MB
expert tier upload	2.8–3.0 s	1.9 s

Same binary, same config; only the BIOS setting changed. In the ReBAR-off case the log reported 320 hot experts resident (6.04 GB VRAM) while the card sat at 82 MB and every access crossed PCIe — slower than the pure CPU path, with nothing in the output indicating the degraded mode. It took a sysfs check to identify.

Suggestion: compare the heap size in VkPhysicalDeviceMemoryProperties against the size of the host-visible+device-local type at init and warn, e.g.

[VK] warning: only 256 MB of 16304 MB VRAM is host-visible (Resizable BAR
     appears disabled) — expert tier will fall back to system RAM and be slow

(This comparison is n=1 per side — toggling ReBAR needs a BIOS change and reboot — but the 2.2× gap is far outside the ±2% run-to-run spread measured everywhere else.)

2. VRAM-matched Vulkan vs ROCm/HIP

10 runs per arm, medians, no overlap between arms:

arm	tok/s	hit rate	resident experts	tier size	VRAM peak
Vulkan COLI_VK_EXPERTS=320	0.2414	38.8%	320	6.04 GB	6,724 MB
HIP CUDA_EXPERT_GB=6	0.3189	57.3%	282	5.99 GB	6,215 MB
Vulkan COLI_VK_EXPERTS=640	0.2475	39.6%	640	12.08 GB	13,124 MB
HIP CUDA_EXPERT_GB=12	0.3331	59.9%	565	12.00 GB	12,160 MB

HIP +32.1% at ~6 GB, +34.6% at ~12 GB. Note Vulkan holds more resident experts in both pairs (320 vs 282, 640 vs 565) and still loses, so this is not a residency-budget artifact.

Note on buckets: Vulkan reports its VRAM tier as a separate vk bucket; HIP with CUDA_RELEASE_HOST=1 folds its VRAM prefix into pin (npin += prefix_est). Total hit rate and tok/s are comparable across backends; the per-bucket split is not.

3. Control: CUDA_RELEASE_HOST does not explain the gap

CUDA_RELEASE_HOST defaults to (g_cuda_ndev>1), i.e. 0 on a single-GPU box, so I tested whether an untuned ROCm baseline would account for the discrepancy. It does not — the default still builds the full 565-expert tier:

~12 GB budget	tok/s	hit rate	resident
Vulkan (640)	0.2475	39.6%	640
HIP, CUDA_RELEASE_HOST unset (default 0)	0.3051	54.7%	565
HIP, CUDA_RELEASE_HOST=1	0.3331	59.9%	565

HIP leads by +23% even untuned; the flag adds a further ~9%.

4. experts loaded/token stays flat with the tier resident — is this expected?

Identical in every run of every arm:

arm	experts loaded/token	per-layer (baseline topk=8)
Vulkan 320	677.0	9.03
Vulkan 640	674.5	8.99
HIP (all three configs)	330.1	4.40

Vulkan is still streaming ~9 experts per layer per token with 320–640 experts resident in VRAM — roughly the topk=8 baseline, i.e. the tier does not appear to be preventing the host-side load. HIP halves it, consistently, in every configuration including the default.

This is also why doubling the Vulkan tier changes so little (see below): if the tier were short-circuiting the load path as described in #418 ("no RAM slot, no disk read, no prefetch"), this number should fall roughly in proportion to residency.

I don't know the backend internals well enough to say whether this is a genuine gap or specific to this setup — this box's RAM pin tier is auto-capped to 10 experts (cap lowered 320->10 (projected peak 50.7 GB)), so it leans unusually hard on streaming.

5. COLI_VK_EXPERTS 320 → 640: +2.4% for 2× VRAM

Separate 20-run interleaved A/B (same controls):

	320	640	delta
tok/s	0.2417	0.2474	+2.35%
hit rate	39.0%	39.9%	+0.9 pp
— vk	13.2%	19.9%	+6.7 pp
— pin	9.1%	2.3%	−6.8 pp
VRAM peak	6,724 MB	13,124 MB	1.95×
tier upload	1.9 s	3.7 s	+1.8 s

Real and cleanly separated (all 10 doubled runs beat all 10 default runs), but small. The vk gain (+6.7 pp) is almost exactly offset by the pin loss (−6.8 pp): the extra 320 experts largely re-serve experts already held in RAM rather than converting cold reads. The 320 default looks well-chosen.

Correctness

50/50 runs produced clean, on-topic output, byte-identical across every configuration and both backends:

"Sunlight is composed of all colors, but it's scattered in all directions by gases and particles in Earth's atmosphere. Blue light is scattered more than other colors because it travels as shorter, smaller waves…"

This is a genuine fmt=4 / gs=64 conversion of the full 744B checkpoint, so it also exercises the #298 per-group scale semantics on Vulkan against a real quantized container — no garbage, no drift.

Minor

COLI_VK_SHADERS unset falls back to "shaders/qmatmul.spv" (colibri.c:6955), relative to CWD, so it only resolves when launched from c/. Resolving relative to the binary's directory — or a docs mention — would smooth the first run. It also expects the full path to the .spv file, not the shader directory.

What would you change about this setup?

Genuinely asking — I'd rather fix the experiment than argue about the result, and a lot of these knobs I chose by reading the source rather than from experience with the engine. Specific things I'm unsure about:

• Did I pick a fair ROCm baseline? I used CUDA_EXPERT_GB + CUDA_RELEASE_HOST=1 as the closest analogue to Vulkan's resident tier, and also ran the untuned default as a control. If there's a more representative HIP configuration, I'll re-run against it.
• Is 40 tokens too short? Startup effects could be over-weighted. A longer generation (512/1024 tokens) is easy to run if that's more meaningful.
• Should COLI_VK_DENSE / COLI_VK_ATTN be part of the comparison? I only exercised the expert tier, so this may be measuring the least favourable slice of the backend.
• Is freezing .coli_usage the right control? It makes runs reproducible, but if the tier is meant to co-adapt with a live learning cache, freezing it may understate Vulkan specifically.
• Would a different container help? This is fmt=4/gs=64; happy to test fmt=2 or another quantization.

And the open one: what hardware and configuration produced the ~35% faster than ROCm on RDNA4 for int4 experts measurement in c/Makefile:260? A different RDNA4 SKU, container format, generation length, or the dense/attention tiers enabled would all be plausible explanations, and knowing the intended setup would let me reproduce it directly rather than guess. The card and the full 744B checkpoint are set up here, so re-running is cheap — just say what would be most useful.

Caveats

• One machine, storage-bound (Gen3-capped NVMe on a 5700G). Every backend here is gated by expert streaming, not compute.
• Only the expert tier was exercised; COLI_VK_DENSE and COLI_VK_ATTN were not enabled.
• The ReBAR comparison is n=1 per side (BIOS toggle + reboot); everything else is a 10-run median.
• None of this affects the portability case for Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) #418, which works as advertised: correct output on RDNA4 with no vendor stack installed.

[JustVugg]

This is the most rigorous third-party benchmark this project has received. Same binary across arms so only environment differs, VRAM-matched pairs rather than "default vs default", 10 runs per arm with medians and no overlap between distributions, a control for the one flag that could have explained the gap, and an A/B on the tier size. You also filed it where the numbers stay citable instead of burying them in a review thread. Thank you — and specifically thank you for measuring a backend the author and I both cannot test.

On your question 4 — no, that is not expected, and I think you have found the actual defect rather than a configuration mistake on your side.

experts loaded/token counts host-side expert loads. If a VRAM tier is doing its job, an expert served from VRAM must not be loaded from RAM or disk at all, so the number has to fall roughly in proportion to residency. On HIP it does: 4.40 per layer against a topk=8 baseline. On Vulkan it sits at ~9.0 per layer — the baseline — with 320 and again with 640 experts resident. Doubling residency moves it from 9.03 to 8.99. That is not a tier that is helping a little; that is a tier that is not on the critical path at all.

Everything else you measured follows from that single fact and stops being mysterious:

• Why doubling the tier buys 2.4% — of course it does, if the tier does not prevent loads. The gain you see is compute-side only.
• Why the vk hit-rate gain is cancelled by the pin loss (+6.7pp against −6.8pp) — the extra experts are re-serving work RAM already served. The buckets are reshuffling, not shrinking.
• Why HIP wins with fewer resident experts (282 vs 320, 565 vs 640). It is not winning on shader throughput; it is winning because its tier genuinely short-circuits the host load and Vulkan's apparently uploads to VRAM while the fetch path still goes to RAM or disk anyway.

So the hypothesis I would test first, @steve-m: the Vulkan path is being used as a compute accelerator rather than as a cache tier — the upload happens, but whatever residency check the expert-fetch path consults does not know about it, so the load proceeds regardless. That is consistent with every number here and it is a bug in the load path, not in the shaders (your correctness results say the shaders are right). If that is it, the fix should move this from 23–35% behind ROCm to competitive, on a storage-bound box, without touching a line of GLSL.

On ReBAR: the warning should ship, and I would go further. A backend that reports 320 hot experts resident (6.04 GB VRAM) while the card holds 82 MB and every access crosses PCIe is not slow, it is lying, and it ends up slower than the CPU path with nothing in the output to say so. This project's rule elsewhere is refuse-or-warn rather than silently degrade — an untrusted container that does not add up gets refused rather than guessed at. Same discipline applies here. Your VkPhysicalDeviceMemoryProperties heap comparison is the right check, and I would make the message say what to do about it (enable Resizable BAR in firmware), since the person seeing it will not know the term.

On the ~35% faster than ROCm on RDNA4 note in the Makefile: you are not misconfigured, and the note has to go or be qualified. You measured the opposite on the same GPU family with a documented setup. Neither result needs to be wrong — yours is storage-bound with a Gen3 x4 link and the tier not short-circuiting; the original may have been compute-bound or measured before the residency behaviour above. But an unqualified performance claim in the source that a careful third party cannot reproduce is worse than no claim. @steve-m, could you say what workload and hardware produced the 35%? If it was compute-bound the note should say so; if it predates the current tier behaviour it should come out until it can be re-measured.

None of this diminishes the headline you led with, which stands and matters: the backend is correct on RDNA4 with no ROCm installed, on a real fmt=4/gs=64 container. That is the path onto a lot of hardware ROCm will never reach, and correctness is the hard part. The performance gap now has a specific, testable cause instead of a shrug.

If you are willing to run one more thing: the same VRAM-matched pair with the CPU expert cache made deliberately tiny (so RAM cannot mask disk), to see whether Vulkan's experts loaded/token stays pinned at ~9 even then. That would close the argument beyond doubt.

[JustVugg]

I have to correct myself, and the correction matters more than what I got wrong.

In my previous comment I told you that you had found the actual defect, and I proposed that the Vulkan tier uploads experts to VRAM while the fetch path ignores residency. A code review of the branch shows that hypothesis is wrong. The residency short-circuit exists, in moe()'s expert classification loop, and it fires before the pin scan, the LRU scan and the miss path:

if(vk_active && layer < c->n_layers){
    ColiVkTensor **rg = vk_reg_at(layer, eid);
    if(rg && rg[0]){ vk_hit[j]=1; m->hits++; m->hit_vk++; continue; }   // <-- short-circuit
}

A registry-resident expert never enters missk[], so the load path is never called for it. The same guard is mirrored in every prefetch path (next-block readahead, pilot_realload, pilot_uring_batch, couple_prefetch, pilot_prefetch). The tier is genuinely on the load path. I owed you a code check before endorsing a mechanism, and I gave you a plausible story instead. Sorry.

What actually produced your numbers — and it's a real finding

experts loaded/token cannot show what either of us used it to show. The counter behind it (m->ereq) is incremented in the router, per position, before any tier is consulted. No cache tier — pin, LRU, CUDA VRAM or Vulkan VRAM — can move it. It is a routing counter with a misleading name, and I'll get that name fixed.

The 2.05× ratio between your arms is speculation being on in one and off in the other, in the same binary. You left DRAFT unset, so this runs:

#ifdef COLI_CUDA
    int cuda_mtp = getenv("COLI_CUDA_MTP") ? atoi(getenv("COLI_CUDA_MTP")) : 0;
    g_draft = (m.has_mtp && (!g_cuda_enabled || cuda_mtp)) ? 3 : 0;   // MTP disabled under CUDA/HIP
#else
    g_draft = m.has_mtp ? 3 : 0;
#endif

Your HIP arm sets COLI_CUDA=1 → g_draft=0, one position per forward. Your Vulkan arm leaves it unset → g_draft=3, MTP drafting, ~2 positions per forward. 677.0 vs 330.1 is exactly that factor. The 50-run VK-vs-HIP comparison is confounded and should be re-run with DRAFT=0 on both arms (or COLI_CUDA_MTP=1 on the HIP arm).

But your instinct was right — there IS a defect, and it's arguably worse

Look at what that ratio implies: ~2.05 routed positions per emitted token on the Vulkan arm means MTP acceptance is essentially 0%. That is the exact failure mode the CUDA guard exists to prevent (#163): cold streamed experts compute on the CPU while resident experts compute on the GPU, the floating-point accumulation orders differ between draft and verify, and acceptance collapses. The Vulkan expert tier creates the identical split, and #418 does not extend the guard — g_draft still only consults g_cuda_enabled.

On a storage-bound box a rejected draft still pays full expert I/O, so this is pure loss. It is plausibly most of the 23–35% deficit you measured — and it means the honest comparison, once you re-run it, may look quite different.

So: you did find a real bug. Not the one either of us named, and one that only surfaced because your setup was rigorous enough that a confound this specific could be traced at all. That is worth more than a correct guess.

@steve-m — this is the substantive item on #418 from my side: either g_draft accounts for an active Vulkan expert tier the way it accounts for CUDA, or measured MTP-acceptance data showing it doesn't collapse. Two smaller ones: the ~35% faster than ROCm on RDNA4 claim in c/Makefile and README.md is stated as unqualified fact and a careful third party measured the opposite — please qualify it with the workload and hardware or drop it until re-measured; and @noobdev-ph's ReBAR warning should ship, since pick_memtype() never compares the chosen memory type against the heap size, which is how the 256 MB-BAR case silently lands in system RAM while the log says 320 hot experts resident (6.04 GB VRAM).

And the headline you led with stands untouched: the backend is correct on RDNA4 with no ROCm installed, on a real fmt=4/gs=64 container. That was always the hard part.

[noobdev-ph]

Thank you for both comments, and especially for the second one — going back to read the code and retracting a mechanism you'd already endorsed publicly is not a small thing, and it made the picture much clearer.

You're right about the confound, and I want to own my half of it. I presented experts loaded/token as evidence the tier wasn't on the critical path. It can't show that: m->ereq += Ke happens in the router (colibri.c:3069, inside the top-p selection loop) before any tier is consulted, exactly as you say. Normalized per position rather than per emitted token, my two arms come out at 338.5 vs 330.1 — essentially identical. My "finding" was the speculation difference wearing a disguise, and I should have checked what the counter actually counted before building an argument on it.

So the 50-run comparison is confounded and is being re-run. That part stands.

One correction before anyone writes code — @steve-m, please hold off

The inference that ~2.05 positions per emitted token implies ~0% MTP acceptance is inverted, and our existing logs already report the acceptance directly. Every Vulkan run printed it:

speculation: 2.00 tokens/forward (20 forwards per 40 tokens) | MTP acceptance 33% (20/60)

Identical in 10/10 runs at 320 experts and 10/10 at 640. The arithmetic, with g_draft=3 (1 real + 3 drafted positions per forward):

acceptance	positions per emitted token
0%	4.0
33%	2.0 ← what we measure
100%	1.0

0% acceptance would have shown ~4 positions/token, not ~2. And the MTP acceptance 0% (0/0) on the HIP arms is zero proposals — drafting disabled by the guard — not collapsed acceptance.

For reference the [MTP] auto-pause at colibri.c:4910 fires when acceptance drops below 10% over 24 proposals. It never fired in any of the 20 Vulkan runs.

So on this hardware, MTP acceptance does not collapse under an active Vulkan expert tier — 33% is in the normal band for the int8 MTP head (we've historically measured ~40% on CPU here). That's the "measured MTP-acceptance data showing it doesn't collapse" you asked for, and on this evidence extending the #163 guard to Vulkan would cost real throughput for no benefit.

I'd treat that as preliminary until the clean re-run confirms it, which is exactly why I'd suggest not building anything yet. If it holds, the g_draft item on #418 dissolves.

What we're running now

All phases, ~4 hours total, same controls as before (interleaved arms, .coli_usage frozen to one snapshot, fresh process per run, 10 runs per arm, throughput from decode_s rather than the rounded tok/s):

1. Clean backend comparison — the four VRAM-matched arms with DRAFT=0 forced on both backends, so speculation is off everywhere. The HIP arms should reproduce their earlier numbers, which doubles as a drift check on the rig.
2. MTP on both — VK and HIP at ~12 GB with DRAFT=3 and COLI_CUDA_MTP=1, to measure directly what speculation is worth per backend. At 33% acceptance MTP costs ~2 positions of expert I/O per emitted token, which on a storage-bound box may well be a net loss — relevant to whether the guard is earning its keep on HIP here too.
3. The tiny-CPU-cache run you asked for, so residency can't be masked by RAM.

Version note: testing at 550df5c, which is still the head of #418. I checked v1.1.1 (released after the PR's last update) and it doesn't touch the comparison surface — the g_draft guard is byte-identical, the npin+=prefix_est tier logic is unchanged, backend_cuda.cu has no diff, and backend_vulkan.c exists only on the PR branch. Rebasing would mean benchmarking something that isn't the PR, so I've left it where it is; happy to re-run on a rebased branch if @steve-m pushes one.

Results here when they're in. Given that I got one call wrong already in this thread, I'd rather post numbers than another theory.

[steve-m]

@JustVugg @noobdev-ph — following the thread carefully. Two things from our side: the fixes you both flagged are on the branch, and since our box runs both backends we ran the clean comparison that's in flight here, plus the MTP-acceptance measurement you asked for directly.

First, credit: @JustVugg, re-reading the branch and retracting a mechanism you'd already endorsed is the right instinct and it landed on the real cause — the DRAFT split is exactly it. @noobdev-ph, your acceptance-arithmetic correction (2.0 positions/emitted-token = 33%, not 0%) is right, and it matches what we measure. Nothing below contradicts either of you; it's corroboration from a second box plus the data point @JustVugg asked us for.

Matched DRAFT=0 comparison (the re-run @noobdev-ph is doing — here it is from our box)

RX 9070 (RADV, Mesa 26.1.4), one binary with both backends, full stack on each (dense + attention + expert tier), DRAFT=0 forced on both arms, VRAM-matched (VK COLI_VK_EXPERTS=320 / HIP CUDA_EXPERT_GB=6 CUDA_RELEASE_HOST=1), DPM pinned high, .coli_usage frozen to one snapshot, 2-drive NVMe mirror, tg128, fresh process per run, interleaved:

tg128	r1	r2	mean	decode-attn	hit rate
Vulkan (full stack)	1.71	1.88	1.80	15.8 / 15.7 s	60.1 / 66.2%
HIP (full stack)	1.63	1.54	1.59	20.9 / 20.8 s	68.4 / 61.9%

Clean separation — both Vulkan runs beat both HIP runs, ~+13% mean. And the decomposition is the honest part: Vulkan wins on the attention core, not the expert tier. Its absorb-attention decode is ~15.7 s against HIP's ~20.9 s, and that margin carries the run even though HIP has the higher cache hit rate here (its 40% RAM pin vs our VK config's PIN_GB=0). Both arms are disk-bound — felt wait 24–30 s is the largest single term and the source of the r1/r2 spread — so this is a "close, VK ahead" result, not a landslide, exactly as one expects when streaming dominates. It's the mirror image of the confounded 23–35% HIP lead, which is the point: with the DRAFT confound removed the sign flips.

(This is a 2-run interleave, not your 10-run medians — treat it as an independent corroboration of the direction while your cleaner run lands, not a replacement for it. Storage here is a 2-drive Gen4 mirror rather than a single Gen3 x4 link, so absolute numbers won't transfer; the ratio should be more comparable.)

MTP under an active Vulkan tier — the data @JustVugg asked us for

Same box, VK full stack, DRAFT=3, tier active, 2 runs:

r1: speculation 1.60 tok/forward | MTP acceptance 20% (47/240) | decode 0.94 tok/s
r2: speculation 1.86 tok/forward | MTP acceptance 29% (59/207) | decode 1.08 tok/s

Two findings, and they cut in different directions — so I'll give you both rather than the convenient half.

Acceptance does not collapse (20–29%, [MTP] auto-pause at colibri.c:4910 never fired). That refutes the specific #163 mechanism for Vulkan: #163's collapse is a draft-vs-verify kernel divergence (the S==1 fused-pair kernel vs the S≥2 IDOT kernel give different FP-accumulation orders for the same expert), and the Vulkan coli_vk_expert_group runs the same shader for S==1 and S≤4, so a VK-resident expert computes bit-identically in draft and verify. @noobdev-ph's 33% and our 20–29% agree: no collapse.

But your underlying concern is dead right — MTP is a large net loss here: 0.94–1.08 tok/s with DRAFT=3 against 1.80 with DRAFT=0, ~44% slower, with experts/token more than doubling (772–895 vs 349). At 20–30% acceptance on a storage-bound box the rejected 70–80% of drafts each pay full expert I/O, and that swamps the win. So the outcome you were protecting against is real; it just isn't the acceptance-collapse mechanism.

Which makes the g_draft item a throughput-default question, not a numerical-guard one: extending the #163 guard as written (keyed on "acceptance will collapse") would be justifying the right default with the wrong reason — and it would also wrongly punish a hypothetical VK box where the experts are fully resident (no streaming, MTP could pay off). Our suggestion: make the VK path default DRAFT=0 when the tier is streaming-bound (RAM-capped, the case where the tier exists at all), documented as a throughput default with an override, rather than folding it into the numerical guard. We already force DRAFT=0 in our own launch config for exactly this reason. But this is a product call and you have a 10-run phase-2 ("MTP on both backends") landing from @noobdev-ph that measures it more carefully than our 2 runs — happy to implement whatever you two land on (guard extension, streaming-aware default, or docs-only). Not touching it until that data is in, per @noobdev-ph's request to hold.

The ~35% faster than ROCm note

You're right, and it's reworded in c/Makefile and README.md to say what was actually measured: the VRAM-resident int4 expert-group primitive, K=8 distinct experts, fused gate+up+silu→down, per-call including readback — 0.117 ms/expert (Vulkan) vs 0.179 ms/expert (the production ROCm coli_cuda_expert_group), RX 9070, both fully resident. That's a compute-bound micro-measurement; end-to-end on a storage-bound box it's noise, which is why your box and ours both see the tier's e2e contribution washed out by disk. The matched table above is the honest e2e number on our box: ~13%, not 35% — and it comes from the attention core, not the expert primitive.

Fixes shipped (on the rebased branch)

• ReBAR warning — init now compares the chosen memory type's heap against the largest DEVICE_LOCAL heap and warns, naming the fix (enable Resizable BAR in firmware), exactly the check @noobdev-ph sketched and @JustVugg endorsed. The driver-side GTT fallback is invisible at allocation time, so this init-time topology check is the one reliable catch. docs/vulkan.md states the requirement up front.
• Shader path — COLI_VK_SHADERS now accepts the .spv file or the shader directory; unset, it resolves shaders/ next to the binary (/proc/self/exe) before the CWD fallback, and the startup error names what it tried.
• One more prefetch path — the residency skip you enumerated (vk_reg_served before pin/LRU/miss, mirrored across the prefetch paths) now also covers the PREFETCH=1 speculative cross-layer readahead (layer_forward_rows), which was the one path without it. It's off by default (PREFETCH defaults to 0), so it affected nobody's numbers here — just completing the set for the opt-in case.

Rebased

Both branches now sit on e4b5bf3 (v1.1.1 + the Inkling merge); clean replay, diff scope still VK-only. @noobdev-ph — you noted the comparison surface is unchanged across v1.1.1, which matches what we see, so a re-run on the rebased head should reproduce; the one thing it adds for you is the ReBAR warning and the shader-path fix, if either is useful mid-series. Pushed to the PR branch.

[noobdev-ph]

Correction and apology: I got this backwards — Vulkan is faster

The headline of this issue is wrong, and I'm retracting it. With speculation properly controlled on both backends, Vulkan is 19–24% faster than ROCm/HIP on this hardware — not 32–35% slower as I reported.

@steve-m, I owe you a real apology. I publicly challenged a performance claim in your PR, on the basis of a benchmark I had misconfigured, and asked you to justify a number that was right all along. The ~35% faster than ROCm on RDNA4 note in c/Makefile:260 is directionally correct and I withdraw my challenge to it entirely. Having a careful-looking third party tell the maintainer your measurements couldn't be reproduced is a genuinely bad thing to have happen to your work, and it happened because I didn't check that both arms of my comparison were running the same decode configuration. That check would have taken one minute. I'm sorry.

@JustVugg, thank you for catching it, and for going back to the source and retracting your own first hypothesis rather than letting a plausible story stand. This thread would have ended with a wrong number in a citable issue if you hadn't.

The full re-run is below: 80 runs, all three phases you asked for, same controls as before (interleaved arms, .coli_usage frozen to one snapshot, fresh process per run, throughput from decode_s), tested at 550df5c = head of #418.

Phase 1 — clean backend comparison, DRAFT=0 forced on both

arm	n	tok/s	hit%	pin	lru	vk
Vulkan 320	10	0.3958	54.7%	11.1	19.4	24.3
HIP 6 GB	10	0.3200	57.3%	40.0	17.3	—
Vulkan 640	10	0.3975	54.7%	2.3	19.4	33.0
HIP 12 GB	10	0.3342	60.0%	44.7	15.3	—

Vulkan +23.7% at ~6 GB, +18.9% at ~12 GB — no overlap between arms in either pair.

Worth noting Vulkan wins with a lower hit rate (54.7% vs 60.0%), so this isn't a caching artifact — the shaders are genuinely doing more work per unit time.

Both arms confirm fw/tok 1.03 and acceptance 0%, i.e. speculation actually off everywhere. The HIP arms reproduced their previous numbers (0.3200/0.3342 vs 0.3189/0.3331), which also confirms the rig didn't drift between batteries.

What I got wrong, specifically

1. The headline comparison. Setting COLI_CUDA=1 on the HIP arms tripped the g_draft guard and disabled MTP there, while my Vulkan arms drafted at 2.00 tokens/forward. That single asymmetry produced the entire "HIP is faster" result. The engine printed MTP ACTIVE (draft=3) on one arm and draft=0 on the other, in every log I collected — I saw it, misattributed it to a difference in generation length, and moved on without checking. The evidence was in front of me the whole time.
2. experts loaded/token. I built a second argument on top of the first: that the Vulkan tier wasn't on the critical path. It's a router-side counter (m->ereq += Ke, colibri.c:3069) incremented before any tier is consulted, so it cannot show that. Per position the arms were always near-identical (338.5 vs 330.1). I should have read the counter's definition before drawing a conclusion from it, and it's the part I most regret, because it turned a bad number into a plausible-sounding mechanism that cost other people time to disprove.
3. A prediction I made in this thread. I said shrinking the CPU cache would widen the backend gap. It narrowed it (see Phase 3).

The common thread is that I was rigorous about the things I knew to control for — interleaving, frozen usage, fresh processes, medians over 10 runs — and that rigour made the output look trustworthy while an uncontrolled variable sat underneath it. Careful methodology around an unchecked assumption is worse than sloppy methodology, because it earns confidence it hasn't paid for. That's on me.

Phase 2 — what MTP actually costs (and it isn't a Vulkan problem)

	MTP off	MTP on	cost
Vulkan 640	0.3975	0.2387	−39.9%
HIP 12 GB	0.3342	0.2176	−34.9%

Both backends lose roughly equally, so this is a property of speculation on storage-bound hardware rather than anything backend-specific. Vulkan still leads under MTP, by +9.7% (no overlap).

The mechanism is arithmetic. With g_draft=3 each forward evaluates 4 positions; at acceptance a you emit 1+3a tokens, so:

positions per emitted token = 4 / (1 + 3a)

At our 33% that's 2.0, and measured ereq/token goes 330.1 → 674.5, a 2.04× match. Only at 100% acceptance is MTP I/O-neutral; below that it always routes more experts per emitted token.

So speculation trades fewer forward passes (saving dense compute) for more positions (costing expert streaming). On a compute-bound box that's a win. Here, where a Gen3-capped NVMe is the bottleneck and dense compute is nearly free, it's paying 2× the scarce resource to save on the abundant one. Hit rate also drops ~14pp with MTP on, because rejected drafts load experts the real token stream never wanted and evict ones it did — so the real disk traffic grows by more than the 2× routing factor.

This may reframe the #163 guard. Acceptance does not collapse under an active Vulkan tier — it's a healthy 33%, stable across all 20 runs, and the [MTP] auto-pause at colibri.c:4910 never fired once. But MTP is still a ~35–40% loss here, on both backends. If that generalises, the useful predicate isn't "is a GPU tier active" but "is expert I/O the bottleneck" — which would cover Vulkan without a special case, and might also mean the guard is currently helping CUDA users for a reason other than the one documented.

Phase 3 — the tiny-cache run you asked for

cap=16 (per-layer LRU cut from 320), DRAFT=0, ~12 GB tiers:

arm	tok/s	vs cap=320
Vulkan 640	0.3741	−5.9%
HIP 12 GB	0.3429	+2.6%

Vulkan +9.2% over HIP, no overlap (after excluding one slow outlier).

This answers your question: RAM was not masking residency. Cutting the LRU by 20× barely moved either backend, because on this box the RAM pin tier was already auto-capped to 10 experts — the VRAM tiers were doing essentially all the work in both cases. It also shrank rather than widened the gap, contrary to what I predicted.

Data quality notes

• Three arms contained a single slow outlier each; medians are robust and conclusions hold after trimming.
• HIP12mtp was bimodal — five runs at ~0.175, five at ~0.22, a clean step at run 12. It's environmental, not algorithmic: acceptance (33%), fw/tok (2.00), ereq (674.2) and hit rate are identical across all ten, so the same work took different wall time. Both arms step up together, consistent with MTP's doubled working set warming the page cache. Phase 2 figures above use the warm subset (run ≥ 12) for both arms; the un-trimmed medians give −42.0%/−40.8% and the same ordering.
• v1.1.1 checked and deliberately not merged into the test branch: the g_draft guard is byte-identical, npin+=prefix_est unchanged, backend_cuda.cu has no diff, and backend_vulkan.c exists only on the PR branch. Rebasing would have meant benchmarking something that isn't Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84) #418.

What still stands

• ReBAR is required and fails silently — unchanged, and if anything more important now that the backend is measurably the faster one. pick_memtype() never compares the chosen type against the heap size, so a 256 MB BAR lands the tier in system RAM while the log reports 320 hot experts resident (6.04 GB VRAM). Worth a warning that names the fix.
• Correctness — 50/50 clean outputs earlier, and all 80 runs here produced correct output on a real fmt=4/gs=64 container.
• The COLI_VK_SHADERS relative-path nit.

Raw CSVs and all 80 run logs are kept if anyone wants to check any of this, and I'm glad to re-run on a rebased branch or any configuration you'd find more representative — including anything designed to catch a mistake like the one above.

@steve-m, if it's any use: on this box your backend is the faster one at every VRAM budget I tested, it's correct on a real fmt=4/gs=64 container, and it needed no ROCm to get there. That was true before I filed this issue and I'm sorry my measurement said otherwise for a day. If you'd like a datapoint from this hardware for the PR — with whatever configuration you consider representative — I'm happy to produce it and to have you specify the setup this time.

[xxxajk]

I have a unique setup here, and can test with it.
Currently I use the Vulkan backend on llama-server to pair an old P40 with a 5700XT, and depending on the model, it's mostly actually usable for people who have patence.
I will try to find some time to get some kind of benchmark or whatever is needed, bit that is going to be a bit hampered because I do not have enough space to dedicate an NVME to it, therefore it will be skewed, however it will open up some sort of benchmark until I get an NVME to test with.
One thing that does speed up other llm engines is to avoid mmio, by using some of the slab as an additional cache. Turns out that when I do that for llama-server, I get up to a 5 time performance gain on the warmup, and double performance afterwards.
Just thought I would point that insight out. I understand that the streaming can come off of storage, but is the LRU using MMIO or not?
Great project btw. Looks very interesting.

[noobdev-ph]

@xxxajk — this is a great question and you called it correctly. Short answer: no, the LRU is not using MMIO, and the reason it isn't is exactly the effect you discovered in llama-server.

Even better, colibrì exposes both paths behind a single flag, so your hunch was directly measurable rather than something we had to argue about. I ran it. Your observation reproduces here, and it's worth about a quarter of throughput. Thanks for prompting it — I'd have assumed rather than measured.

What the expert path actually does

By default an expert is pulled with one coalescing pread into an owned slab (colibri.c:159) — no mapping, no page-fault path. That's the same shape as the optimisation you found in llama-server.

The mmap behaviour you're describing is the opt-in here, COLI_MMAP=1, and the source comment is explicit that it's modelled on llama.cpp (colibri.c:1352, original in Italian):

"COLI_MMAP=1: experts become views inside mmap of the safetensors files (no pread, no slab, no copy: the kernel's page cache IS the cache). The maps are registered with Metal (newBufferWithBytesNoCopy on file-backed pages, like llama.cpp), so the GPU reads the same bytes."

static int g_mmap=0; — off unless you ask for it.

Measured, since the flag makes it a one-line A/B

20 runs on the RX 9070 XT (Vulkan tier, 640 experts, DRAFT=0), arms interleaved and .coli_usage frozen so both see an identical hot-expert set:

	default (pread + slab)	COLI_MMAP=1
tok/s, median	0.349	0.286
RSS	40.8 GB	44.2 GB
expert tier upload	4.50 s	4.90 s
hit rate	55.2%	55.2%

Because this run was noisier than my earlier batteries I'm reporting it paired (arms alternate, so each pair is adjacent in time and sees the same machine state):

The slab path wins 9 of 10 pairs, median advantage +25.4%, sign test p = 0.021.

So your instinct transfers cleanly to this engine. The hit rate is identical in both arms, which is the satisfying part — it isn't a caching difference at all, it's the cost of the fault path plus the memory pressure that rides along with it. kswapd0 was visibly busy during the mmap runs, and RSS is 3.4 GB higher because the mapped file pages are resident; on a box already sitting at ~41 GB that's enough to start reclaiming. Which is a fairly direct mechanism for the thing you noticed.

What I have not tested, and won't claim

• Your "5× on the warmup" figure — untested here, neither confirmed nor denied. That's a cold-page-cache measurement and everything above is deliberately warm (frozen usage file, warm cache, fresh process per run). Doing it properly needs drop_caches between runs, which I'd have to set up separately. So please read my +25% as the steady-state number only — the analogue of your "double performance afterwards" — and not as any comment on your warmup result. If anything I'd expect warmup to favour your finding even more strongly, but I haven't earned the right to say that.
• This is one box, one model, one prompt shape. Ryzen 7 5700G, RX 9070 XT, Gen3-capped NVMe, GLM-5.2 744B at fmt=4/gs=64, 40-token greedy decode. Storage-bound, which is precisely the regime where an extra fault path hurts most — so 25% may well be an upper bound rather than a typical figure.
• The paired framing is doing real work here. This battery was noisier than my earlier ones (the slab arm ranged 0.269–0.412 against a tight 0.390–0.401 in an earlier run of the same config), so an unpaired comparison of the two ranges overlaps and proves nothing. Pairing is legitimate because the arms alternate and each pair sees the same machine state — but you should know the raw ranges overlap, rather than take a clean-looking p-value at face value.
• I did not isolate the mechanism. RSS and kswapd are consistent with memory pressure being the cause, but I measured throughput, not page faults. Treat the explanation as a plausible story and the 25% as the actual result.

On "using some of the slab as an additional cache"

That knob exists and is the first positional argument: ./colibri <cap> sets the per-layer LRU expert cache size. Worth knowing before you tune it, though — I measured this recently on the same box and it barely moved anything: cutting the LRU 20× (320 → 16) changed throughput by −5.9% on Vulkan and +2.6% on HIP. The reason is specific to my machine: with 64 GB the RAM pin tier gets auto-capped to 10 experts (cap lowered 320->10, projected peak 50.7 GB), so the VRAM tiers were doing essentially all the work and the RAM LRU was nearly decorative. On a box with a different RAM/VRAM balance that knob could matter a lot more, so it's worth sweeping on yours rather than trusting my number.

Please do run it — your setup is the interesting one

A P40 + 5700XT paired over Vulkan is a configuration nobody else in this thread can produce, and multi-GPU is a supported axis (COLI_GPUS). A Pascal card and an RDNA1 card cooperating through a vendor-neutral backend is close to the whole argument for #418, so honestly your box is a better demonstration of the point than mine is.

And please don't talk yourself out of it over the NVMe. For backend or flag comparisons both arms see the same storage, so the comparison stays perfectly valid even if absolute throughput is depressed — storage only undermines cross-machine absolute numbers, which nobody needs from you. "Slow but internally consistent" is a completely publishable benchmark.

Two things I'd save you from, both learned the hard way in this thread:

• Set DRAFT explicitly on every arm. g_draft resolves as (m.has_mtp && (!g_cuda_enabled || cuda_mtp)) ? 3 : 0, so passing COLI_CUDA=1 silently disables speculation on that arm only. I missed it, and it inverted my entire published result — see the correction above.
• Check Resizable BAR is enabled in firmware before trusting any Vulkan number. Without it the expert tier lands in system RAM while the log cheerfully reports it resident in VRAM. Worth verifying for the P40 in particular, since older cards and their BAR sizes are exactly where this bites.

If it'd help I'm happy to hand over the harness I've been using (interleaved arms, frozen .coli_usage, per-run parsing) so you don't have to rebuild that scaffolding — and I'd genuinely like to see the mmap number from a two-GPU box, since mine is a single-card result and yours would be the first indication of whether it generalises.

Thanks again for the pointer. It turned a shrug into an actual measurement, which is the good outcome.