[hardware wanted] Strix Halo / Ryzen AI MAX+: CPU + iGPU + NPU in Colibri — Windows and Linux notes

[Kenneth-Javier]

A thread for anyone running Colibri on AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, XDNA2 NPU), to compare notes on using the three processors together. This started as a side conversation with @huppiflupp on #1511, and moved here so it doesn't clutter that PR and so others with the same hardware can join in.

Prior context. NPU support was asked about before in #959, and the answer then was a clear no. XDNA is a graph-compiler target while Colibri decides at runtime which experts to move, and on this hardware the bottleneck is disk and memory bandwidth, not compute. Both points shaped #1261. Its kernels are compiled ahead of time for fixed shapes, but weights and activations are passed at runtime, and it only targets the shared expert, which is dense and runs for every token, so nothing depends on routing. The bandwidth point still stands: that is why #1261 makes no speed claim, and why power, not throughput, is the open question below.

Where things stand

iGPU (HIP, Windows). Native Windows HIP support for gfx1151 landed in #788.

NPU (XDNA2, Windows). #1261 (under review) adds an optional, explicitly requested lane for one operation family, the GLM shared expert's gate/up projections. It is deliberately small: explicit opt-in, qualified shapes only, and a fail-closed fallback to exactly what the normal path computes. It makes no speed claim. Raw operations on the NPU beat the CPU path on real GLM shapes, but at model level weight conversion and device setup eat most of that, and end to end it's roughly a wash so far.

NPU vs iGPU, whole-model decode. @huppiflupp measured the iGPU at about 4.5x the NPU's decode throughput on the same box (numbers and caveats here): Gemma4 via Lemonade rather than Colibri, and power not measured.

Linux. @huppiflupp has the NPU working under Linux with XRT 2.26.0 and the amdxdna driver, and published the setup: https://github.com/huppiflupp/strix-halo-npu-linux

The direction

The aim isn't NPU-only or GPU-only, but CPU + iGPU + NPU together, with each operation placed where it pays. Locally I've measured that one process can hold live XRT/XDNA2 and HIP/Radeon contexts at the same time, in either init order, and that HIP and XDNA work genuinely overlaps rather than serialising. Neither of those is a result about model speed yet. They only show the combination is mechanically possible.

Open questions

1. Power. Nobody has measured energy per token or per operation for NPU vs iGPU vs CPU on this hardware. Throughput alone says little about where the NPU belongs in a mixed placement.
2. Linux portability of the NPU lane. The helper in feat(xdna): optional explicit AMD XDNA2 lane for the GLM shared expert #1261 has a plain C ABI and is loaded at runtime, so the lane isn't tied to Windows by design. The open question is whether its .xclbin artifacts (whole-array BF16 GEMM kernels built for XDNA2 on Windows) load and run under XRT on Linux. Nobody has tried yet.
3. Placement. Which operations, at which batch sizes, are worth moving off the CPU, and onto which device?

If you have this hardware

Numbers are more useful with context: OS, driver and runtime versions, which backend, model and quantisation, what exactly was timed, and power if you can get it.

[huppiflupp]

First numbers from the Linux side. Machine: GMKtec EVO-X2, Ryzen AI Max+ 395, 128 GB, Nobara 44, kernel 7.2 with in-tree amdxdna, XRT 2.26.0, NPU fw 1.1.2.65, Mesa 26.2.1, ROCm 7.1. Full profile: https://github.com/huppiflupp/strix-halo-npu-linux/blob/main/HARDWARE.md

All numbers below are from the firmware's Balanced mode (85 W socket limit). A 120 W round follows.

How power is measured, and the limits of it

On Linux the NPU has no sensor of its own. xrt-smi examine -r platform returns -EOPNOTSUPP from the in-tree driver. RAPL package and the amdgpu PPT reading turn out to be the same socket figure (85.08 vs 84.99 W in a dry run). The RAPL core counter is not usable either: it reports ~5 W with 32 threads at full load. So everything here is socket power, sampled at 20 Hz. "Delta" means socket power minus an idle baseline of 9.7 W taken right before the run.

Q2: Linux portability of the #1261 lane

I ported the loader to Linux: dlopen of libcoli_xdna.so beside the executable, located via /proc/self/exe, with the same no-search contract. The helper only needed its export macro changed and builds against the XRT 2.26.0 headers as is. Registry, integrity gate and dispatch are untouched. The helper binds on Linux (AVAILABLE), and the run then stops exactly where it should, at the missing artifacts. Could you share the four qualified files (wa_F3_M{64,256}_K6144_N2048.xclbin and _insts.bin)? The hash gate will tell us whether I have the exact bytes. Then I'll run xdna_physical_probe for both buckets unchanged and add the NPU column below. The port is on a branch; I'm happy to push it wherever suits you, as a follow-up PR or as commits for #1261.

Q1a: the shared-expert gate/up pair (K=6144, N=2048, fmt=4 gs64), CPU vs iGPU

The bench is steady state with warm-up excluded, and it uses the engine's own matmul_qt and vk_matmul_qt. It cycles through 75 distinct weight pairs, one per layer. A single int4 pair is ~14 MB and would otherwise sit in the 64 MB L3. The CPU runs 32 threads; iGPU runs use 4 host threads.

M	CPU ms	CPU mJ/pair	iGPU (Vulkan) ms	iGPU mJ/pair
1	0.20	16.0	0.27	18.1
16	3.20	255	2.20	171
64	10.2	815	8.6	654
256	40.8	3076	33.7	2557

At M=1 the CPU wins. From M=16 up the iGPU is 17–33 % cheaper per operation. But the iGPU only reaches ~380 GFLOPS at M≥16, a small fraction of what gfx1151 can do. The reason is structural: qmatmul.comp is a batched GEMV. vkCmdDispatch(..., S, 1) gives every token its own workgroups, and each of them streams the full weight again, so at M=256 the weight is read 256 times. RADV on this chip exposes VK_KHR_cooperative_matrix, VK_NV_cooperative_matrix2 and VK_KHR_shader_integer_dot_product. A tiled int4 GEMM path for S>1, along the lines of llama.cpp's mul_mm/mul_mmq, should move the iGPU row for M≥16 by a large factor. That matters for Q3: the NPU is competing against a GPU kernel that is not yet trying.

Q1b: whole model, Gemma 4 E4B, NPU (FastFlowLM) vs iGPU (llama.cpp Vulkan)

512 tokens, three runs each, run-to-run spread under 3 %:

	NPU	iGPU
decode	12.1 tok/s	57.3 tok/s (4.7x)
socket power while generating	22.3 W	72.4 W
energy per token, absolute	1.92 J	1.28 J
energy per token above idle	1.08 J	1.10 J

The NPU does not save energy per token. Above idle the two cost the same. The iGPU draws three times the power and finishes 4.7x sooner. Including the rest of the machine, which keeps drawing power for the whole run, the iGPU is a third cheaper. What the NPU does have is low power: 22 W, quiet, and it leaves the iGPU free. Caveat: these are different runtimes and probably different quantisations of the same weights.

Memory bandwidth (spec: 256 GB/s, shared)

• iGPU (HIP, 3 × 2 GiB in GTT): 226 GB/s read, ~200 GB/s copy/triad, i.e. 88 %.
• CPU (STREAM-style, 32 threads): 108 GB/s, i.e. 42 %. The cores sit behind the two CCD links.

So on this part a bandwidth-bound decode belongs on the iGPU, and the CPU cannot use the whole memory system even when it is idle otherwise. A CPU+iGPU concurrent figure is still to come; my first attempt reported best-of rather than same-window numbers and is discarded.

Next

• the same series at 120 W
• the NPU column for Q1a, once the artifacts are here
• the concurrent bandwidth run, fixed

[huppiflupp]

The missing pieces from my first post: the NPU column for Q1a, the 120 W series and the concurrent bandwidth run. Same machine and method as before: whole-package power from the amdgpu PPT sensor, read 20 times per second (the NPU has no sensor of its own), an idle baseline taken right before each run, and energy reported above that baseline.

Q1a with the NPU (shared-expert gate/up, K=6144, N=2048, fmt=4 gs64)

The NPU rows use Kenneth's F3 artifacts through the Linux lane from #1746 (xdna_physical_probe PASS for both buckets right before the run). The bench calls coli_xdna_try_matmul, cycles through 75 distinct weight pairs and uses 4 host threads, 30 s per point. There were zero fallbacks to matmul_qt in every run. Idle baseline 9.2 W before, 8.4 W after. The CPU and iGPU columns are the ones from my first post (Balanced, 85 W).

ms per pair / mJ per pair above idle:

M	CPU	iGPU (Vulkan)	NPU	socket power, NPU run
1	0.20 / 16	0.27 / 18	5.97 / 89	24 W
16	3.20 / 255	2.20 / 171	5.97 / 89	24 W
64	10.2 / 815	8.6 / 654	5.99 / 94	25 W
256	40.8 / 3076	33.7 / 2557	11.8 / 168	23 W

• The NPU has two fixed buckets. Everything up to M=64 runs as M=64, with the extra rows padded, and 65–256 runs as M=256. That is why M=1, 16 and 64 all cost ~6 ms. About 3 ms per matmul is a fixed floor.
• From M=16 up the NPU is the cheapest device per operation. From M=64 up it is also the fastest. At M=256 it is 2.9x faster than the iGPU and uses ~15x less energy, at ~1090 GFLOPS and 23 W for the whole socket.
• At M=1 (decode) it loses clearly: 30x slower and 5x the energy of the CPU. The lane's job is batched prefill, not decode.
• With a single weight pair instead of 75, the numbers barely move (81 mJ at M=1, 157 mJ at M=256, same times).

Caveats:

• It is not the same arithmetic. The NPU computes BF16 against int4→f32 on CPU and iGPU. Oracle A in the probe reported 0 mismatches.
• The one-time BF16 preparation of each weight is excluded, as the warm-up touches every set once. It also costs memory: 75 prepared pairs are ~3.8 GB, 4x the int4 size.
• The NPU rows were measured on a different day from the CPU and iGPU rows. The NPU draws so little that the conditions hardly matter for it.
• The iGPU row is still the batched GEMV I described before. A tiled path for S>1 would narrow the gap at M≥16. It would not close a 15x energy gap.

120 W (Performance mode, set via ryzenadj), quiet machine

Idle 9.2 W before, 9.8 W after. ms per pair / mJ above idle / socket power:

	Balanced 85 W	Performance 120 W	speed
CPU M=1	0.202 / 16.0 / 89 W	0.184 / 19.6 / 116 W	+10 %
CPU M=16	3.20 / 255 / 90 W	2.33 / 244 / 114 W	+37 %
CPU M=64	10.2 / 815 / 90 W	9.15 / 1015 / 120 W	+12 %
CPU M=256	40.8 / 3076 / 85 W	36.0 / 3966 / 119 W	+13 %
iGPU M=1	0.266 / 18.1 / 78 W	0.265 / 18.2 / 78 W	0
iGPU M=16	2.20 / 171 / 88 W	2.15 / 202 / 103 W	+2 %
iGPU M=64	8.60 / 654 / 86 W	8.61 / 672 / 87 W	0
iGPU M=256	33.7 / 2557 / 86 W	34.2 / 2659 / 87 W	0

• 120 W only helps the CPU: 10–37 % faster for 20–30 % more energy per operation. M=16 is the exception, where the energy stays flat.
• The iGPU alone never draws more than ~87 W. The 85 W budget does not limit it, so the mode changes nothing for it.
• Gemma 4 E4B gives identical numbers in both modes: NPU 12.1 tok/s at 22 W, iGPU 57.3 tok/s at 72 W. Decode is bandwidth-bound.
• The NPU at 23–25 W does not come near either limit.

Concurrent bandwidth, same time window, mean values

• CPU alone ~106 GB/s, iGPU alone 226 GB/s.
• Running together, the iGPU keeps 180–188 GB/s and the CPU drops to 17–18 GB/s.
• The memory controller strongly favours the GPU, and together they reach ~200 of 256 GB/s.
• For placement this suggests that CPU work next to a busy iGPU starves on bandwidth. Batched expert compute next to a GPU-resident decode would sit better on the NPU, which reads only ~4 GB/s at M=256, than on the CPU. I have not measured NPU and iGPU running at the same time.

[huppiflupp]

Update to the post above: #1834 landed the same day, and it changes the picture. Here are the same measurements on gfx1151 against current dev (8001d05): the shared-expert gate/up pair, 75 distinct weight pairs, 4 host threads, Balanced 85 W, energy above idle.

The VK_TEST harness passes completely on this device (Radeon 8060S, RADV, Mesa 26.2.1), including both GEMMs for every format and the cooperative-matrix cases. RADV reports cooperative matrix, so the coop shader is used from S=2.

ms per pair / mJ per pair above idle:

M	CPU	iGPU, GEMV (before)	iGPU, #1834 coop	iGPU, #1834 fp32 GEMM	NPU
1	0.20 / 16	0.27 / 18	0.27 / 18 (GEMV)	0.27 / 19	5.97 / 89
16	3.20 / 255	2.20 / 171	0.73 / 49	1.00 / 74	5.97 / 89
64	10.2 / 815	8.6 / 654	1.37 / 96	1.29 / 97	5.99 / 94
256	40.8 / 3076	33.7 / 2557	2.67 / 201	3.39 / 256	11.8 / 168

• At M=256 the iGPU goes from 33.7 to 2.67 ms, 12.6x faster, ~4.8 TFLOP/s with the coop shader. The fp32 GEMM reaches 3.8 TFLOP/s.
• That reverses my conclusion from earlier today. The iGPU is now 4–8x faster than the NPU at every M ≥ 16. Per joule it wins clearly at M=16 (49 vs 89 mJ) and ties at M=64. Only at M=256 is the NPU still ~17 % cheaper (168 vs 201 mJ), at a quarter of the speed.
• So on Strix Halo, with this kernel, the NPU lane's remaining case is narrow. It saves a little energy on large batches when the iGPU is busy with something else, or it runs at 23 W instead of ~80 W when power, not time, is the limit.
• The coop shader beats the fp32 GEMM at M=16 and M=256. At M=64 the fp32 GEMM is slightly faster (1.29 vs 1.37 ms). Both use the same energy there.

I have not measured end-to-end prefill yet. Is there an engine/model you'd like timed before and after on this box (e.g. Qwen3.6 with #1837)?

[huppiflupp]

Answering my own question from the update above: end to end on Qwen3.6, before and after #1834/#1837, same box (Radeon 8060S, RADV, Performance 120 W, everything else stopped).

The container is Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 with --cap 256 (all experts in RAM). The prompt is 583 tokens, followed by 64 new tokens, page cache warm, two runs each. Prompt speed is 583 / TTFT. Generation is the 63 tokens after the first.

	prefill	generation
CPU only, dev 8001d05	48.9 tok/s (TTFT 11.9 s)	29.6 tok/s
COLI_VULKAN=1, before #1834 (2b002f4)	22.9 tok/s (TTFT 25.5 s)	20.5 tok/s
COLI_VULKAN=1, dev 8001d05	48.5 tok/s (TTFT 12.0 s)	20.4 tok/s
same, COLI_VK_GEMM_MIN_S=0 (GEMV forced)	20.3 tok/s	20.9 tok/s

• Vulkan prefill doubles and lands on CPU parity, the same outcome as your 780M table.
• VK_PROF on the new build shows why it stops there. The prefill took 470 cooperative-matrix GEMMs in 0.73 s out of ~12 s. The rest is routed-expert work on the CPU, which the Vulkan path does not take for this engine.
• The ~18,300 GEMVs in the same report are the 64 decode tokens.
• Generation is unchanged by the PR, as expected. With Vulkan it stays below the CPU here (20.4 vs 29.6 tok/s), as it did before. This is the unified-memory effect from Experiment report: GLM-5.2 on AMD Strix Halo (Ryzen AI MAX+ 395, 128 GB unified) — 0.26-0.38 tok/s, CPU-bound not disk-bound, and the Vulkan expert tier double-counts unified memory #1502: the trunk on the iGPU shares the same memory controller with the CPU experts.

So on this APU the next lever for prefill would be the routed experts at S>1, not the dense trunk. The op-level numbers above show the iGPU kernel can carry them (0.73 ms for a gate/up pair at M=16, 2.7 ms at M=256). Whether that is worth a resident expert copy in GTT on a unified-memory part is a separate question. #1502 found the Vulkan expert tier double-counting memory there.

[huppiflupp]

A follow-up on the Vulkan numbers above. It is a proposal rather than a PR, because I can't measure the gain on my box right now.

What limits the iGPU at M>1. qmatmul.comp is a batched GEMV: coli_vk_matmul dispatches ((O+7)/8, S, 1), so every token row gets its own workgroups, and each of them streams the full weight row set again. At M=256 the shared-expert weight is read 256 times. That is why the iGPU sits at ~380 GFLOPS for M=16…256 in my table: it stays bandwidth-bound exactly as it is at M=1, while the matrix units stay idle. RADV on gfx1151 exposes VK_KHR_cooperative_matrix, VK_NV_cooperative_matrix2 and VK_KHR_shader_integer_dot_product (Mesa 26.2.1).

Proposal. Add a tiled path for S>1 alongside the current GEMV, which stays as it is for decode. Each workgroup would dequantise a weight tile (fmt 4 grouped int4, and presumably fmt 1/2/5) into shared memory once, then apply it to a tile of token rows through cooperative-matrix MMA or packed int8 dot products. llama.cpp's mul_mm.comp / mul_mmq.comp are the obvious template. The GEMV keeps S=1, and the host picks the tiled path when S ≥ some threshold, with correctness tests against matmul_qt as for the existing kernels.

What I'd expect it to buy (estimate, not measured).

• Decode: essentially nothing. At S=1 the op is bandwidth-bound; the CPU was even slightly ahead there. On this APU, GLM decode is SSD-bound anyway (1.29 tok/s tuned, Experiment report: GLM-5.2 on AMD Strix Halo (Ryzen AI MAX+ 395, 128 GB unified) — 0.26-0.38 tok/s, CPU-bound not disk-bound, and the Vulkan expert tier double-counts unified memory #1502).
• Prefill / TTFT: substantial. Rough split for GLM-5.2 per token: ~24B routed-expert parameters (CPU, streamed) and ~16B attention + shared + dense (iGPU with COLI_VK_DENSE/COLI_VK_ATTN). For a 1000-token prompt that is ~150 s CPU plus ~85 s iGPU today, so ~4 min. With the iGPU part at a few TFLOPS it becomes ~2.5 min, a 35–40 % shorter TTFT. Computing the routed experts batched on the GPU during prefill as well would push it towards the SSD floor, all experts read once: ~1.5 min.
• It should also help any discrete AMD/Intel Vulkan card, where the GPU tier is a win, and the S>1 verification step of MTP.

It also matters for the NPU question: a placement comparison is only fair against a GPU kernel that actually uses the matrix hardware. I'm not planning to write it myself right now; GLM is the only family on this path and doesn't fit this box usefully. I'm happy to benchmark and energy-measure a candidate on gfx1151 if someone picks it up.

[Kenneth-Javier]

Sorry for the slow reply. I held the artifacts back until #1261 landed, so what you verify against is upstream rather than a PR branch. It merged into dev last night.

The four qualified files are attached (colibri-xdna-artifacts-f3.zip, 17 KB). The registry they have to match is g_xdna_production_rows[] in c/backend_xdna.c on dev, so the hash gate will tell you straight away whether they arrived intact:

59309a70af9fd45ec5ed9467661332e08b9440276cf41afccecf439d34f0a4de  wa_F3_M64_K6144_N2048.xclbin
bde093ac7bcf2df39159376bb11f6f631be3f5fdf6dac6219ce177bd3c80fc97  wa_F3_M64_K6144_N2048_insts.bin
e49c34388efea654eb63d006c4024ffe0e0ff763d62c3f769c0362ac3bb17263  wa_F3_M256_K6144_N2048.xclbin
1a17753ce2516d8ccd7b992b61e1aa7c6b7a9fb655c4c0c0179eff075130d4b1  wa_F3_M256_K6144_N2048_insts.bin

They're whole-array BF16 GEMM kernels built for XDNA2 against the Windows XRT toolchain, so whether they load under XRT 2.26 on Linux is exactly the open question. For comparison, on Windows xdna_physical_probe reported 6 dispatches, 6 completions and 1 device open for both buckets, and above the range it declined with 0 dispatches.

On the port: now that the lane is in dev, it could go up as its own PR against dev, building on #1261. Whether it's taken is JustVugg's call, of course. One thing that would make it reviewable without NPU hardware: CI tests the lane against synthetic helper libraries, and test_xdna_execution / test_xdna_failure are Windows-only today only because the fake helper is a __declspec DLL. A fake .so would let those two run on the Linux CI leg too.

On your measurements: thank you, and for being explicit about what the socket figure can and can't tell us. Three things stood out:

• Energy per token answered the question I put at the top of this thread, and not in the NPU's favour. Above idle it costs the same as the iGPU, and the iGPU finishes 4.7x sooner. So the NPU's case comes down to low absolute power and running alongside a busy iGPU, not speed or energy per token. That's consistent with feat(xdna): optional explicit AMD XDNA2 lane for the GLM shared expert #1261 making no claim of either.
• The CPU vs iGPU table for the exact shape the lane handles is the most useful thing here for placement: the CPU wins at M=1, and the iGPU is 17–33% cheaper from M=16 up, even with a kernel that isn't using the matrix units. The NPU column you're planning for that table is the real head-to-head for feat(xdna): optional explicit AMD XDNA2 lane for the GLM shared expert #1261, so that's the result I'm most curious about.
• The bandwidth split, 108 GB/s for the CPU against 226 GB/s for the iGPU, says more about where decode belongs on this part than any kernel number.

Looking forward to the 120 W series and the concurrent bandwidth run.

The tiled Vulkan proposal deserves JustVugg's eyes, and it's buried in a hardware thread here. I checked your reading against the code: the dispatch really is one set of workgroups per token row, and there's no cooperative-matrix path anywhere. A 35–40% shorter time to first token, with decode unchanged, is a strong case, and your offer to benchmark and energy-measure a candidate on gfx1151 is exactly what someone picking it up would need. It might be worth its own Ideas discussion, since it affects every AMD/Intel Vulkan user, not only this box.

colibri-xdna-artifacts-f3.zip

[JustVugg]

Thank you both, this thread is exactly the kind of evidence that moves decisions.

On the NPU, to be clear about where I stand: when #959 asked, the answer was no, because what was on the table did not fit how colibri works. That was about that proposal, not about the hardware. If someone can make the NPU do useful work inside colibri, measured and fail-closed the way #1261 is, the answer is yes. #1261 is in dev now.

@huppiflupp, the Linux port: yes, as its own PR against dev, building on #1261. Two things would make it reviewable without an NPU on my side:

• a fake helper .so, so test_xdna_execution and test_xdna_failure run on the Linux CI leg the way they run against the fake DLL on Windows (Kenneth's suggestion, and I agree);
• the xdna_physical_probe output from your box in the PR body, for both buckets, plus the declined case above M=256 and the three failure cases (helper missing, artifact missing, one byte flipped).

The attached zip checks out: I hashed the four files and they match g_xdna_production_rows[] on dev exactly. Whether they load under XRT 2.26 on Linux is the open question, and your probe will answer it.

On the numbers: energy per token above idle is the same for the NPU and the iGPU (1.08 and 1.10 J), which settles the question at the top of the thread and is consistent with #1261 claiming neither speed nor energy. The NPU column in the shared-expert table (M = 1, 16, 64, 256) is the head-to-head I'd most like to see next, along with the 120 W series. The bandwidth split, 108 GB/s for the CPU against 226 GB/s for the iGPU, is the number that matters most on this part: a bandwidth-bound decode wants the iGPU.

The tiled Vulkan path for S>1: I checked your reading too. coli_vk_matmul dispatches ((O+7)/8, S, 1), one set of workgroups per token row, and there is no cooperative-matrix path anywhere in the tree. A tiled int4 path above a row threshold, next to the unchanged GEMV, with correctness tests against matmul_qt like the existing kernels, is welcome. The shorter time to first token is an estimate, as you say, so the PR should carry measured prefill before and after; your offer to benchmark and energy-measure a candidate on gfx1151 is what makes it doable. Agreed that it deserves its own Ideas thread, since it concerns every Vulkan user and not only this box.