[Feature]: NPU + iGPU build for Ryzen AI processors

[antonysv]

Problem

I've seen some bechmarks on these CPUs, I also built the project on Ryzen AI 365 notebook with 32Gb RAM, and a 1Tb nvme ssd (GLM 5.2). And when I ran "colibri chat" I saw in the task manager, that it is not using the NPU at all. So my question is how should I build the project so it uses not only the CPU but also the NPU and probably the integrated graphics. The second question: what best parameters/options would you propose for such config?

Proposed solution

Add the possibility to use NPU and iGPU.

Alternatives considered

No response

Scope and compatibility

No response

[JustVugg]

Straight answers to both questions, because they have different ones.

The NPU: no, and deliberately. The XDNA NPU is driven through graph compilers (ONNX Runtime / Vitis AI): you hand it a pre-compiled static graph, it executes it. colibri's execution model is the opposite — experts are chosen per token at runtime and streamed from disk, and the compute they need is a stream of small int4 GEMVs against buffers that did not exist a moment earlier. There is no sane path from that to an NPU graph today. But the deeper reason is measured, not architectural: on a machine like yours the bottleneck is disk and DDR bandwidth, not compute — the CPU already waits on bytes. The NPU would accelerate the part that is not the problem. If that balance ever flips (it might, with entropy-coded containers), this becomes worth revisiting.

The iGPU: partially yes, today. The Vulkan backend runs on RDNA iGPUs — try VK=1 and the expert tier will use your Radeon as a cache layer for hot experts. On Windows, AMD device discovery has a gap that #931 (in review) closes; on Linux it should already be found. The interesting property of your APU is unified memory — the same story as the GB10-class boxes in #933/#936, where zero-copy views beat explicit uploads. That work is in review for CUDA; a Vulkan analogue for APUs would be a natural follow-up.

Parameters for your box (32 GB, NVMe, GLM-5.2): the defaults are the right start — the RAM budget is sized automatically from available memory. Worth trying on top: VK=1 (iGPU tier), and if you want to squeeze the disk, check docs/ for the pin/prefetch knobs. What would genuinely help the project: post an iobench run and your tok/s with and without VK=1 as a [datapoint] issue — Ryzen AI machines are exactly the hardware class we have no rows for, hence the label.

[antonysv]

thanks for an expanded answer, my tests are in the issue 999

[antonysv]

one more thing. i think it should go to docs about vulkan. when i built it under windows there was an issue about missing headers(vulkan/vulkan.h). after installing the vulkan sdk. the solution was simple, the sdk sets its path in windows way (like C:\VulkanSDK\1.4.357.0) so gcc or make could not find it. changing the \ to / solved the problem. also in windows the library is vulkan-1, not just vulkan. here is a change i made in Makefile so it compiled fine:

VK=1 adds an opt-in Vulkan compute backend (any Vulkan 1.2 driver, e.g. Mesa/RADV)

for resident expert tensors. Independent of CUDA/HIP: works on GPUs ROCm doesn't

support (Polaris/RX 580); the VRAM-resident int4 expert-group PRIMITIVE measured ~35%

faster than the ROCm/HIP one on an RX 9070 (per-call incl. readback — end-to-end tok/s

depends on what else bounds the box; see docs/vulkan.md, benchmarking notes included).

Needs libvulkan + glslc (shaderc). Default build unchanged (all colibri.c hooks under

#ifdef COLI_VULKAN). The .comp shaders compile to .spv, loaded at runtime from

COLI_VK_SHADERS (default: alongside the binary / ./shaders).

VK ?= 0
VK_OBJ =
VK_SPV =
GLSLC ?= glslc
VULKAN_SDK=C:/VulkanSDK/1.4.357.0
ifeq ($(VK),1)
CFLAGS += -DCOLI_VULKAN -I$(VULKAN_SDK)/Include
LDFLAGS += -L$(VULKAN_SDK)/Lib -lvulkan-1
VK_OBJ = backend_vulkan.o
VK_SPV = shaders/qmatmul.spv shaders/qmatmul_gate_up.spv shaders/attention_absorb.spv shaders/rmsnorm.spv
endif

the VULKAN_SDK version and path may differ, so set it to your own.

[JustVugg]

Closing — answered definitively, and your measurements have moved to a better home.

On the NPU: no, and deliberately. XDNA is a graph-compiler target — you hand it a static graph ahead of time. colibrì's whole design is deciding at runtime which experts to move and when, based on what the router just did. Those two models are fundamentally incompatible, and pretending otherwise would produce a worse product with an NPU checkbox.

Your Ryzen AI numbers now live in #999, which is where the follow-up belongs.

One residual worth extracting before this closes: you noted that Windows builds fail on a missing vulkan/vulkan.h until the Vulkan SDK is installed. That's a real docs gap and it belongs in docs/vulkan.md — if you want to open a one-line docs PR for it, it'd help the next person on Windows.