RFC: Add Atlas Inference backend for native Windows/Linux ROCm inference via pure Rust

[AzeezIsh]

Proposal size

Major feature (spans multiple well-scoped PRs)

Updates

No response

User story

As an AMD user on Windows or Linux, I want to run local models through Lemonade using a native, pure-Rust ROCm backend (atlas-inference, https://github.com/Atlas-Inf/atlas), so that I get maximum inference throughput and deterministic latency without needing WSL2 or bloated Python/PyTorch runtimes.

Today this is painful because running modern LLMs on AMD hardware under native Windows has historically lacked stable, performant runtime support forcing users into WSL2 virtualization, heavy container setups, or runtimes with significant Windows performance and accuracy trade-offs. Running natively as the checkpoint intended, without Vulkan or fancy GGUFs.

High-level design

I propose adding atlas-inference as a first-class inference backend option in Lemonade:

1. The Core Backend (atlas-inf)

• Pure Rust + Direct ROCm/HIP: Built from the ground up in Rust without Python or PyTorch dependencies, talking directly to ROCm user-mode drivers on Windows and Linux.
• Native Windows First: First-class acceleration for AMD RDNA3/3.5 architecture delivering full performance and accuracy natively on Windows without virtualization.
• Speed & Determinism: High TTFT efficiency, low token latency, and reproducible outputs critical for multi-step agentic pipelines.
• External Validation: Backed by published results in MLPerf Inference v6.1 Edge/Agentic https://mlcommons.org/2026/07/mlperf-inference-v61-edge-agentic/ and enterprise testing by StorageReview https://www.storagereview.com, showing competitive parity with NVIDIA baselines.

2. Lemonade Integration Surface

Following Lemonade's standard backend pattern (adding-a-backend.md), integration requires one directory and three appends:

Descriptor (src/cpp/include/lemon/backends/atlas/atlas.h):

recipe: "atlas"
binary: "atlas-server"
default_device: DEVICE_GPU
slot_policy: SlotPolicy::Standard
selectable_backend: true (auto-exposes --atlas and atlas_backend in the CLI/config)
support: Configured for windows and linux on AMD GPUs.
supported_modes: {"chat"}

• Server Class (src/cpp/server/backends/atlas/atlas_server.{h,cpp}): Subclasses WrappedServer, orchestrating the lifecycle (load(), unload(), chat_completion()) over a local subprocess or IPC.

Wiring & Models:
• One-line append to CMakeLists.txt: LEMON_BACKENDS += "atlas|atlas"
• Version-pinned binary release in src/cpp/resources/backend_versions.json
• Supported model entries in src/cpp/resources/server_models.json

Breaking changes

None. This adds an opt-in backend alongside existing providers.

Maintenance plan

Binary Distribution: I can provide pre-built, version-pinned atlas-server binaries for Windows (x86_64) and Linux (x86_64) hosted on public releases.
CI & Testing: Add automated integration tests in CI verifying descriptor registration, BackendModeContractTest compliance, and mock/headless lifecycle testing.
Windows and Linux AMD hardware regression testing maintained by me. I can also maintain the adapter and bump backend_versions.json as new engine optimizations and model architectures are released.

Risks

Hardware & Driver Compatibility requires recent AMD ROCm/Adrenalin drivers on host machines. Lemonade will detect driver availability at startup and fall back gracefully with clear setup documentation if requirements are not met.

CC @jeremyfowers @Geramy

[Geramy]

I mean it looks good to me and is the first RUST engine we have. It could prove nice to adopt something RUST with high performance.
@jeremyfowers ? I know the rust people will love this, I am not speaking to the performance aspect that is another angle as well as windows stability. A lot of people do run WSL2 etc.... on windows.

[jeremyfowers]

The biggest benefit of this engine, compared to alternatives, that I can see is the sub-50MB native binary. halo-box/strix-llama is a 500 MB binary, while Gufo and Halogen are containers. But the nice thing about containers is that we have spec'd out a way to sandbox them here: #3697

@Geramy probably the biggest question I have here is security. We don't have any sandboxing in place yet. My thought was that we needed sandboxing before considering externally-maintained engine distributions. What is your analysis there?

[Geramy]

We either need sandboxing, or a security review in place where a single maintained item cannot be an attack vector.

[superm1]

🍋

[jeremyfowers]

The AGPL license will be challenging for us, we'd need to put this through AMD legal review.

@AzeezIsh in order to help motivate the legal review, can you give me some high-level quantitative claims? Using apples-to-apples numbers, why would someone use Atlas instead of llama.cpp (any backend), Donato's toolboxes, Halogen, and vLLM? Install size? Performance?

[AzeezIsh]

Thanks Jeremy! Hmm, I'll have to get back to you regarding the AGPL. My argument would be if it's externally maintained, and lemonade adds it as an abstract backend and I work the integrations as long as those are free to use, hopefully it's not an issue. Here are the key numbers. I'll ensure to show distinctly what's been peer-reviewed, and is my own measurement.

1. Audited: MLPerf Inference v6.1, Closed division, Edge Agentic (Qwen3.6-27B)

Atlas is the only AMD client/APU result anywhere in MLPerf Inference v6.1. Every other AMD entry in the 492 results is Instinct MI3xx. Atlas is also the only AMD result in the Edge Agentic category.

The Edge Agentic workload replays 1,007 agentic coding turns (20 SWE-bench Verified trajectories, prompts up to about 23K tokens) at a single stream. Its accuracy gate is BFCL v4, 995 samples, temperature 0.

MLCommons' reference implementation is stock llama.cpp with Qwen3.6-27B Q4_K_M, and its published reference run is on an NVIDIA Jetson AGX Thor.

	Hardware	Engine	Wall, 1,007 turns	Median TPOT	BFCL overall
MLCommons reference	Jetson AGX Thor	llama.cpp cfff1fc, Q4_K_M	9,420 s (2 h 37 m)	~91 ms	86.23
Atlas	Strix Halo (Ryzen AI Max+ 395 / 8060S)	Atlas, native HIP, NVFP4 + MTP	7,059 s	47.3 ms	86.23
Atlas	DGX Spark (GB10)	Atlas, CUDA, NVFP4 + MTP	3,808 s	n/a	87.24

Strix Halo with Atlas finishes the MLCommons agentic workload 1.33x faster than the MLCommons llama.cpp reference does on Jetson Thor. Per-token decode is about 1.9x faster, and accuracy is identical to the reference's 86.23. The same engine and quantization recipe also produced a closed-division result on NVIDIA GB10. No other engine in v6.1 has closed Edge Agentic results on both AMD and NVIDIA.

One caveat: our median TTFT on Strix (2.71 s) is slower than the Thor reference's (2.15 s). The overall win comes from decode.

2. Our measurements: same box, same model, Atlas vs llama.cpp barebones

Strix Halo (gfx1151), Qwen3.6-27B, single stream. Neither engine was put through MLPerf audit in this comparison. To be clear, the figures in this are also without MTP as a reference config.

	Atlas NVFP4 + MTP	llama.cpp Q4_K_M	llama.cpp ROCmFP4-lean
Decode	16.2–16.7 tok/s	10.9 tok/s	12.5 tok/s
Prefill (cold)	~178 tok/s	58.8 tok/s	—
BFCL v4 single-turn	89.82	88.60	86.06

At matched 4-bit quality, Atlas leads on decode, prefill (about 3x) and accuracy (+1.2). I haven't run many toolboxes, Halogen or vLLM on this workload, I'd like to refer to the numbers I have strictly measured. Again, these prefill numbers are considering using native NVFP4 dtype, which as you know isn't directly compatible and requires workarounds. I worked diligently to maintain accuracy with a bearable prefill speed here. The MLCommons harness takes any OpenAI-compatible endpoint, so a like-for-like run is straightforward.

3. Install and runtime footprint

• One 43 MB binary (Rust host plus our own HIP kernels). Nothing else is loaded at runtime: no Python, PyTorch, rocBLAS/hipBLAS or MIOpen. The process maps only libamdhip64, the HSA runtime and comgr from ROCm core, plus libc.
• Native HIP on gfx1151, with no SCALE or other translation layer at runtime
• For comparison, I believe a vLLM ROCm deployment brings a full PyTorch plus ROCm Python stack.

4. What this means for AMD client GPUs

The one audited datapoint showing a Ryzen AI Max+ 395 running an agentic workload came from Atlas: 1.33x the MLCommons reference on Jetson Thor, at identical accuracy. I'm also upstreaming the Strix Halo work into the wider ecosystem:

• gfx1151 support in HF kernel-builder (Kernel-builder drops for Strix Halo architecture huggingface/kernels#834, Apache-2.0);
• a Strix Halo Hub kernel for Qwen3.5/3.6/3.8 Gated DeltaNet, wired into transformers (in progress, following #46423). That kernel is AGPL-3.0 like the engine, and is loaded from the Hub at runtime, not vendored.

References:

• MLPerf Inference v6.1 results announcement, which names Atlas Inference as a first-time submitter and lists the AMD Ryzen AI Max+ 395: https://mlcommons.org/2026/09/mlperf-inference-v6-1-results/
• Edge Agentic call for submission (workload, reference implementation): https://mlcommons.org/2026/07/mlperf-inference-v61-edge-agentic/
• Reference implementation and the Thor reference run: https://github.com/mlcommons/endpoints/tree/main/examples/10_Edge_Agentic_Example
• Our submission, with source at the exact commits, harness config.yaml and seeds: https://github.com/mlcommons/submissions_inference_v6.1/tree/main/closed/Atlas_Inference
• StorageReview's v6.1 coverage ("The Ryzen AI Max+ 395 appears through Atlas Inference…"): https://www.storagereview.com/news/mlperf-inference-v6-1-5-7x-per-accelerator-gains-a-512-gpu-run-and-vera-rubins-first-peer-reviewed-numbers

[jeremyfowers]

Thanks for the analysis :) The 43 MB binary is definitely attractive. Besides the security question I posted above, the other thing I'll look for is people to vote this up to indicate that they want to use it.

In the meantime, something we could do is have @iswaryaalex add it to the "backend battleground" performance tracker. We probably need an OK from AMD legal first though, which would take 2+ weeks.

[AzeezIsh]

Sure thing @jeremyfowers, if there's anything I can do to help with the legal review, please let me know. I have signed an NDA previously on behalf of Atlas Inference before as well and worked with the team internally regarding compute/engineering efforts :)