v26.10.1 - Speed Across the Board - GGUF on Our MLX Engine - Sushi Flash Next EXL3 🍣

[ddalcu]

Highlights

• Faster across the board, with identical results. From Gemma 4 E4B to Qwen3.8 27B and Flash Next, models decode up to 66% faster and read prompts up to 51% faster, and the words that come out are exactly the same as in 26.9.6 (checked on 18 models: Gemma 4, Llama, Mistral, Qwen, Muse, LFM2.5, Ling, Bonsai and more). Measured against the shipped 26.9.6 in the same session, arms alternating, llmprobe taking every number; decode is the headline tokens per second, then the context rungs, prefill last:

  Model	Mac	Decode	Prefill
  Qwen3.8 27B 4-bit + drafter	M5 Ultra	+66%	+14%
  Qwen3.8 27B 4-bit + drafter	M4 Max	+28%	0%
  Qwen3.8 27B 4-bit + drafter	M1 Pro 32 GB	+37%	·
  Qwen3.8 Flash Next	M5 Ultra	0%	+51%
  Qwen3.6 35B-A3B	M5 Ultra	0%	+12%
  Qwen3.6 35B-A3B	M4 Max	-2%	+2%
  Gemma 4 26B-A4B 4-bit	M5 Ultra	+14%	+18%
  Gemma 4 E4B 4-bit	M5 Ultra	+19%	+6%
  Bonsai-2 27B 2-bit	M5 Ultra	+1%	+6%

  Differences within a few percent are run-to-run noise on speculative cells. The M1 Pro's 27B prefill has no 26.9.6 figure because 26.9.6 refused the long prompts (see Fixes). Bonsai-2's new tensor-unit verify route pays on quoted text (edits, rewrites, repeats): a verbatim-rewrite prompt decodes 48% faster on the M5 Ultra (185 to 273 tok/s), while novel-text decode is unchanged.

• MLX-Serve Skills When you launch Pi, you can now ask it to build you something using MLX-Serve API's, like build you a game with Audio and Images generated locally. Or a Laya / Kev Decision / Music generator Web App. (install pi so it shows up in launcher)

• Chats no longer freeze while another request reads a long prompt. Streams that are already answering keep moving instead of stalling for seconds behind a 32k-token prompt (feat: --prefill-decode-share keeps decoding streams moving through another request's prefill #568). Thanks @STRML.

• Qwen3.8 Flash Next is faster everywhere.

  • On every Apple Silicon chip, not just Ultra: speculative decoding gains 5-8% on an M4 Max (code 101.7 → 109.2 tok/s, chat 83.6 → 90.4).
  • After long prompts: past 32k tokens it is about 12% faster, long agent sessions stay cached instead of re-reading the whole conversation every turn, and several chats at once share the speedup.
  • Optional --ple-gpu (or the app toggle): prefill 2-4% faster and decode +15% at 128k. Off by default because it needs the whole table in memory (perf: serve bf16 KV with the fused split-K QSA verify kernel #555, perf: PLE n-gram gather on the GPU #539, perf: solo greedy MTP rounds draft the next chain before the host read #545, perf: gqa-aware causal sdpa split keeps MTP verify widths on the vector kernel #554, perf: one fused kernel for the QSA indexer's pooled-key upkeep #556, perf: with no --prefix-cache-mem, qwen4_exp's hot cache holds one full session #575, perf: batched decode rows take the fused QSA attention kernel at S=1 #580, perf: qwen4_exp decode ladder, flushing the deferred PLE leaf first #584). Thanks @STRML and @cowboycoderhq.

• Qwen3.8 27B is much faster on M5 Macs. On an M5 Ultra: 245 tok/s at 4-bit (was 197), 190 at 6-bit (was 134), 168 at 8-bit (was 127), with identical output.

• Nemotron-H reads prompts 9x faster (perf: Nemotron-H prefill 9x (fused Mamba2 step over the whole chunk) + MTP loader leak fix #574). Thanks @sbusso.

• Qwen-Image-2.1 can edit pictures from instructions. One model makes images and edits them: give it a photo plus up to 10 reference images (Qwen-Image-2.1 instruction editing (unified t2i+edit, up to 10 references) #512). Thanks @codysk.

• Every model gets a drafter slot, and Check for Updates. Add a drafter to any model in the app; a model's bundled drafter loads automatically; 27B 6-bit and 8-bit get the full-speed drafter too. My Models can check everything against Hugging Face and asks before replacing any files (feat: tensorfolding #590).

• GGUF models run on our own MLX engine. A GGUF no longer has to go through the slower llama.cpp fallback: Qwen 3.5, 3.6 and 3.8 (dense and MoE) and Gemma 4 GGUFs now load straight onto the MLX path through an engine we wrote ourselves in Zig and Metal, reading the GGUF tensors as they are, so they get the same prefix cache, batching, speculative decoding and tool-calling pipeline as every MLX pack. Turn it on under Settings > Engines (or --mlx-gguf); every other GGUF keeps the embedded engine it had.

• Sushi-format Qwen3.8 Flash Next packs run directly. Sushi's EXL3 expert packs load and serve like any other Flash Next pack, with no conversion, so the smallest Flash Next downloads work out of the box. Speculative drafting is now on by default for every model that supports it.

• Bonsai 2 gets an optional speed boost. A new "Int8 prefill" setting reads prompts faster on 2-bit Bonsai 2 at a small accuracy cost (off by default), and M5 Macs verify drafts faster too (feat(bonsai2): opt-in int8 prefill, per-model int8_prefill setting #619, perf(bonsai2): 2-bit tensor-unit verify route for Prism packs on M5 #615). Thanks @h9q2cyxvgm-ui.

• Kev joins Laya for quick decisions. /v1/decisions answers fast yes/no and multiple-choice questions (Kev packs on /v1/decisions #603). Thanks @alinselea.

• Simplified Chinese and a light theme. The console is translated, with a light mode, and so are the app's alerts and hints (i18n(console): translate the built-in console into Simplified Chinese #487, feat(console): add a light theme with a switch #485, i18n(app): route the alerts and hints through the catalog #463). Thanks @LXD-8.

• Fresher recommended models. MiMo 9B is the new small pick, Qwen 3.8 replaces 3.6 27B, and the tier-list pages get one-click Run buttons (feat: rec list matches ddalcu's catalog — MiMo 9B in, 3.6 27B out; tier-list rows get Run buttons #577). Thanks @Fe2-O3.

Fixes

• On a 32 GB Mac, Qwen3.8 27B with its drafter at a 16k context refused every request in 26.9.6 once its cache filled; 26.10.1 serves it, and a drafter that would not fit beside its model is now left out at load with a log line instead of starving the server.
• Broken model config files now give a clear error instead of loading a damaged model (fix: reject invalid model config field types and ranges #608). Thanks @alinselea.
• Fixed a long-standing cause of agents looping: lookup drafting could get stuck repeating itself, and it now drafts more tokens sooner after a shorter match (fix: prompt-lookup drafts take exact acceptance under --mtp-typical, so agents stop looping #614). Thanks @STRML.
• The SSD cache reports its disk use accurately (fix(disk-cache): keep the SSD tier's bill equal to the files it owns #601). Thanks @brandondyal.
• Fixed a rare crash on M1/M2 Macs during drafting.
• A DFlash2 drafter with an unsupported setting is refused with a message instead of misbehaving (fix(dflash): refuse a quantized DFlash2 selector codebook at load #454 #624). Thanks @brandondyal.
• A second server refuses a port that's already in use instead of silently sharing it (fix: a second server can no longer listen on a port already in use #569). Thanks @sbusso.
• Asking for a model by its path never returns another model's answer: a model that failed to load reports why, and an unknown path returns 404 (Failed model load silently falls back to the previously-loaded model — user chats with the wrong weights without any indication #585, fix(server): a path names its own model, never the default (#585) #607). Thanks @lborloz.
• Qwen-Image-2.1 packs with unusual quantization or mlx-community layouts (like mlx-community/Qwen-Image-2.1-MLX-4bit) now load, generate and edit correctly (fix(qwen-image): mlx-community packs generate and edit (VAE layout, table group size) #496).

Also new

• Short model aliases. Give a model a nickname in Model Settings (or "alias" in model-settings.json) and use it anywhere, including Ollama, load and unload; /v1/models lists it too (Model alias #520, feat: per-model alias in Model Settings (#520) #612). Thanks @lborloz.
• Consistent results. Seeded requests now give the same text whether or not the prompt was cached. DFlash on Nemotron-H and dense Qwen3.5/3.8 matches normal decoding exactly, sampled requests included (4, 6 and 8-bit), so seeded output with a DFlash drafter differs from earlier versions.
• Qwen3.8-27B with DFlash2 tries a tree of drafts each round instead of one path.
• More models load: Nemotron-H 3.5 MoE (fix: load Nemotron-H MoE checkpoints (layers_block_type list + MoE blocks) #559, thanks @sbusso) and Qwen3.8 Flash Next GGUFs (fix(gguf): serve qwen4exp GGUFs ds4 cannot load on the llama engine #546, thanks @zeeshanhaque21).
• --mtp-greedy-tail (or "mtp_greedy_tail": true per model) lets --mtp-typical accept longer runs on sampled requests; off by default (perf: MTP greedy tail for sampled requests (--mtp-greedy-tail) #602). Thanks @beamivalice.
• App polish: sidebar folders for chats, agent threads and terminals, and terminals reopen where you left them; the memory meter splits weights, KV cache and the rest; the Image pane matches the other Create panes and keeps your draft, and app text never drops below 12pt (Lay the Image pane out like the other Create panes #587, app: the type floor is 12, a role picks the step, and every piece of text is on the ladder #572, thanks @lojza3d and @LXD-8).
• Agents get an mlx-serve skill when launched from the app or mlx-serve launch, for wiring code to the local APIs.
• Models from mlx-serve pull register under their org/name id, and Ollama's /api/show reports thinking and vision on models not yet loaded (fix(registry): register pulled models under their org/name id #578, fix(ollama): advertise thinking and vision on /api/show for unloaded models #579). Thanks @brandondyal.

---

This discussion was created from the release v26.10.1 - Speed Across the Board - GGUF on Our MLX Engine - Sushi Flash Next EXL3 🍣.