feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints

[jason-fxz]

Summary

Serves llama.cpp llama GGUF files on the existing llama model classes: Llama 3.x and the Mistral models converted before llama.cpp had a mistral3 arch (Mistral-7B, Mistral-Nemo, Mistral-Small-3.1/3.2, Magistral, Devstral-2507). Text only. Also fixes three GGUF tokenizer bugs that Llama 3, Mistral, DeepSeek-R1-Distill-Llama, Muse-Glimmer and MiniMax files hit.

Changes

• models/llama/gguf.py (new):
  • Builds the config from the GGUF metadata.
  • llama.cpp runs interleaved rope on this arch, so its converter permutes the rows of each q and k head. The loader undoes the permute and the model keeps HF's NeoX rope. The permute moves whole rows, so it works on packed ggml rows.
  • Llama 3 rope scaling survives only as the rope_freqs divisors. The loader sets llama3 scaling with factor = max(rope_freqs), low 1, high 4, original 8192 (every Llama 3.1–3.3 release), and refuses the file unless the result reproduces rope_freqs.
  • Reads the single attn_qkv matrix of files converted with --fuse-qkv as qkv_proj.
  • Refuses Mixtral experts and biases (also arch llama), partial rope, other rope scaling, prism files, per-tensor scales, metadata-only GGUFs, and early unsloth Mistral-Nemo conversions whose metadata gives head_dim 160 for 128-wide heads (llama.cpp rejects them too).
• models/llama/model.py: embed_tokens takes the quant config, so a quantized token_embd is served packed.
• models/gguf/tokenizer.py:
  • Without an add_bos_token key, uses llama.cpp's default: BOS for SPM and for the BPE pre-tokenizers llama.cpp lists (llama-bpe, tekken, ...). bartowski's Llama-3.1-8B and unsloth's Llama-3.2-1B lack the key, so raw /v1/completions prompts got no <|begin_of_text|>.
  • Matches every control and user-defined token whole at its file id. transformers skips tokens with characters outside the byte alphabet, so DeepSeek-R1-Distill-Llama's <｜User｜> and MiniMax's tokens with spaces were cut into pieces. A token the merges build stays a normal piece (unsloth's MiniMax-M3 marks U+FFFD as control).
  • Keeps no added token past the vocabulary. transformers' GPT2Tokenizer adds <|endoftext|> at id = n_vocab (128256 for Llama 3, 131072 for tekken), past the embedding table.
  • The 19 Qwen 2/3/3.5/3.6/3.8 and Gemma 4 GGUFs I have give the same token ids and chat encodings as main.
• gguf/config.py, register.py, kernel/aot_models.py: register the llama arch and claim it in the AOT model table.

Tested models

RTX PRO 6000 Blackwell (sm_120), ft serve --model-path <file>. GSM8K test problems 1–64 × 1 sample with each model's own sampling (Llama 3: temperature 0.6, top_p 0.9; Mistral-Small-3.2: 0.15; Mistral-Nemo: 0.3; Mistral-7B: 0.7). llama.cpp f2cc728 ran with top_k off and min_p 0. Also 48 staggered requests at concurrency 16.

Model	Quant	bs=1 tok/s	GSM8K FreeToken / llama.cpp f2cc728
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	Q4_K_M	230	55/64 / 52/64
unsloth/Llama-3.2-1B-Instruct-GGUF	Q8_0, tied embeddings	682	27/64 / 26/64
bartowski/Mistral-7B-Instruct-v0.3-GGUF	Q4_K_M, SPM	242	35/64 / 36/64
bartowski/Mistral-Nemo-Instruct-2407-GGUF	Q4_K_M, tekken	159	54/64 / 55/64
unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF	Q4_K_M	92	61/64 / 61/64

Staggered requests: no corrupted output on any file.

Results

• Weights: Llama-3.2-1B Q8_0 after ggml.dequantize reaches cos 0.99998 against unsloth/Llama-3.2-1B-Instruct (q, k, v, o, gate_up, down and the embedding at layers 0 and 15), and the norms are exact. Without the un-permute, q has cos 0.03. Dry load against the meta-built model matches every param of all five files (98–263). The recovered llama3 rope parameters equal the HF config (factor 32 for 3.2-1B, 8 for 3.1-8B).
• Tokenizer: on 11 strings with and without special tokens, the GGUF tokenizer equals HF tokenizer.json for all six files. Before, the two Llama 3 files without the key got no BOS, and <|endoftext|> encoded to 128256 / 131072. DeepSeek-R1-Distill-Llama-8B (unsloth and bartowski Q4_K_M) now equals HF on every special token and on a rendered multi-turn chat.
• vs llama.cpp f2cc728: same file, greedy, 6 prompts × 128 tokens. Identical outputs: 8B 3/6, 1B 3/6, Nemo 4/6, Small-3.2 5/6, Mistral-7B 3/6 (llama.cpp got FreeToken's prompt tokens, because its SPM tokenizer adds a ▁ after [INST] that HF does not). Every divergence is a near-tie in llama.cpp's own probabilities: both engines' picks are its top 2 (e.g. 0.495 vs 0.489).
• Coverage: header check (dry load, kernel coverage, tokenizer, chat template) of 922 llama-arch files from 59 HF repos: 859 pass. The rest: 26 IQ1_M, 15 in the removed Q4_0_4_4/4_8/8_8 types, 9 without a chat template (mistralai's Magistral GGUFs), 7 early unsloth Nemo conversions (refused), 6 old Llama-3-8B conversions without tokenizer.ggml.pre.
• --fuse-qkv: HF Llama-3.2-1B converted to Q8_0 with and without --fuse-qkv loads to byte-identical state dicts (98 tensors), with qkv_proj read from attn_qkv.
• Unit tests: tests/models and tests/tokenizer pass, except 5 test_quant_config cases that also fail on main.

Not covered

• The mistral3 arch (Ministral 3, Devstral-Small-2, current Mistral Small reconversions): it needs the Llama 4 attention temperature in the mistral model.
• Mixtral, and llama files with biases or linear / yarn rope scaling (refused).
• IQ1_M weights (no MMQ kernel).
• TP > 1.