Found while fixing the CPU readers in #3110 (PMAT-1101). I read the CUDA Q5_K kernels in crates/aprender-gpu/src/kernels/quantize/q5k/ against ggml's block_q5_K (d, dmin, scales[12], qh[32] at offset 16, qs[128] at offset 48; value l of sub-block s takes its fifth bit from bit s of qh[l]; sub-blocks 2c/2c+1 share qs[32c..32c+32], low nibbles then high). Neither kernel matches.
gemv.rs (Q5KGemvKernel, the M=1 decode path; WeightQuantType::Q5K => GemvKernel::Q5K)
qs offset and nibble selection match ggml.- The fifth bit is read from
qh[val_idx / 8] >> (val_idx % 8), a sequential bitmask. ggml puts it at qh[val_idx % 32] >> (val_idx / 32), so it differs for most values.
gemm.rs (Q5KKernel)
- Assumes
qs sits at 16..144 and qh at 144..176. ggml has qh at 16..48 and qs at 48..176.
- Takes sub-block
s's qs bytes from 16 + s*16 + lane/2, and the bit from (s*32 + lane) as a sequential bitmask.
- Unpacks the scales as a 12-bit stride (
bit_offset = s*12) and normalises by 1/63. That is not ggml's get_scale_min_k4.
Not measured yet. This comes from reading the code. The plan is to reuse the gguf-py golden super-block from FALSIFY-QDOT-009 (#3110) as a GPU-side test: gemv against the gguf-py dot, and gemm against the reference matmul. It will run on lambda (sm_89) and gx10 (sm_121), then the kernels get fixed.
Impact, if confirmed: --gpu on any model with Q5_K tensors (every Q5_K_M/Q5_K_S file, plus the Q5_K tensors inside Q4_K_M files) computes wrong projections on CUDA, and the CUDA release assets ship that path.
Found while fixing the CPU readers in #3110 (PMAT-1101). I read the CUDA Q5_K kernels in
crates/aprender-gpu/src/kernels/quantize/q5k/against ggml'sblock_q5_K(d,dmin,scales[12],qh[32]at offset 16,qs[128]at offset 48; valuelof sub-blockstakes its fifth bit from bitsofqh[l]; sub-blocks2c/2c+1shareqs[32c..32c+32], low nibbles then high). Neither kernel matches.gemv.rs(Q5KGemvKernel, the M=1 decode path;WeightQuantType::Q5K => GemvKernel::Q5K)qsoffset and nibble selection match ggml.qh[val_idx / 8] >> (val_idx % 8), a sequential bitmask. ggml puts it atqh[val_idx % 32] >> (val_idx / 32), so it differs for most values.gemm.rs(Q5KKernel)qssits at 16..144 andqhat 144..176. ggml hasqhat 16..48 andqsat 48..176.s'sqsbytes from16 + s*16 + lane/2, and the bit from(s*32 + lane)as a sequential bitmask.bit_offset = s*12) and normalises by1/63. That is not ggml'sget_scale_min_k4.Not measured yet. This comes from reading the code. The plan is to reuse the gguf-py golden super-block from FALSIFY-QDOT-009 (#3110) as a GPU-side test: gemv against the gguf-py dot, and gemm against the reference matmul. It will run on lambda (sm_89) and gx10 (sm_121), then the kernels get fixed.
Impact, if confirmed:
--gpuon any model with Q5_K tensors (every Q5_K_M/Q5_K_S file, plus the Q5_K tensors inside Q4_K_M files) computes wrong projections on CUDA, and the CUDA release assets ship that path.