fix(moe): refuse e8m0 scale code 0xFF when MXFP4 expert pieces are read

[Avicennasis]

Fixes #655. One commit on main @ 2d8b9cc; no kernel, no model family, no GPU involved.

What this is

e8m0 has one code with no finite value: 0xFF is NaN by the OCP MX spec, every other code is 2**(code-127). The ds_fp4 and gpt-oss expert kernels fold that straight into the GEMM, so one 0xFF byte in a scale tensor makes its 32-wide block NaN, and the NaN reaches the logits with nothing in the log naming the file. Neither MXFP4 reader looks for it today.

Change

• moe/expert_pieces.py: check_e8m0_scales(name, tensor, file) counts the bytes equal to 0xFF in one e8m0 scale tensor (torch.count_nonzero(tensor.view(torch.uint8) == 0xFF), one pass, U8 or F8_E8M0 storage alike, strided slices included) and raises ValueError naming the tensor, the count and the shard: e8m0 scale tensor 'layers.0.ffn.experts.1.w2.scale' in 'model-00001-of-00002.safetensors' carries 1 NaN scale code(s) (0xFF) out of 64: the checkpoint is corrupt. A clean tensor is returned unchanged. ValueError is what the sibling checks in this module raise (Incomplete expert tensors in checkpoint), and the engine's _weight_load_context reports it as WeightLoadError on the log and /health like any other unreadable checkpoint.
• models/deepseek_v4/weight.py::iter_expert_pieces: every *.scale tensor passes through the check as it is read, before per_expert_pieces groups it, on the serial and the parallel reader (the shard name comes from the index's weight map, which the serial reader already opens).
• models/gpt_oss/weight.py::iter_expert_pieces: every *_scales tensor passes through the check as it is read, serial (the open shard) and parallel (this rank's intermediate slice; shard name from safetensors_weight_map).

Refusal rather than zeroing: a NaN scale is never a valid value, and the published checkpoints do not carry it (below), so a 0xFF is a corrupt file and the operator should see which one. Not covered: an FTW checkpoint converted before this change keeps whatever the raw checkpoint had, since the banks are read back without the raw reader; NVFP4 experts carry fp8 (e4m3) block scales and are not e8m0, so they are not scanned.

How tested

CPU-only box (Ubuntu 24.04, AMD Ryzen 7 5800X, no GPU), Python 3.12.3, torch 2.11.0+cu130, safetensors 0.8.0, flashlib 0.3.0, pytest 9.1.1, PYTHONPATH=python, C++ extension not built, no flashinfer. The readers stop at the piece, so nothing here needs a kernel.

tests/moe/test_e8m0_scale_check.py, 12 cases:

• the helper refuses one 0xFF byte naming the tensor, the count (1 ... out of 32) and the file, as U8 and as F8_E8M0, and counts two; every finite code 0x00..0xFE passes and the same object comes back, including a strided view;
• DeepSeek-V4 ds_fp4 synthetic checkpoint (two shards, one MoE layer of two experts plus an MTP layer the reader skips, F8_E8M0 scales as deepseek-ai/DeepSeek-V4-Flash stores them): one 0xFF in layers.0.ffn.experts.1.w2.scale is refused by that name and its shard, serial and parallel; the clean checkpoint yields both expert pieces with the six roles, and torch.count_nonzero is called exactly six times, once per scale tensor and each over exactly that tensor's bytes (the weights and the MTP layer are never scanned);
• gpt-oss synthetic checkpoint (U8 stacked scales as openai/gpt-oss-20b stores them): one 0xFF in gate_up_proj_scales and two in down_proj_scales are refused by name, shard and count, serial and parallel; the clean checkpoint yields the layer piece.

$ PYTHONPATH=python python -m pytest -q tests/moe/test_e8m0_scale_check.py
12 passed in 30.64s

The same file against unmodified main @ 2d8b9cc: 10 failed, 2 passed (the two clean gpt-oss cases pass on both; the refusals and the per-tensor count fail because main does not scan).

Full suite on this box: pytest -q tests on this branch 8 failed / 1528 passed / 635 skipped (2153 collected, 9:25); unmodified main @ 2d8b9cc in the same venv 8 failed / 1516 passed / 635 skipped (2153 collected, 9:20). Failure sets identical, all environment-bound (two test_cache_budget and one test_offload case that need flashinfer, the swiglu-clamp and deepseek_v41/test_engram cases that need the C++ extensions, three test_quant_config cases that probe this box's HF cache); +12 is exactly the new file.

Real checkpoints, read over HTTP range requests of the scale tensors alone (no full download): all 48 expert-scale tensors of openai/gpt-oss-20b (597,196,800 bytes) and 9 of the 46 shards (6,939 of the 33,924 expert-scale tensors, 1,811,953,152 bytes) of deepseek-ai/DeepSeek-V4-Flash carry zero 0xFF bytes, so the refusal does not touch either published checkpoint.

git diff --check clean, ruff 0.17.0 on the touched files: no new findings against main.

Not run on a GPU: the change does not reach a kernel; it ends at the piece the banks are packed from.