Thanks for releasing Step-3.7-Flash! The NVFP4 checkpoint can't run MTP (multi-token prediction) speculative decoding, while the FP8 and BF16 checkpoints can. Two problems in the NVFP4 export:
1. MTP (next-n predict) weights are missing
config.json declares num_nextn_predict_layers: 3, but model.safetensors.index.json contains no tensors for the 3 MTP layers (indices num_hidden_layers .. num_hidden_layers + 2). The MTP block present in Step-3.7-Flash (BF16) and Step-3.7-Flash-FP8 — e.g. model.layers.45/46/47.{enorm, hnorm, eh_proj, self_attn.*, mlp.*, transformer.shared_head.*} — is absent from Step-3.7-Flash-NVFP4.
This looks like the known ModelOpt behavior where MTP keys are stripped during the calibration load unless the modeling class is patched (cf. NVIDIA/Model-Optimizer#750; ModelOpt has since added MTP-preserving export, e.g. for GLM-4.7).
2. Several per-layer config lists are truncated
In config.json text_config, these are length 45 (= num_hidden_layers) in the NVFP4 export but 48 in the original, so they don't cover the MTP layers: layer_types, partial_rotary_factors, swiglu_limits, swiglu_limits_shared. A drafter that indexes e.g. config.layer_types[45] hits IndexError.
Result
Serving Step-3.7-Flash-NVFP4 with --speculative-config '{"method":"mtp","num_speculative_tokens":3}' fails — the MTP drafter can't be built/loaded.
Request
Please re-export Step-3.7-Flash-NVFP4 with the MTP block preserved and the per-layer config lists at full length (48). Since ModelOpt now supports keeping MTP through the calibration load, this should be reproducible.
As a stopgap we grafted the BF16 MTP weights from Step-3.7-Flash into the NVFP4 snapshot and extended the config lists, and MTP then works (mean acceptance length ~2.4–2.6 on dual DGX Spark GB10) — but a corrected official export would make that unnecessary for everyone. Happy to share details.
Thanks for releasing Step-3.7-Flash! The NVFP4 checkpoint can't run MTP (multi-token prediction) speculative decoding, while the FP8 and BF16 checkpoints can. Two problems in the NVFP4 export:
1. MTP (next-n predict) weights are missing
config.jsondeclaresnum_nextn_predict_layers: 3, butmodel.safetensors.index.jsoncontains no tensors for the 3 MTP layers (indicesnum_hidden_layers .. num_hidden_layers + 2). The MTP block present inStep-3.7-Flash(BF16) andStep-3.7-Flash-FP8— e.g.model.layers.45/46/47.{enorm, hnorm, eh_proj, self_attn.*, mlp.*, transformer.shared_head.*}— is absent fromStep-3.7-Flash-NVFP4.This looks like the known ModelOpt behavior where MTP keys are stripped during the calibration load unless the modeling class is patched (cf. NVIDIA/Model-Optimizer#750; ModelOpt has since added MTP-preserving export, e.g. for GLM-4.7).
2. Several per-layer config lists are truncated
In
config.jsontext_config, these are length 45 (=num_hidden_layers) in the NVFP4 export but 48 in the original, so they don't cover the MTP layers:layer_types,partial_rotary_factors,swiglu_limits,swiglu_limits_shared. A drafter that indexes e.g.config.layer_types[45]hitsIndexError.Result
Serving
Step-3.7-Flash-NVFP4with--speculative-config '{"method":"mtp","num_speculative_tokens":3}'fails — the MTP drafter can't be built/loaded.Request
Please re-export
Step-3.7-Flash-NVFP4with the MTP block preserved and the per-layer config lists at full length (48). Since ModelOpt now supports keeping MTP through the calibration load, this should be reproducible.As a stopgap we grafted the BF16 MTP weights from
Step-3.7-Flashinto the NVFP4 snapshot and extended the config lists, and MTP then works (mean acceptance length ~2.4–2.6 on dual DGX Spark GB10) — but a corrected official export would make that unnecessary for everyone. Happy to share details.