Repository navigation
perf(moe): skip inactive block-fp8 decode routes - #665
Open
taking-lying-flat wants to merge 1 commit into
Open
taking-lying-flat wants to merge 1 commit into
taking-lying-flat wants to merge 1 commit into
Conversation
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 12, 2026
… decode routes Assisted-by: Claude Opus 5.5
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 12, 2026
Follow-up to upstream PR FlashML-org#665: hybrid decode passes CPU-assigned routes as slot -1, which must add nothing even with a nonzero weight; fails before FlashML-org#665. Assisted-by: Claude Opus 5.5
KarrAcaRn
pushed a commit
to KarrAcaRn/FreeToken-ByAI
that referenced
this pull request
Oct 12, 2026
FlashML-org#665 adopted with a negative-slot test, FlashML-org#378 deferred again (ROCm, now ready); FlashML-org#399 landed upstream via FlashML-org#662. Assisted-by: Claude Opus 5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With hybrid block-FP8 decode, CPU-assigned routes currently reach the GPU as slot 0 with routing weight 0. The gate/up GEMM still reads and computes that expert, and the down GEMM only multiplies by zero after its K-loop. With two GPU routes out of top-k 8, six additional routes therefore run both GEMMs.
Handle negative slots in the shared decode kernel by writing zero to the route's output tile and returning before reading activations, expert weights, scales or routing weights. Both gate/up and down use this kernel. Writing the outputs is necessary because the following activation/reduction reads those buffers, including on CUDA Graph replay. Then enable
supports_inactive_slotsso hybrid preserves its-1sentinels.The launch shapes stay fixed, and the activation still processes the zeroed buffer. Follow-up to #662.
Validation
Tested against main
95ac86f21567bc244c7a9d91972838fbf049b4edon an NVIDIA RTX A1000 Laptop GPU (4 GiB, sm_86), Intel Core i7-12800HX, Ubuntu 26.04.1, NVIDIA driver 595.91.07, PyTorch 2.12.1+cu130 and Triton 3.7.1. This exercises the FP8 compatibility path. The existing test environment's PyTorch version is below main's currently declared >=2.14 requirement.ruff check --select E9,F63,F7,F82,py_compileandgit diff --checkpassedRegression and benchmark scripts were kept outside the repository; this PR contains only the two implementation files. Local commands:
python -m pytest tests/kernels/test_fp8_blockscale_moe.py /tmp/freetoken-fp8-inactive-validation/test_inactive_slots.py -q --tb=short compute-sanitizer --tool memcheck --error-exitcode 99 --log-file /tmp/freetoken-fp8-inactive-validation/memcheck.log python -m pytest /tmp/freetoken-fp8-inactive-validation/test_inactive_slots.py -k 'both_gemms or all_inactive or cuda_graph' -q --tb=short python /tmp/freetoken-fp8-inactive-validation/bench_inactive_slots.pyGPU decode A/B
Synthetic block-quantized expert banks, batch size 1, top-k 8, two resident expert banks, bf16 activations and padded block scales. No model checkpoint was loaded. The baseline maps inactive slots to 0 and zeroes their routing weights; the new path keeps negative slots. Times cover gate/up, activation, down and reduction under CUDA Graphs. Each value is the median of five 100 ms measurements, alternating baseline/new order.
These are local GPU decode measurements on the sm_86 compatibility path. End-to-end model tokens/s, the complete CPU/GPU overlap path and native-FP8 GPU runtime performance have not been measured.