[taking-lying-flat]

With hybrid auto-fetch, a single miss at fetch_fraction=0.4 rounds to zero fetches. A repeatedly used expert can therefore keep running on the CPU while GPU cache slots remain empty. Recency prioritization cannot help when the fetch count stays zero.

Allow the recency policy to promote one previously used missing expert into an empty slot when the automatic split selects zero. This lets a stable working set finish warming the cache. The CPU reference and Triton kernel use the same rule; explicit fixed caps, lowest_id, and full-cache eviction decisions retain their existing behavior.

Only python/freetoken/moe/offload_kernels.py is changed. No test files are included.

Validation

Compared against main at 9c635a157798da4aa43e6d051e92620521ed1858 on an NVIDIA RTX A1000 Laptop GPU (4 GiB, driver 595.91.07), Intel Core i7-12800HX, Linux x86_64, Python 3.12.13, PyTorch 2.12.1+cu130, Triton 3.7.1.

Executed the upstream CPU reference and the actual CUDA Triton cache-management kernel with metadata fixtures matching the cache tensor layouts. The unrelated flashlib import was omitted from the standalone harness; the hybrid function bodies were executed unchanged. No model checkpoint, expert-weight transfer, GEMM, CPU expert executor, or end-to-end serving benchmark was involved.

At fraction 0.4, 8 layers, 16 experts per layer, 64 slots, top-8, and 256 decode steps per active layer:

Scenario	main	This PR
One active layer, 7 experts initially resident: additional fetches	0	1
Same scenario: CPU-assigned routes	256	1
All 8 layers active with fixed top-8, empty cache: total fetches	56	64
Same scenario: CPU-assigned routes	2104	88
Same scenario: empty slots after 256 steps	8	0

The cold-cache scenario becomes fully resident after the fifth step on each layer. CPU/CUDA state and routing agree. Additional checks passed for fixed caps 0/1, lowest_id, a full cache, recurring-expert selection among first-time misses, and CUDA Graph replay using live cache state.

Metadata-only A/B timings (microseconds per input-id copy plus cache-management kernel, CUDA Graph, median of 9 alternating trials, each with 20 replays of 128 calls):

Slots	Scenario	main	This PR
64	all hits	2.845	2.911
64	full cache, one recurring miss	2.228	2.652
2048	all hits	2.497	2.564
2048	full cache, one recurring miss	2.014	2.612

The additional empty-slot check has a measurable cost. These are management-overhead measurements, not an end-to-end speedup claim; model-level tokens/s remains unmeasured.

Existing test expectation to update

test_hybrid_fraction_gpu_matches_cpu_reference currently requires the fetched count to equal the stateless _balanced_fetch(...) result on every step. That assertion encodes the zero-fetch behavior this PR changes. Replaying its seed-0 routing sequence with the production CPU reference gives five policy-expectation differences (zero-based steps 16, 17, 18, 21, 34): one miss is promoted instead of the old expected zero. CPU/GPU parity remains checked separately.

This production-code-only PR leaves that test file unchanged. The policy assertion needs to account for the empty-slot recurring-miss exception before the existing test can pass under the new behavior. The repository's requested model-level end-to-end A/B is also outstanding.

Standalone reproduction command (PyTorch and Triton required)

Run from the repository root on both the base revision and this branch:

python - <<'PY'
import ast
from pathlib import Path
from types import SimpleNamespace
import torch

p = Path('python/freetoken/moe/offload_kernels.py').resolve()
tree = ast.parse(p.read_text(), filename=str(p))
tree.body = [n for n in tree.body if not (
    isinstance(n, ast.ImportFrom) and n.module == 'flashlib.kernels.slot_cache'
)]
ns = {'__name__': 'hybrid_repro', '__file__': str(p)}
exec(compile(tree, str(p), 'exec'), ns)
ensure = ns['ensure_experts_hybrid']

for device in ('cpu', 'cuda'):
    def tensor(shape, value, dtype):
        return torch.full(shape, value, dtype=dtype, device=device)
    c = SimpleNamespace(
        num_layers=8, num_experts=16, cache_size=64,
        slot_for_id=tensor((8, 16), -1, torch.int32),
        id_of_slot=tensor((64,), -1, torch.int32),
        usage=tensor((64,), 0, torch.int64),
        step=tensor((), 0, torch.int64),
        active_mask=tensor((16,), 0, torch.int32),
        evict_slots=tensor((64,), -1, torch.int32),
        src_indices=tensor((64,), -1, torch.int32),
        num_indices=tensor((1,), 0, torch.int64),
        num_missing_full=tensor((1,), 0, torch.int64),
        expert_recency=tensor((8, 16), -1, torch.int64),
    )
    ensure(c, 0, torch.arange(7, dtype=torch.int32, device=device), 7)
    fetches = cpu_routes = 0
    for _ in range(256):
        ids = torch.arange(8, dtype=torch.int32, device=device)
        ensure(c, 0, ids, 1, 0.4)
        fetches += c.num_indices.item()
        cpu_routes += (ids < 0).sum().item()
    print(device, dict(fetches=fetches, cpu_routes=cpu_routes,
                      resident=c.slot_for_id[0, 7].item() >= 0))
PY

Expected on main: fetches=0, cpu_routes=256, resident=False on both devices.
Expected on this branch: fetches=1, cpu_routes=1, resident=True on both devices.

AI-assisted implementation and validation.