Repository navigation
[Model] DeepSeek-V4.1-Flash (deepseek_v41) — hybrid CPU-MoE makes it viable on a single GPU, and Engram is already isolated in 2 shards #444
Description
Before you start
Checked the Roadmap (#79 — no mention of deepseek / v4.1 / engram), the FAQ (#84), and existing issues: searching deepseek_v41, engram, V4.1 and v41 across this repo returns nothing as of 2026-09-11.
Hugging Face link
https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
Is the model architecture already supported
No — deepseek_v4 is supported, but this is a new architecture: model_type: deepseek_v41, architectures: ["DeepseekV41ForCausalLM"].
Is the quantization already supported
Partly. Top-level quantization_config is {quant_method: fp8, activation_scheme: dynamic, weight_block_size: [32, 32], scale_fmt: ue8m0}, and the reference config declares dtype: fp8 with expert_dtype: fp4 — i.e. FP4 routed experts with FP8 dense. FP8 and NVFP4 are listed as supported today, so the open question is whether this FP4 expert layout maps onto an existing bank schema or needs a new one.
What happens when you load it
ERROR Backend supervisor: ValueError: Model architecture DeepseekV41ForCausalLM not supported
Traceback (most recent call last):
return _MODEL_REGISTRY[model_architecture]
KeyError: 'DeepseekV41ForCausalLM'
(Registry lookup fails before anything else, so this is an architecture gap rather than a checkpoint/quant issue.)
Anything else
I think FreeToken is in an unusually strong position for this model, and the checkpoint turns out to be laid out in a way that helps. Sharing the measurements in case they are useful for scoping.
Why the hybrid MoE backend matters here
The checkpoint is 475.3 GiB. Engines whose offload path copies weights back to the GPU each forward pass are PCIe-bound; on a 32 GiB GPU that means streaming ~256 GiB per token, i.e. fractions of a token/s. ft bench bw on my box (2026-08-29):
| path | measured |
|---|---|
| PCIe H2D, linear | 57.8 GB/s |
| host DRAM stream read | 266.9 GB/s |
hybrid ds_fp4: CPU MoE, overlapped |
114.5 GB/s |
hybrid ds_fp4: PCIe gather, overlapped |
53.1 GB/s |
| hybrid combined | 167.7 GB/s |
Keeping expert weights in host RAM and computing them on CPU workers — only activations crossing the bus — is ~3x the effective bandwidth of weight streaming. As far as I can tell it is the only architecture that makes this checkpoint viable on a single consumer GPU.
The checkpoint is already partitioned along the useful line
Of 96,085 tensors, only 12 are Engram (layers.1.engram.* and layers.14.engram.*), and they occupy exactly the last two shards:
model-00047-of-00048.safetensors 94.6 GiB ┐ Engram, 196B params
model-00048-of-00048.safetensors 94.6 GiB ┘ = 189.2 GiB
model-00001..00046-of-00048 286.1 GiB backbone, 552B params
─────────
475.3 GiB
So the 196B "conditional memory" can be handled entirely separately from the 552B backbone with no repacking.
Engram does not need to be resident
From inference/engram.py and the reference config.json:
engram_layer_ids = [1, 14],engram_max_ngram_size = 4,engram_n_heads = 8,engram_head_dim = 256, fp8engram_num_embeddings = [384006168, 384016682](~384M rows each)- per token: 2 layers x (4-1) n-gram sizes x 8 heads = 48 row gathers, 256 B each
It is a token-hash lookup, not a GEMM — an ideal candidate for staying on NVMe. I benchmarked that exact access pattern against a PCIe 5.0 SSD (Samsung 9100 PRO), using O_DIRECT so neither the page cache nor the host's VHDX cache can flatter the result:
random 4 KiB read, QD=1: median 80.6 us p95 117.6 us (through a WSL2 VHDX)
48 gathers/token, serial: 3.78 ms
48 gathers/token, 8-way: 1.14 ms <-- best
Against a 30 ms/token decode budget that is ~3.7% overhead, and it is a pessimistic bound: every gather is assumed to miss, the I/O crosses a virtualized block device, and there is no batching (io_uring or a batched gather would cut it further). N-gram frequencies are Zipf-distributed, so spare host RAM acting as page cache should absorb a large share in practice.
What that does to the memory budget
Config: 512 GiB host RAM, one RTX 5090 (32.6 GiB), PCIe 5.0 NVMe.
| Engram placement | host RAM needed |
|---|---|
| resident in host RAM | ~482 GiB — does not fit |
| read from NVMe | ~293 GiB — fits, ~160 GiB to spare |
The GPU side is where V4.1 is surprisingly friendly:
dense/attn ~23.6 GiB
KV cache @ 1M context 0.89 GiB (CSA2: 890 B/token, global)
CUDA graphs ~1 GiB
---------
expert slot cache ~5 GiB (~263 slots at 16.88 MiB/expert)
For comparison, DeepSeek-V4-Flash-0731 on the same card spends 4.95 GiB of VRAM on KV for a 150K window (Allocating 150016 tokens for DSV4 KV cache, total = 4.95 GiB). V4.1 gets its full 1M context for 890 MB.
Architecture delta vs deepseek_v4
Likely reusable: the hybrid CPU-MoE machinery (384 routed + 1 shared, top-6 — same shape as V4, just larger: 3 x 5120 x 2304 FP4 = 16.88 MiB/expert vs V4's 12.00 MiB), DSpark (dspark_block_size: 5, 3 MTP layers, dspark_target_layer_ids: [37,38,39], dspark_n_routed_experts: 128), and the FP8/FP4 quant paths.
New:
- CSA2 attention — each layer is assigned one of Full / Reindex / Reuse;
kv_source_layers: [2,8,14,20]andindex_source_layers: [2,8,14,20,24,28,32,36]share main KV and indexer K across layers; FP4 main KV (E2M1, one E4M3 scale per 16 channels);index_topk: 512,window_size: 128,compress_ratiosper layer. This is what gets the global KV footprint to 890 B/token, and it is the main departure fromdsv4_compress/dsv4_sparse/dsv4_indexer/ the DSV4 paged pool. - Engram — as above; small in code terms.
- Nested
text_config+vision_config; the vision tower is optional for text-only serving (shards 1-2, 2.1 GiB).
Other engines
SGLang has a preview (lmsysorg/sglang:dev-dsv41) with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, which moves the two Engram tables to a shared host copy — documented as bitwise unchanged, but loaded into host RAM rather than memory-mapped, and its verified configs are 4x GB300 / 4x MI350X. vLLM has an open PR. Neither addresses the single-GPU + large-host-RAM case that FreeToken's hybrid backend already targets.
Offer
Happy to help with the parts I can carry: I can implement and test the Engram-from-NVMe path, and I can supply ft bench bw profiles and end-to-end numbers from a single-GPU + 512 GiB host configuration.
Activity
janit commentedon Sep 11, 2026
Some numbers from the configuration this request targets: two single-GPU hosts that have been serving DeepSeek-V4-Flash on FreeToken's
hybridbackend. Anything marked measured was run on them. Everything about V4.1 itself is arithmetic fromconfig.jsonand the shard headers, not a run.Hosts: RTX 5090 (PCIe 4.0 x16), 64-core EPYC Rome, 8 x 32 GB DDR4-3200 on all eight channels (NPS1), 251 GiB RAM. Two identical boxes.
1.
ft bench bwon 8-channel DDR4 (measured)ds_fp4That is the DDR4 / PCIe 4.0 end of the range, next to your 167.7 GB/s.
What it buys on V4-Flash (measured, streamed, decode timed separately from TTFT): 24.5-26.9 tok/s single stream, 65.6 tok/s aggregate at 8 concurrent.
Decode rate x routed bytes per token comes out at ~72-85 GB/s effective across three different checkpoints on these hosts (V4-Flash: 24.5 x 3.45 GB = 84.5). So on this class of machine, decode is approximately effective bandwidth / routed bytes per token. That makes the next number the one that matters.
2. Routed expert bytes per token
V4.1's routed experts use the
ds_fp4bank layout unchanged, only at a larger geometry. From the shard headers:So the existing packed-e2m1 + E8M0-per-32 CPU-MoE kernel should carry over. The
mxfp4_tritonbank is a different layout (transposed, with bias), and on host A it benches at 49.3 GB/s and resolves tooffload. It would be an expensive path to land on by accident.Per expert, scales included: 3 x 2304 x 5120 x (1/2 + 1/32) B = 18.80 MB (17.93 MiB). The 16.88 MiB in the issue is the e2m1 codes alone. So:
At the ~85 GB/s these hosts reach on V4-Flash, that predicts ~19 tok/s single stream for V4.1 (estimate), against 24.5 measured on V4-Flash. That assumes enough RAM (~512 GB) to keep the experts resident, and excludes Engram reads, DSpark, and any difference in attention cost.
3. CED's prefill saving does not carry over to an offload host (estimate)
Per the model card, CED activates 8B parameters per token in prefill against 16B in decode, because prompt tokens only run the 20 encoder layers. On a VRAM-resident engine that roughly halves prefill.
On an offload host, prefill is paced by expert-bank traffic instead. With 384 experts and top-6, a chunk of more than a few hundred tokens touches essentially every expert in every layer it passes through:
Half the layers, but each layer's bank is 2.1x larger. Expect prefill throughput and TTFT close to V4-Flash's, not 2x better. For scale, V4-Flash on these hosts measured ~1,000 tok/s marginal prefill and 37.4 s cold TTFT at 32k tokens.
4. On a 32 GiB card, the KV pool is not what caps context (measured on V4-Flash)
V4.1's 890 B/token makes a 1M-token pool nearly free. But on V4-Flash, the limit on this card was never the pool; it was the prefill activation peak.
As of 4b94bdc:
_adjust_dsv4_configforces single-chunk prefill.indexer_select_prefillallocateschunk x n_blocks, with n_blocks = context / 4.With
--kv-reserve-tokens 131072the pools sized fine. Then a 56k prompt OOMed inside the indexer and took the scheduler down. The largest reservation we validated is 98,304 atmemory_ratio 0.85: a 97k-token prompt ran with ~800 MiB of VRAM headroom.If the hierarchical indexer inherits that path, the practical context on a 5090 will be set there, not by the pool.
5. Prior art for whoever ports it
qafterwq_b, and V4.1 does not.janit commentedon Sep 11, 2026
A follow-up on the Engram read path. The ~3.7% overhead in the issue budgets decode only, and prefill is where the storage question actually gets decided. These are estimates, not measurements.
Prefill does the same Engram reads for every prompt token. Engram layers 1 and 14 are both inside the 20-layer encoder, so CED's prefill saving does not skip them. Every prompt token costs the same 48 row reads as a decoded token, so a 48k-token prompt is ~2.3M reads on a cold cache. How those reads are issued matters more than which drive serves them:
For scale, V4-Flash prefill at 48k tokens on a single 5090 is roughly a minute (extrapolated from 37.4 s measured at 32k). So a backend that does one synchronous read per row would roughly quadruple TTFT on NAND, while a batched one makes the drive nearly irrelevant.
What that suggests for the backend:
preadper row on the forward-pass thread. JigSawPT measured 11.4x on the Engram input step from prefetching alone: https://github.com/JigSawPT/deepseek-v41-flash-on-5090Low-latency storage (Optane-class drives, whose advantage is QD1 latency) is the hardware workaround if the first backend does end up synchronous. The checkpoint already splits the two tables one per shard, so one table per drive is a symlink. With a batched reader it buys little over good NAND.
minseoc03 commentedon Sep 14, 2026
Hi! I’d be interested in helping with the V4.1 support, especially the model/config bring-up and CSA2 attention path. I also reached out on Slack about it. If no one is already working on these parts, I’d be happy to take them on.
angelbaba01L commentedon Sep 14, 2026
@minseoc03 — in case you have not seen it, #460 already covers much of that ground: V4.1 model loading, CSA2 compressed attention, shared index selection, NVFP4 experts and packed FP8/FP4 Engram lookup, plus a
--kv-cache-dtype fp8-fp4path. 117 files, +13,252/-317, 319 focused tests passing, and a full-model run validated at 32,768 context on a single RTX 5090 (32 GB) with 512 GB of host RAM.It is
CONFLICTINGagainst main and has no review yet, so rebasing it or reviewing the CSA2 path would probably land support sooner than a parallel implementation would.One thing worth having in front of whoever works on it: JigSawPT's llama.cpp port write-up enumerates nine silent V4 → V4.1 incompatibilities that are easy to inherit, and the dangerous ones fail quietly rather than loudly — the fp8 scale block changing from 128×128 to 32×32 (every shape stays valid, every dequantized weight comes out mis-scaled), and the absent q-norm. https://github.com/JigSawPT/deepseek-v41-flash-on-5090
minseoc03 commentedon Sep 15, 2026
@angelbaba01L Thanks for pointing me to #460 -- I hadn't seen it. I’ll take a look at the CSA2 path and the V4 / V4.1 differences you mentioned. Happy to help with the review/fixes there!
jason-fxz commentedon Oct 5, 2026
Support for DeepSeek-V4.1-Flash is here #603. Rough requirements for the official checkpoint:
We validated it on a 96 GB GPU with 499 GiB of host RAM; smaller GPUs are untested.