CUDA loader: Q4-class GGUF load transient is ~1.8× mapped tensor size on unified memory (GB10) — freezes 128 GB DGX Spark

[snhwang]

Environment

• 2× NVIDIA DGX Spark (GB10, 121 GiB unified memory each), aarch64
• CUDA 13.0, driver 580.173.02
• ds4 commit 80df56a, built with make cuda-spark
• Models from antirez/deepseek-v4-gguf (thank you for publishing these)

Summary

Loading a layer slice of the Q4 Flash GGUF
(DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf,
153.3 GiB) consumes roughly 1.8× the mapped tensor size during startup.
On 121 GiB nodes this reproducibly exhausts memory at the same progress
mark, and on a node running a desktop session it does not fail cleanly — it
freezes the machine (pinned allocations racing page-cache reclaim), needing
a power cycle. The Q2 imatrix GGUF shows no such behavior: we load its full
80.76 GiB single-node and 39–42 GiB distributed slices routinely on the same
machines.

Reproduction (4/4 failures, identical failure point)

Distributed pipeline mode, worker or coordinator role, --ctx 65536+.
Every attempt died immediately after logging:

ds4: CUDA loading model tensors 64.00 GiB cached
ds4: CUDA prepared model tensor mappings 64.12 GiB

#	Source file	Slice (--layers)	Mapped size	Outcome
1	monolith 153.3 GiB	0:21	77.7 GiB	desktop froze, hard reboot
2	monolith	18:output	89.8 GiB	OOM-killed by our watchdog at 0.6 GiB free
3	monolith + posix_fadvise(DONTNEED) loop on the GGUF	18:output	89.8 GiB	same kill — fadvise does not help (pages pinned)
4	per-node split GGUF (92.6 GiB file containing only layers 19–42 + shared tensors)	19:output	86.3 GiB	watchdog kill at 0.2 GiB free

Attempt 4 is the interesting one: we split the GGUF so the file contains
only the tensors being mapped (tool:
https://github.com/snhwang/binary-dwarf/blob/main/tools/split_gguf.py —
reuses your splice_mixed_expert_layers_gguf.py parser). File ≈ slice ≈
86–92 GiB, node had ~117 GiB available, and it still died at the same
"64.12 GiB prepared" mark with ~117 GiB consumed. So the transient is not
explained by page cache of unrelated file regions — it tracks the load
path itself at ~1.8× mapped bytes.

Control (works)

Same nodes, same build, same launch flags, Q2 imatrix GGUF
(...IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf, 86.7 GB):

• full 80.76 GiB single-node load: fine (peak ≈ mapped size)
• 39–42 GiB distributed slices: fine
• has been serving for days via ds4-server distributed without incident

Hypothesis (unverified)

The Q4 file's F16 tensor groups (F16HC, F16Compressor, F16Indexer)
appear to take a staging/conversion path in the CUDA loader that the Q2
file's fully-quantized equivalents skip, holding both staging and
destination copies for a large fraction of the model before releasing. The
consistent death right after the 64 GiB progress line (16 GiB chunk
granularity) suggests the peak scales with cumulative loaded bytes, not a
single tensor.

Impact / ask

• 128 GB-class unified-memory machines (DGX Spark, and presumably Strix
  Halo per STRIXHALO.md's OOM note) cannot load the Q4 Flash GGUF at all,
  even split across nodes whose steady-state budget would comfortably fit.
• On a machine with a GUI the failure mode is a full system freeze, which
  is rough for first-time users.

Is this memory model expected? If a fix is impractical, a documented
peak-memory formula (e.g. "budget ~1.8× the mapped bytes for Q4-class
files during load") or a staged/streaming load for the F16 groups would
both help. Happy to run instrumented builds or provide more data — the
setup is reproducible here in minutes, with memory watchdogs armed.

(Report written with assistance from Claude; all measurements are from our
cluster as described.)

[yhayato]

I have the same problem. I did not check all the commits. When I checked the older commits, the ds4-server works up to commit 519c4d8.
With this commit, I can still load the latest weight file (DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf), but it seems not to be working properly, and the response is terminated while reasoning.