Conversation
The auto tier decremented `remaining` by coli_cuda_tensor_bytes(), which is
the LOGICAL size of an expert. The allocator takes more than that, and
nothing charged the difference, so the budget drifted optimistic by a term
that grew with the tier. That is why auto could claim a card to within
4 MiB and then fail every lazy dense upload afterwards while reading as
healthy.
THE MECHANISM. It is not workspace overhead. cudaMalloc rounds a request
up, and a GLM-5.2 int4-g64 expert is six allocations of two sizes:
3x weights 6,291,456 B = 6.00 MiB already on a boundary, +0
3x scales 786,432 B = 0.75 MiB lands in 1.00 MiB, +0.25 each
0.75 MiB per expert uncounted. Measured here on sm_86; @terrizoaguimor
measured 0.741 MiB/expert (sigma 0.019, five clean configurations) on
H100/H200 in #687 from the other direction, and 0.750 sits inside that
interval. At the 6,235 experts auto selects on an H200 that is 4.6 GB
against a flat 2 GB reserve.
That reframes the fix. The conclusion in #687 was that 0.741 is
model-and-card specific and should be measured live from the first uploads
rather than frozen into the source. The distrust of the constant was right
and the reason was not: the term is roundup(scale_bytes) - scale_bytes,
which is a property of the allocator and the model geometry, so it is
knowable before a single expert is placed. No fixed-point iteration and no
measurement pass over real uploads.
coli_cuda_alloc_footprint() probes it, once per distinct size, cached. It
is PROBED rather than modelled on purpose: the rounding is a driver and
architecture property and a table fitted to one card would be silently
wrong on the next. coli_cuda_tensor_vram() applies it per ALLOCATION,
since the weights and the scales are separate cudaMallocs and each is
rounded on its own - summing first and rounding once would miss the scale
array's padding entirely, which is the whole term.
coli_cuda_tensor_bytes() is untouched and still logical: it mirrors upload
and free so the three cannot drift, and there is a test pinning that.
m->gpu_expert_bytes also stays logical, because it is reported to the user
as the tier's size and quoting allocator padding as model bytes would
trade one wrong number for another. Only `remaining` moves.
TWO IMPLEMENTATIONS THAT DID NOT SURVIVE MEASUREMENT, both recorded in the
source so nobody rebuilds them:
cudaMemGetInfo per placed expert is the obvious exact answer and it is
O(live allocations) - 0.52 us empty, 68.2 us with 12,000 live. A tier
that places thousands would make that quadratic.
A single-allocation probe over-reports by 3x. One cudaMalloc reserves a
whole 2 MiB VMM page, so one allocation of anything smaller reads as a
flat 2 MiB: 786,432 B measured 2.00 MiB at n=1 and 1.00 MiB at n=64.
That version would have shrunk the tier worse than the bug it fixes. Its
own test caught it.
Verified on Windows, RTX 3090 (sm_86), CUDA 13.1, MSVC 19.50.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight assertions against the LIVE allocator rather than against a rounding table, because the rounding is a driver and architecture property and a test that hardcoded one card's numbers would fail on the next. The load-bearing one is "footprint never below the request". The whole bug was a budget charged less than reality, so a footprint that under-reports would reintroduce it somewhere new. Two are negative controls, and they are the reason this suite means anything: an aligned 6 MiB request must be charged EXACTLY 6 MiB. Without it the suite passes on a function that simply doubles everything, which is precisely the failure the first draft had. 500 repeat calls must move free VRAM by 0 bytes. Checked by reading free VRAM rather than by timing, which would be flaky under load. The probe is also cross-checked against an independent bulk measurement in the same run, so the function and the ground truth cannot drift together. Wired in at the END of cuda-test: it is the only test here that allocates in bulk to measure an amortised cost, so it should not leave the card fragmented underneath a kernel test that follows it. Full run on RTX 3090 (sm_86), CUDA 13.1, MAKE_EXIT=0, all eight binaries: backend_cuda q8/q4/q2/f32/e8 correctness ok ragged_attention ok fp8_warp shared-lut 0, hw-cvt 0, mutated entry 1 (fires) absorb_determinism batch 0/145, ragged 0/29 fp8_cuda oracle 0 mismatches, API 0 mismatches weights_owned 8 fail cycles, 0 bytes cumulative mxfp4 ok alloc_footprint ok, per-expert unaccounted 0.7500 MiB Python suite unchanged at 696 tests, OK, skipped=52. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught this: on Windows the host does not link backend_cuda.cu, it links backend_loader.c and resolves coli_cuda_* out of coli_cuda.dll at runtime. Every exported symbol is hand-wrapped there, and I added two without wrapping them, so `make colibri CUDA_DLL=1` failed to link: undefined reference to `coli_cuda_tensor_vram' (x6 call sites) RESOLVE_OPT, not RESOLVE, and that distinction is the whole point. RESOLVE prints "missing symbol" and calls FreeLibrary, so pairing a new host with a coli_cuda.dll built before #687 would take the ENTIRE CUDA backend down over one absent function - far worse than the over-commit this branch fixes. The e8_set_grid and fp8_set_lut entries above it set that precedent for exactly this reason and carry the same comment. The wrappers degrade to the old behaviour rather than to zero: tensor_vram -> tensor_bytes, the LOGICAL size, which is what the tier charged before this branch. An under-count, and identical to what shipped. Returning 0 would be much worse: the caller would charge nothing at all for an expert it just placed. alloc_footprint -> the request unchanged, i.e. no padding, same as before. Verified locally on the same two commands the CI job runs: make colibri CUDA_DLL=1 exit 0 (was: undefined ref) make cuda-dll CUDA_ARCH=sm_80 exit 0 and both symbols are in the DLL's PE export table beside tensor_bytes: [ 0] coli_cuda_alloc_footprint [47] coli_cuda_tensor_bytes [53] coli_cuda_tensor_vram Negative control run rather than assumed: reverting backend_loader.c alone reproduces the six undefined references, restoring it links clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_abi_is_derived_from_the_loader_source pins the loader's export surface by count, and it caught this branch: 5 optional against an asserted 3. The test's own docstring says what to do - update the counts and name the symbol you added, so the next person reading a failure gets a reason rather than a different integer - so both, and the reason is the RESOLVE_OPT choice rather than the symbols themselves. mandatory 47 -> 47 unchanged, neither symbol is required optional 3 -> 5 exports 50 -> 52 Mandatory is the number that matters here and it deliberately did not move. coli_cuda_tensor_vram and coli_cuda_alloc_footprint are resolved with RESOLVE_OPT, so a coli_cuda.dll built before #687 leaves both pointers NULL and the wrappers fall back to the logical byte count - exactly what the expert tier charged before this branch. Adding them as mandatory would have widened the contract every Windows DLL must satisfy and taken the entire CUDA backend down on any older DLL, over a sizing refinement. Python suite 707 tests, OK, skipped=50. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mw_ur3_raw_adapter.py parses a raw capture of the engine's batch wire protocol and re-checks it against a frozen corpus: frame parsing for the profile, done, echo and data kinds and for the group frames, with every malformed field a named error carrying the frame's byte offset and every unknown kind refused by name; identity proof, in which the request digest is checked against the submitted bytes and the corpus, the token payload digest is recomputed from the frozen tokenizer, and a capture that omits an authority fails closed; and the longest-common-prefix and log-likelihood-ratio helpers, pinned by hand-computed literals including a direction-sensitive case, so a reversed comparison cannot pass. Where a capture contains none of the frame kinds a check family needs -- which is every capture today's engine produces for some families -- the adapter reports that family as UNSUPPORTED by name and never degrades it to a partial pass. The process exit statuses are documented in the module docstring exactly as main implements them. The three real-authority digests the module ships are pinned against literals stored in the test, loaded through an unpatched copy of the module so the check is independent of the synthetic-authority patch the rest of the suite runs under. The one test that needs the real frozen corpus is opt-in through MW_UR3_CORPUS_DIR and names that variable in its skip reason; every other test in the module is self-contained. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…latform The raw adapter treats the corpus and comparisons authorities as byte-exact JSONL and rejects a CRLF record as noncanonical. The test module wrote those two files with write_text() and no newline="", so on Windows text mode translated every "\n" to "\r\n" and 41 tests failed at corpus load with `corpus:1: noncanonical JSONL record` (reproduced on macOS by forcing CRLF into the same two writers: the identical 40 failures + 1 error). The other byte-sensitive writers in the module already passed newline=""; these two now do as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n leak they found Stacked on #1344. Its three regression tests each pin one scenario; this adds the rules those scenarios are instances of, checked directly against tests/qwen36_fake_cuda.h in the plain CPU build: 1. Budget accounting balances. On every device, bytes in use never exceed the budget and, once the queue is drained, equal exactly resident experts x bytes per expert: every reservation is consumed by an upload or handed back. Checked on one and two devices, across an LFRU swap (budget-neutral by construction, so `used` must not move), and across the path where a planned expert is reported without weights. 2. Shutdown wakes every waiter at once. All four cv_take sleepers -- the uploader's victim wait, qt_note_block, qt_note_planned, qt_fill_wait -- are parked simultaneously behind a full queue and an open group; qt_shutdown has to bring every one of them home, under a watchdog thread rather than alarm() so it runs on MinGW too. Afterwards no slot may still read as queued and the abandoned swaps' victim keeps its tensor and its resident flag. 3. Issue geometry under random routing. Random resident sets, random K up to the row limit, one to three devices, 200 seeds each: every device block inside the replica buffer, pairwise disjoint, at its device's slot; the mask names exactly the routed experts that were resident on a device whose issue succeeded; hits + misses add up to everything routed. The row limit is read from the array the rows index (sizeof G.is_k[0] / sizeof G.is_k[0][0]), so the stride cannot drift from it without failing here. Under test-asan this is a fuzz for the #1339 class. ## What the first rule found qt_plan_fill reserves budget and sets planned=1; qt_note_planned returned early on NULL weights without undoing either, and the warmstart did not call it at all when the loader came back empty. The bytes stayed out of the budget for the life of the process and "if(resident||queued||planned) continue" never reconsidered the expert. #1331 was this leak for every expert of an int8 container; the class survived its fix. Fix: qt_note_planned hands the reservation back when it receives no weights, and tier_warmstart reports every planned expert, with or without them. Without the fix the new test fails eight checks (all this leak) and passes clean under ASan; with it, all green. ## Verified - test_qwen36_tier_invariants: ok; under ASan+UBSan: 0 diagnostics - without the fix: 8 FAIL, 0 sanitizer diagnostics (a red test, not a crash) - the four #1344 tests, test_qwen36_ctx and the qwen36 build unchanged: ok - one lesson kept in the file: the watchdog's first draft passed its timeout by pointer into the arming function's frame -- ASan flagged the stack-use-after-return (#1277's class) in the test itself before it could flag anything in the tier Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
qt_fill_wait() returns when the queue is empty, but the expert the uploader
dequeued last is still queued=1 until its upload returns. The residency
check right after it raced that upload and failed about one run in fifteen
locally ("expert did not become resident during warmstart") -- a red that
says nothing about #1339. Poll until no slot is queued before asserting.
The same two-line wait is what test_qwen36_tier_invariants uses
(WAIT_IDLE). Noted on #1344 as review feedback; carried here so the
stacked PRs stop rolling dice in CI. Drop this commit if #1344 takes the
fix first.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
The lm_head GEMV (248320x2048, one call per token at token END, outside the per-layer latency chain) cost 13.8-14.8 ms/token on CPU — 18% of decode. Park the dense-i8 rows + per-row scales on a CUDA device and run them through coli_cuda_matmul (fmt=1, identical y[o]=acc*sc[o] semantics; greedy output bit-identical across all runs). qt_init removes the lm_head device from the expert PLACEMENT list (not just budget=0: home() would still hash experts onto it and the hit rate collapsed 89.9->49.5% in the first attempt) while keeping its CUDA context; if it is the only device, experts stay and lm_head shares it. Measured (gs64, heat-warm, 200 tokens): CPU lm_head 14.8 ms/tok 81.0 total 11.05 tok/s lm_head on Quadro 9.9 ms/tok 76.2 total 11.69 tok/s lm_head on 3070 3.9 ms/tok 72.7 total 12.21 tok/s <- best, single GPU (experts + lm_head share the 3070; token end is idle time) (cherry picked from commit 734e1e3d77344e3bc9613d6fbc6c752746b6a518)
(cherry picked from commit 4064493, ohne die versehentlich committeten Binaries c/qwen36 und c/tools/bench_dnproj; qt_init-Signatur und Device-Auswahl auf den #1344-Stand aufgeloest: die COLI_PLACE-Devices werden nach der COLI_GPUS-/Auto-Auswahl und vor der Leer-Pruefung ergaenzt. tests/qwen36_fake_cuda.h bekommt coli_cuda_matmul als aufzeichnenden Stub, damit die Tier-Tests weiter im CPU-Build linken.)
The hand-written COLI_PLACE list from R4 is a measurement tool; nobody running a 6 GB card should have to work out that 1.2 GB of dense trunk is worth more than 800 experts (#1040). The engine knows every size involved, so it decides. Rule: bytes saved on the memory bus per token, per byte of VRAM spent. Dense weights are read every token (value 1.0 per byte); an expert with the probability p_e that a token routes to it -- its heat share when a HEAT_FILE exists, topk/n_experts otherwise -- and the CPU fallback reads the int8 slot, twice the VRAM bytes of an int4 expert (value 2*p_e; 1*p_e on an int8 container). A trunk item goes to the device with the most room if its value beats that of the coldest experts it would push out, i.e. the tail of the heat order that still fits today. Without heat the tail is worth 2*topk/n_experts per byte and the trunk always wins; with heat, a card whose marginal expert is routed on more than every second token keeps its experts -- the R4 measurement, where moving all projections onto one near-full card cost 0.7 GB of hot experts and the 13 ms it had saved. Mechanics: - qt_trunk_offer(component, layer, bytes) before qt_init: the engine offers lmhead once and dnproj per DeltaNet layer, sizes from the same dense-i8 entries the uploads use. No entries (dense-i8 off) -> no offer. - qt_init prices and places, writes the decision into the table qt_place_of() already reads, and subtracts placed bytes from that device's expert budget. The explicit list gets the same subtraction now; with COLI_PLACE="dnproj=0" the projections used to come out of the expert cache unannounced. - COLI_PLACE unset or "auto" -> automatic; a list -> obeyed, auto off; "off" -> nothing placed. In auto mode the R4 role-split reservation is skipped on purpose: one card shares trunk and experts. - Weight format, exp_bytes and the heat table move ahead of the role-split block in qt_init, and the parse state is re-derived per init. tests/test_qwen36_tier_autoplace.c on the fake backend (free VRAM is what the test says): fits -> placed and budget shrinks by exactly the trunk; too big -> CPU; heat verdicts both ways at p_marginal 0.67 and 0.17, with and without a heat file, and "room nobody would use goes to the trunk"; explicit list obeyed and charged; off; two devices greedy by room with per-device budgets; nothing offered -> no-op. All six tier tests green, ASan/UBSan clean on this and the invariants test, CPU and CUDA builds ok. Not yet: dnout/attnproj (offered components other than lmhead/dnproj are ignored), and the calibration on real cards that decides whether the factor 2 and the greedy-by-room device choice survive contact with a Quadro/3070 pair and a 6 GB card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…with the first calibration Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
cudaGetDeviceProperties says ordinal 0 is the RTX 3070 and ordinal 1 the Quadro RTX 4000 on the calibration box (CUDA orders fastest-first, nvidia-smi by PCI bus, and the two disagree there). The rows labelled "RTX 3070" were measured on the Quadro and the row labelled "Quadro" on the 3070. Numbers unchanged; names swapped; the sentence that explained the Quadro being the faster card is gone, because it was the 3070. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
qwen36 keeps every expert in RAM and lets the tier retain raw pointers into slots that are never recycled; `cap == n_experts` guards exactly that. Qwen3.8 streams 24 576 experts of 4.7 MiB through a per-layer LRU whose slots ARE recycled, so a retained pointer would dangle by the next token, and 121 GB of experts do not fit beside the model anyway. qt_init_fp8(nl, ne, D, Ih, cap, topk, e4m3_lut) starts the tier in a mode that owns what it uploads: - cap may be smaller than n_experts; - weight format 8: e4m3 bytes as they came from the checkpoint, one f32 scale per 128x128 block per matrix -- the layout #817's kernels and tensor_upload(fmt=8) already agree on, so no kernel changes; the decode table is published to the backend at init and the tier refuses to start without it; - staging copies the bytes unchanged (no XOR) and the three block-scale tables, inside the qt_note call while the engine's slot is live, and the slot pointers are dropped before qt_note returns; - therefore a promotion can only happen when the bytes pass by: the LFRU decision moves from the periodic tick into qt_note -- a non-resident expert noted on a full device evicts the coldest resident there when the shared admission rule says it is hotter, as a budget-neutral swap; the tick keeps only the decay. The int4/int8 modes are untouched: same code paths, same `cap == n_experts` refusal, the six existing tier tests unchanged and green. tests/test_qwen36_tier_fp8.c on the fake backend (which now counts fmt 8 at one byte per element and stubs the LUT publish): starts with cap 4 of 16, exp_bytes = 3*D*Ih + three block tables + slack, three fmt=8 uploads per expert with the bytes intact, no pointer survives a note, a cold newcomer is refused on a full device, a hot one is swapped in budget-neutrally, hits and misses count, shutdown clears the mode, a missing LUT refuses the mode, the int4 mode still refuses cap < n_experts. Clean under ASan/UBSan, as are the invariants and placement tests; CPU and CUDA builds of qwen36 ok. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
Budget accounting. The tier charged an expert by its payload. cudaMalloc rounds an allocation above 1 MiB up to a multiple of 2 MiB and one at or below 1 MiB up to 1 MiB (measured with cudaMemGetInfo: 1,638,400 B -> 2 MiB, 819,200 B -> 1 MiB, 400 B -> 10 KiB). In the fp8 streaming mode an expert of three 1.56 MiB matrices was charged 4.69 MiB and took 6.03 MiB, so a 6.5 GB budget filled an 8 GB card to the last megabyte: eight "tensor allocation: out of memory" lines before the uploader's stop-trying fallback shrank the budget to what had fit (1,279 experts, 7.77 GB on the card). exp_bytes is now three weight footprints plus three scale footprints, so the planned count is the resident count. Qwen3.6 is not affected: its int4 matrices are exactly 512 KiB and the allocator serves those exactly, so the payload accounting was already right there. Recovering the fp8 loss needs an arena per device; not done here. COLI_GPU. resource_plan.py writes COLI_GPU for a one-device plan and colibri.c reads it; the tier only read COLI_GPUS, so a `coli chat --gpu 1` landed on every visible device. The tier now takes COLI_GPU when COLI_GPUS is unset. Tests: the two tier tests that compute exp_bytes use the tier's own dev_alloc_footprint. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…d CUDA threads
With OMP_PROC_BIND set, libgomp binds the initial thread to place 0 before
main() runs, and a pthread inherits the CPU mask of the thread that creates
it. The tier's uploader thread and the CUDA runtime's own threads were thus
jailed on the OpenMP master's core: every staging copy and driver call
competed with the master's share of each expert matmul, and the whole team
waited for it. On Qwen3.8 (cap 128, one 8 GB card, 12 threads) the CPU time
per remaining expert rose 64 % while the tier was on -- 3.7 ms against
2.25 ms -- which ate the whole gain of computing 45 % (one card) to 59 %
(two cards) of the routed experts on the GPU: 0.32/0.31 tok/s against
0.32 on the CPU alone. Fewer OpenMP threads did not help, because the
contention is on the master's core whatever the team size.
qt_init now widens the calling thread's mask to every online CPU while it
initializes CUDA and creates the uploader, and restores the caller's mask
afterwards. Raw sched_{get,set}affinity syscalls, Linux only, no _GNU_SOURCE
(the file is #included by tests after the engine's headers). The uploader
records how many CPUs it may run on; test_qwen36_tier_fp8 binds its own
thread to CPU 0 the way libgomp would, starts the tier, and checks that the
uploader is not jailed and that the caller's mask comes back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…urve The first footprint rule charged every allocation at or below 1 MiB a full 1 MiB. That is what a 768 KiB request takes, but not what Qwen3.6's 512 KiB int4 matrices or its 64 KiB gs64 scale tables take: measured over 256 allocations each, small requests round to 8 KiB steps with roughly a sixteenth of overhead (96 KiB -> 104 KiB, 384 KiB -> 416 KiB), requests above 512 KiB take one 1 MiB page, and above 1 MiB multiples of 2 MiB. With the coarse rule a Qwen3.6 gs64 expert was charged 6 MiB instead of its 1.7 MiB and a 4.9 GiB budget planned 836 experts where 2,800 fit (measured: 2,782 resident after the change). The fp8 Qwen3.8 expert is unchanged (three 2 MiB matrices plus three 8 KiB scale tables). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
…tor footprint test_qwen36_tier_int8 compared exp_bytes against the payload formulas and test_qwen36_tier_shutdown sized its one-expert budget from them; both broke when exp_bytes started charging what cudaMalloc takes. They now use the tier's dev_alloc_footprint like the other tier tests. CI caught it; I had run only the five tests I had touched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
test(qwen36): tier invariants on the fake backend, and the reservation leak they found
feat(qwen36): the dense trunk places itself -- COLI_PLACE=auto, priced in bytes saved per byte of VRAM
qwen36 tier: fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU
feat(tools): raw-evidence adapter for the scoring corpus
feat(experiments): validate reproducible performance records
…mmit limit (#1375) glm53.c sized its expert cache from /proc/meminfo. That file exists on Linux only: on Windows and macOS the function returned 0, the budget clamped to 1 GB, and GLM-5.3-Flash ran with one expert slot per layer, silently. colibri.c had the correct measurement (host_statistics64 on macOS, GlobalMemoryStatusEx on Windows) as a function of its own. Two copies, one wrong. The measurement now lives once, in compat.h, and both engines call it. On Windows the number that decides whether the next malloc succeeds is not free physical memory but grantable commit (ullAvailPageFile), which a small page file makes far lower. The shared function returns the smaller of the two. The planner does the same: its MEMORYSTATUSEX declaration skipped the two PageFile fields entirely, so ullAvailPhys was read at the right offset by luck and the commit figure was never looked at. It budgeted 88% of physical, handed the resulting cap to the engine as an argument, and the engine filled it slot by slot until Windows refused the next 14 MB on a 128 GB machine. That is #1375. coli tune now prints the cap each run uses, and the engine's OOM line says the slot size and how to shrink the cache. Tests: test_mem_available.c runs in the Linux, macOS and Windows jobs and fails wherever the measurement is 0 or disagrees with the system's own figure; forcing the measurement to 0 makes it fail. Three planner tests pin the min(physical, commit) rule and the struct layout.
fix(memory): measure available RAM on every platform, and mind the commit limit (#1375)
…and reported with the right number (#1376) Reported: coli web --ctx 65536 on qwen36 accepted the flag, printed a banner with every capacity knob except the context, and the first sign that the ceiling was 8192 was a CONTEXT_EXCEEDED refusal. Reproduced the two halves that are defects on dev; the third, the flag not reaching the engine, does not reproduce: coli serve --ctx 20 on the qwen36 fixture refuses a 30-token prompt and without the flag accepts it, so Q36_MAXT is delivered. The report may come from a build before the family-owned context mapping (2026-08-16); the banner now makes that self-diagnosing. 1. qwen36's startup banner prints ctx=<effective ceiling> next to cache, bits and the rest, so a clamp to the hard limit is visible at start rather than at the first failed request. 2. The server's CONTEXT_EXCEEDED message read the engine's key=value fields by position: fields[2] (requested=M, the completion budget) as the limit, printed raw. Users read "maximum context length is requested=4 tokens" and the real ceiling appeared nowhere. It now parses the fields and reports capacity as the limit and prompt_tokens as the usage. 3. coli refuses --ctx above the family's max_context with the number, instead of passing it on to be clamped in silence. A flag that appears to work and does not is worse than one refused. Tests: the message parser with the real engine line; the refusal above max_context and the pass-through below it. Reverting each fix fails its test. Banner verified on the fixture at default (8192), Q36_MAXT=20 (20) and an absurd value (262144, the clamp now visible).
fix(coli): preserve explicit Windows expert budgets
fix(glm53): release dashboard hit tracking
…res-20260911 test(planner): avoid materializing zero-filled shard payloads
tests: include compat.h directly in the converted tests
…m-glm53 feat(web): drop redundant Medium from GLM 5.3 reasoning selector
dev went red on `Windows UCRT64` / `make check` right after #1390 landed: tests/test_qwen36_tier_fill_wait.c:42:5: error: implicit declaration of function 'setenv'; did you mean 'getenv'? MinGW has no setenv. The tests that include an engine .c inherit compat.h through it, but this one includes qwen36_tier.c, which does not pull it in, so the declaration was never there. Linux never noticed because glibc has setenv, and modern GCC turns an implicit declaration into an error rather than a warning, so the Windows leg is where it surfaced. Same shape as #1440, which added the include to the six tests that already had it. Reproduced and fixed against mingw-w64 with the Makefile's own Windows flags plus -Werror=implicit-function-declaration, then every other gated test was cross-compiled the same way: this file is the only one affected. The three that still fail that sweep (test_uring, test_deepseek_v4, test_v4_ownership) are in TEST_EXCLUDE and are not built on the Windows leg at all. compat.h joins the rule's prerequisites too, so editing the shim relinks the test instead of leaving a stale one.
fix(tests): the qwen36 fill-wait test needs compat.h to build on Windows
metal: fix fmt=0 (raw f32) NULL-scale segfault in coli_metal_matmul
fix(mirror verify): support complete mirrors without a receipt
fix: enable qwen36 GPU tier when building with HIP=1
fix: README.zh-CN formatting error
Serve mode emitted five literal zeros for the PROF phase timings, and the gateway stores them verbatim into the rolling /profile window. A consumer could not tell an unmeasured field from a measured zero, so `expert_disk_s` read 0.0 on exactly the turns where the expert reads were the workload (#1449, filed off the cap matrix in #1441). The disk phase is now measured. `load_expert_merged` is the one place olmoe touches the container for an expert, so timing it is enough; the reads run unlocked and in parallel, so the accumulator is an atomic nanosecond counter and the PROF line reports a per-turn delta of it. Demonstrated on the tiny fixture, serve mode, cap 1 with EXPERT_DROP=1 so the reads actually go to the device: PROF 0.026 83 12 0.020 0.0 0.0 0.0 0.0 13 0.020 s of a 0.026 s turn, which is what a cap-1 streaming turn spends on disk. Before this the same run reported 0.0. The other four fields stay zero because they are still unmeasured: olmoe does not split the rest of its wall time the way glm.c and inkling.c do, and a guessed number would be worse than a zero. That part of #1449 stays open. Oracle still token-exact (8/8), make test-c and the python suite green.
…its cap
`coli cluster worker` without an explicit `--cap` could never start:
cache/layer: expected a whole number, got "None"
`--cap` comes from the shared parser and defaults to `None`, meaning "auto".
Every other direct-engine launcher resolves that through `cap_for_launch`;
this one stringified it into argv, and `str(None)` is the literal "None",
which the engine's argument check refuses before anything else happens. The
documented default was therefore the one value that could not work (#1452),
on Linux and Windows alike.
Fallback 0 is what the other GLM launch sites pass: colibri.c treats a zero
cap as "not given" and resolves its own, after the Metal and SSD probes that
the platform default depends on.
tests/test_env_defaults.py gains the regression: the worker's argv must parse
as an integer, and an explicit --cap must still pass through untouched. Put
the old line back and the first of the two fails.
fix(olmoe): measure the expert disk phase instead of reporting a zero
fix(coli): the cluster worker handed the engine the string "None" as its cap
feat(glm53): add Metal routed MoE acceleration
fix(build): clean up Clang compiler warnings
…anup holds #1458 silenced -Wmissing-field-initializers on the three expert_mats initialisers by naming the `vk` field. #1462 landed alongside it and gave Mat two more trailing fields, `resident` and `metal`, so the warning did not go away, it moved: glm53.c:1411:11: warning: missing initializer for field 'resident' of 'Mat' The zero the compiler was already supplying is the right value -- a streamed expert slot is not resident and has no Metal tensor -- so this only says so explicitly. glm53 now builds clean under -Wextra again.
Version to 1.11.0 in the six places it lives (four READMEs, c/version.py, site/index.html) and a CHANGELOG entry for the 54 pull requests merged since v1.10.2, headed by the ninth engine, DeepSeek V4.1 Flash. The family count in the READMEs said eight while the enumeration next to it already named nine (V4.1 was added to the list by #1453 without the number following), and the site's meta descriptions still said six. All say nine. Every PR number cited in the entry was checked against `git log --merges v1.10.2..dev`: all 54 post-tag merges are cited, and none from before the tag. The heading keeps the file's `## [x.y.z] — date` form because release.yml extracts the notes by matching it; the body carries no em dashes or emoji, since it ships verbatim as the release notes.
feat(glm53): add routing usage telemetry and .coli_usage support
fix(glm53): correct RAM sizing and Apple unified-memory planning
release: 1.11.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 1.11.0. Fifty-six pull requests since v1.10.2: a ninth model family, five real bugs closed across four engines, and the two platforms the C tests never built on (Windows UCRT64, musl) now building them in CI.
Merge with
--admin(main is protected), then tagv1.11.0and push the tag:release.ymlbuilds the three archives on the tag and takes the notes from the## [1.11.0]section of the CHANGELOG. This time the Windows archive carriesdeepseek_v41.exe.The ninth engine: DeepSeek V4.1 Flash (#1453)
552B, 510 GB on disk, streamed from an SSD on a CPU box, no conversion: the released checkpoint is read natively. Engram, the DSA indexer, hyper-connections, the compressor, the 32-layer vision tower, DSpark speculative decoding and DSML tool calling are all in. Held token-exact in CI against a torch-only CPU reference at three cache capacities and under every speculative mode; vision matched to 5e-06.
Measured cold on the released checkpoint: a turn went from 78.7 s to 25.1 s during the work (0.305 to 0.957 tok/s), every step bit-exact against what it replaced. A five-turn chat runs at 1.14 to 1.58 tok/s. Verified end to end through
coli serve(text andimage_url) andcoli chaton the real checkpoint before this PR was opened. Two ways of hiding the expert reads were built, measured worse, and removed; both are documented with the numbers.If you hit one of these on 1.10.2, this is the fix
qt_fill_waitreturned on an empty queue, but the uploader frees the ring slot before the backend copies. Fixed by counting what is in flight (fix(qwen36 tier): qt_fill_wait returns only when the last upload is resident #1390, @crichalchemist).fmt=0has no scale array and the matmul wrapped a NULL pointer. Kernel and sizing fixed, plus a fail-closed guard.coli cluster workercould never start without--cap([Bug]: [MMAP] expert = viste zero-copy nei file (page cache = cache) when running in cluster #1452): the defaultNonereached the engine as the string"None"(fix(coli): the cluster worker handed the engine the string "None" as its cap #1461).--ramwas inert and "no--cap" meant a constant eight slots (fix(olmoe): --ram was inert, and "no --cap" meant a constant eight slots #1445). A 2,291-token prompt went from 675 s to 353 s with no flags on the reporter's box.coli tunewent OOM on GLM-5.3-Flash on Windows ([Bug]: GLM 5.3-Flash puts Coli Tune OOM in 1.10.2 #1375): available RAM is measured on every platform now (fix(memory): measure available RAM on every platform, and mind the commit limit (#1375) #1377)./profilereported zero disk time while the disk was the workload ([olmoe] serve-mode PROF hardcodes the five phase timings —expert_disk_sreads 0.0 while the experts are the workload #1449): the disk phase is measured (fix(olmoe): measure the expert disk phase instead of reporting a zero #1460).CANCELmid-turn (fix(glm53 serve): honor CANCEL while the turn is still running (#1332) #1414), the lazy HITS table race that segfaulted the parallel warmstart (engines: publish the lazy HITS table under a lock (the parallel warmstart segfaulted on first touch) #1423), explicitCUDA_EXPERT_GBoverwritten (fix(coli): preserve explicit Windows expert budgets #1421), the CUDA tier charged logical instead of real VRAM (fix(cuda): charge the expert tier real VRAM, not logical bytes (#687) #1244), and the single-GPU VRAM prefix starved by the RAM pin plan (cuda tier: size the VRAM prefix from the VRAM budget on a single GPU, dense trunk on a card that holds it (#1409) #1410, cuda tier: size the VRAM prefix against the measured headroom, not the requested budget (#1405) #1411).Added
COLI_METAL=1, feat(glm53): add Metal routed MoE acceleration #1462, @trigger2k20), CPU path unchanged when off; GLM-5.3 routing telemetry and.coli_usage(feat(glm53): add routing usage telemetry and .coli_usage support #1457); planner reports unified memory from the host and stops suggesting knobs glm53 ignores (fix(glm53): correct RAM sizing and Apple unified-memory planning #1456);coli mirror verifyaccepts a complete copy with no receipt (fix(mirror verify): support complete mirrors without a receipt #1447); qwen36 tier withHIP=1(fix: enable qwen36 GPU tier when building with HIP=1 #1454);TRUNK_RESIDENT_LAYERSmmap trunk streaming (feat(trunk): add TRUNK_RESIDENT_LAYERS=N mmap streaming for the dense trunk #1399, @SebaWag); the qwen36 dense trunk places itself (feat(qwen36): the dense trunk places itself -- COLI_PLACE=auto, priced in bytes saved per byte of VRAM #1361, qwen36 tier: fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU #1374, @kreuzzelg); Brain and Profile tabs on every engine; the portable Windows CUDA DLL built and published by CI (ci: build the portable Windows CUDA DLL and publish it as an artifact (#1405) #1407).setenvis visible togetenvin-process on Windows (fix(compat): setenv must be visible to getenv in the same process (Windows) #1428);malloc_trimguarded to glibc (fix(build): guard malloc_trim to glibc for musl libc compatibility #1435, @iiEliJas).Verified before opening
devat the head of this PR:make test-cgreen, the python suite green (883 tests), every engine builds, and the CHANGELOG cites exactly the 56 merges after the v1.10.2 tag, none from before it. Full detail inCHANGELOG.md.