DSpark scheduler break-even model appears not to account for replay cost (Metal, net -1327ms despite 70% accept rate)

[datanerdie]

On Metal (M4 Max, Flash q2-q4, 0731 DSpark support GGUF, 6747e77), DSpark is consistently net-negative on generation:

corpus	baseline gen	--dspark	change
prose, 9485 tok	29.31 t/s	23.02 t/s	-21.5%
prose, 28790 tok	27.82 t/s	23.10 t/s	-17.0%
code, 9452 tok	29.53 t/s	22.90 t/s	-22.4%

Reproducible to 0.2% across reps; prefill is unaffected as expected.

This is not the ROCm no-draft failure in #597 — here DSpark proposes normally and its proposals are usually accepted. DS4_DSPARK_STATS=1, prose:

cycles=88 proposed=44 accepted_draft=31 accept_rate=70.45% avg_accept=0.352
full=14 partial=2 miss_first=7 no_draft=65 scheduler_skips=52
draft_len_hist=1:10,2:6,3:6,4:1   accepted_len_hist=0:72,1:6,2:5,3:5
time_ms propose=332.0 verify=964.4 replay=1063.1 spec_total=2039.9
        saved=1045.3  net_saved=-1326.6

The economics: each accepted token saves ~34 ms (one baseline token at 29.3 t/s), and 31 were accepted, giving saved=1045 ms. But replay=1063 ms over 44 proposals is ~24 ms per speculation cycle — roughly one full token. So a cycle must accept more than one token just to pay for its own replay, while avg_accept is 0.35.

The scheduler is clearly trying to avoid this: 52 of 88 cycles were skipped, and the no-draft pause logic fires. But it still ends up net-negative, which suggests its break-even model may not include the replay term — the largest single cost in the breakdown.

Two observations that may help localise it:

1. Code is worse than prose, contradicting the README guidance that predictable continuations benefit most: accept_rate 62.22% vs 70.45%, avg_accept 0.308 vs 0.352, 77/91 vs 72/88 cycles accepting zero. Corpus was a 9452-token prefix of ds4.c itself, length-matched to the prose prompt within 0.3%.
2. Drafts never exceed 4 tokens in either corpus, and --mtp-draft has no effect across 1/4/8 (22.90 / 22.94 / 22.94 t/s) — expected, since that flag drives the legacy MTP path, but worth stating so others do not try it as a workaround.

Not asking for the replay fix (7fb2830) to be reverted — it is a correctness fix, and removing it would only move net_saved from -1327 ms to roughly -264 ms, still a loss. The suggestion is narrower: if the scheduler counted replay in its break-even estimate, it could pause speculation earlier on low-yield content and converge to baseline instead of below it.

Happy to run experiments if useful — the telemetry above is easy to regenerate.

[Flor1an-B]

Independent confirmation from a different angle — this extends the finding to the best possible case for DSpark, not just prose/code.

Setup

M5 Max 128GB, current main (e0c63d9, includes the 7fb2830/af80694 replay fix), Flash q2-q4 (asymmetric mixed quant, ~98GB), official DeepSeek-V4-Flash-DSpark-support-0731.gguf, --dspark-confidence 0.7 (default).

Best-case content: maximally predictable numeric continuation

Greedy (--temp 0, reasoning_effort: none to rule out thinking-mode content), continuing an already-started 1, 2, 3, ... sequence to 300 — about as repetitive as real content gets, and specifically the kind of input the README singles out as DSpark's ideal case:

• Before this fix (measured 2026-07-20, same content type, older main): baseline 37.6 t/s → DSpark 89.3 t/s (+137%)
• After this fix (today): decode is flat, 31.67 t/s average, no ramp-up at all across the whole generation

So it's not just prose/code that went net-negative — even the textbook best case (which used to be DSpark's strongest argument) now sits below the non-speculative baseline (~37-40 t/s on this quant). Consistent with your read: the replay fix is correct and shouldn't be reverted, but the scheduler's break-even estimate not counting replay means it no longer has any regime where it clearly wins on this hardware.

Separate but related data point

Worth having in the same place: at real (non-greedy) sampling — temp=1.0 top_p=1.0, DeepSeek's own documented recommendation for this model — DSpark's accept rate collapses independently of the replay-cost issue, because the verify step compares drafts to sample_argmax(target) rather than a true sample of the configured distribution (discussed at length on #482). So on Metal today, DSpark doesn't have a currently-reachable win condition for anyone running the model at its recommended sampling settings, on top of the replay economics above.

Happy to run more configurations if useful (different --dspark-confidence values, block sizes, etc.) — I have the quant and sidecar already downloaded.

[Flor1an-B]

Tested the opt-in scheduler timing knobs mentioned in the code comments (DS4_DSPARK_SCHEDULER_MAX_EXTRA_SAVED_RATIO_MILLI, DS4_DSPARK_SCHEDULER_MAX_MS_PER_ACCEPT_MILLI) against my best-case content from above (the 1→300 numeric continuation) — sharing since the result is a bit counter-intuitive.

They exist (ds4.c, ds4_dspark_scheduler_max_ms_per_accept_milli() / ds4_dspark_scheduler_max_extra_saved_ratio_milli()) and are wired into the break-even check, but both default to 0 (disabled) — the comment on ds4_dspark_scheduler_timing_enabled() explains why: "Timing-sensitive scheduling changes which arithmetic path advances a token. Keep it opt-in so greedy DSpark output is reproducible across runs." So out of the box, the only active backoff signals are the count-based ones (low_accept, many_no_draft), not the wall-clock ones — which matches your read that the break-even model isn't really counting replay by default.

I turned them on (MAX_EXTRA_SAVED_RATIO_MILLI=1200, MAX_MS_PER_ACCEPT_MILLI=40) expecting the scheduler to back off and at least recover to baseline. It didn't — same flat ~32 t/s as with defaults, no measurable change. My reading: on this content the accept rate stays high enough that the count-based skip logic never fully disengages, so the timing knobs don't get much chance to act within an already-running sequence — and since replay cost is paid per accepted token rather than per cycle, a high accept rate doesn't dilute it the way it would dilute a fixed per-cycle overhead. If that's right, backing off earlier wouldn't recover the old speedup here even with the knobs on by default; it would just cap the downside on genuinely low-yield content, which is still valuable but is a narrower fix than the break-even model catching this case.

Not confident in that read — happy for someone to correct it — but wanted to flag that enabling the existing opt-in mechanism isn't a free win here, at least not with these threshold values.

[datanerdie]

Adding a code-level confirmation of @Flor1an-B's per-accepted-token reading, because it changes what I asked for in the original report.

Replay is one full target-model forward per accepted token

At 6747e77, accepted drafts are replayed in a sequential loop:

for (int i = 0; i < replay_budget; i++) {
    ok = metal_graph_eval_token_raw_swa(&s->graph, &e->model, &e->weights,
                                        drafts[i], (uint32_t)s->checkpoint.len,
                                        row_logits);
    if (!ok) { ... }
    token_vec_push(&s->checkpoint, drafts[i]);
    accepted[n_accept++] = drafts[i];
    replayed_drafts++;
}

(around ds4.c:61777 at that commit — I don't have af80694 locally, so the line may have moved.)

One metal_graph_eval_token_raw_swa per accepted token is one ordinary decode. The telemetry in my original report says the same thing numerically:

replay 1063.1 ms / 31 accepted   = 34.3 ms per accepted token
one baseline token at 29.31 t/s  = 34.1 ms
saved = 1045.3  vs  replay = 1063.1   -> within 1.7%

An accepted token costs almost exactly what accepting it saved, leaving propose (332 ms) and verify (964 ms) with nothing to pay them.

This makes my original framing too weak

I suggested the scheduler's break-even estimate should include the replay term. If savings and replay cancel by construction, that isn't sufficient: no threshold makes speculation profitable, and the best a perfect scheduler can do is skip every cycle and converge to baseline. That matches the flat 31.67 t/s @Flor1an-B measures on maximally predictable content, where the old behaviour was +137%.

It also explains why the opt-in timing knobs changed nothing: the dominant cost scales with accepts, not with cycles, so a high accept rate cannot dilute it the way it would dilute a fixed per-cycle overhead.

A direction that doesn't touch the correctness fix

Batch the replay — evaluate an accepted run in a single multi-token forward instead of k sequential ones, so k accepted tokens cost ~1 forward rather than k. At my avg_accept of 0.35 this would change little, but the long-draft regime is exactly where the old speedup came from, so it is the case that stands to recover.

There is batched machinery on the prefill side (layer_attention_raw_swa_batch, chunked prefill), though reusing it for a mid-sequence replay against a restored snapshot is clearly not a small change.

Minor: stale comment on the invariant under discussion

The comment on the speculative scratch fields still describes the pre-7fb2830 design:

Prefix slots retain the intermediate frontiers of a tiny verifier block so partial accepts do not need target-model replay.

(ds4.c ~15021.) That is no longer true after the partial-accept shortcut was removed.

Setup otherwise unchanged from the original report (M4 Max, Flash q2-q4, 0731 DSpark GGUF). Happy to regenerate any of the telemetry.

[Flor1an-B]

This is a much sharper framing than what I had — thanks for tracing it through the actual replay loop and the arithmetic. "Savings and replay cancel by construction at the per-accepted-token level" is a real result, not just an observation from one dataset; it explains why my MAX_EXTRA_SAVED_RATIO_MILLI/MAX_MS_PER_ACCEPT_MILLI experiment did nothing, cleanly, rather than as a coincidence.

The batched-replay direction sounds right to me as the actual fix, for what my opinion's worth: it targets the thing that's actually broken (cost per accepted token) instead of working around it with scheduling, and it's the one change that could get back to something like the old long-draft-run economics rather than just capping the downside.

I don't have the Metal graph/kernel familiarity to responsibly attempt "batch an accepted run against a restored snapshot" myself — that's squarely your and @af80694-adjacent territory, not mine. Two things I can usefully do if helpful: (1) keep generating telemetry/A-B runs on an M5 Max across content types as you iterate, since hardware-specific replay/verify timing seems to matter here, and (2) I'll leave the stale comment fix to whoever picks this up rather than send a one-line patch that'll conflict with real work in flight. Ping me if a build to test against would save you time.

[alangiu-gif]

Independent confirmation on a second M4 Max, with a different drafter — plus one data point that I think strengthens your argument, and one that bounds the fix you propose.

The headline: at 90.7% accept rate it is still 23.8% slower than baseline. Your case was 70.45%, and the natural objection to it is "acceptance is just too low here". On a high-acceptance prompt the loss does not go away:

prompt	accept_rate	baseline	--dspark	change
counting (1..120)	90.70%	32.53 t/s	24.80 t/s	−23.8%
prose (~200 words)	78.10%	32.51 t/s	22.44 t/s	−31.0%

The counting run proposes 86 times and gets 78 drafts accepted, and still loses a quarter of the throughput. That is consistent with your reading that the dominant term is per-cycle and not per-rejection.

On the proposal — counting replay in the break-even so the scheduler pauses earlier. The direction is supported, and here is a slope for it. Same binary, same prompt, three configurations ordered by how much they pause:

config	scheduler_skips / cycles	counting	prose
default	113/173 · 68/140	24.80	22.44
DS4_DSPARK_SCHEDULER_NO_DRAFT_SKIP=0	56/138 · 51/143	22.63	22.04
DS4_DSPARK_SCHEDULER=0	0	20.68	20.15

Monotone on both prompts: the more the scheduler pauses, the faster it runs. A break-even model that pauses more would move throughput the right way.

But the same table bounds the gain. The most aggressive configuration that exists today already skips 65% of cycles on the counting prompt and still lands 23.8% under baseline. For the scheduler to converge to baseline rather than below it, it would have to pause on essentially every cycle for this content — that is, the fix converges to "do not speculate here". That may well be the honest answer for low-bit main models, but it is a different change from correcting the estimate, and worth separating in the discussion.

Protocol, since this machine has a fault that ruins roughly one run in three: 5 replicas per arm; a cell is discarded when a single stall over 40 ms appears in the per-token execute profile or when prop_setup exceeds 200 ms (clean cells sit at 32 ms and 20-35 ms respectively, so the thresholds have an order of magnitude of headroom); the value is the max over surviving cells, because the fault is one-sided and the run is otherwise deterministic — surviving cells agree within 0.14%, and scheduler_skips is bit-identical across replicas of the same arm. Verified separately that DS4_METAL_GRAPH_TOKEN_PROFILE=1 does not perturb throughput (32.48 vs 32.52 with it off).

Environment: MacBook Pro M4 Max 128 GB, macOS 26.6, Metal, b030961, DeepSeek-V4-Flash-...-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf (80.7 GiB, fully resident), --temp 0 -n 256, short prompts (hence baselines higher than yours).

One caveat worth stating: my drafter is an 11.5 GiB DSpark support GGUF with tensors renamed to the mtp.<N>. convention, not the 5.6 GiB 0731 support file used in #676. It loads with missing=0 invalid=0. So this is a different drafter on different content reaching your conclusion — which at least rules the drafter out as the cause — but I have not checked whether it changes the numerics of the verify path.

Also relevant to anyone re-running older comparisons: before 7fb2830, --dspark did not reproduce the plain-greedy token stream, and on the prose prompt the four scheduler configurations above produced four different texts. Any t/s comparison taken before that commit is comparing different work. Details in #658.

[datanerdie]

Thanks — the 90.7% run is the data point this needed, and it corrects my original ask.

It closes the objection from the opposite end. My case was 70.45% acceptance, where the fair
pushback is "acceptance is simply too low here". Yours accepts 78 of 86 drafts and still loses a
quarter of throughput, so the loss survives across the whole acceptance range.

Putting the three reports together, the loss does not track acceptance at all:

report	accept_rate	change
mine, prose	70.45%	−21.5%
yours, prose	78.10%	−31.0%
yours, counting	90.70%	−23.8%

That non-monotonicity is itself the argument. If the dominant cost were per rejection, the
90.7% cell should be close to break-even. If savings and replay cancel per accepted token, then
acceptance buys nothing either way and what remains is propose + verify as pure overhead —
which is what all three cells show.

And my proposal was too optimistic — your sweep bounds it. The report says a replay-aware
break-even estimate "could pause speculation earlier on low-yield content and converge to
baseline instead of below it". The second half of that is wrong. Your most aggressive existing
configuration already skips 65% of cycles on the counting prompt and still lands 23.8% under
baseline, so for the estimate alone to reach baseline the scheduler would have to pause on
essentially every cycle. That is not a corrected estimate, it is a decision not to speculate on
this content.

So the two changes are worth keeping separate:

• Counting replay in the break-even stops the bleeding on low-yield content, and your
  monotone slope (24.80 → 22.63 → 20.68 as pausing decreases) shows it moves throughput the
  right way. Its ceiling is baseline-minus-epsilon, approached by speculating never.
• Batched replay — evaluating an accepted run against a restored snapshot rather than one
  metal_graph_eval_token_raw_swa per accepted token — is the only change that can put DSpark
  above baseline again, because it attacks the per-accepted-token cost itself.

The different drafter is more useful than it might look. You are on an 11.5 GiB support GGUF
with mtp.<N>. renamed tensors rather than the 5.6 GiB 0731 file, and you reach the same
conclusion — which rules out drafter mismatch as the cause. That is worth stating explicitly,
because a mismatched drafter is exactly what corrupted my own first measurement of this: the
legacy and 0731 support files are byte-identical in size (5,989,114,272) and differ only by
sha256, so nothing on disk flags the pairing, and it produced a plausible-looking −7.7% that I
had to throw away.

Your #658 note deserves more prominence than a footnote. Any --dspark throughput number
taken before 7fb2830 compares different generated text, so a fair amount of pre-existing
comparison in this area — including some of my own early numbers, though not the ones in this
report — is not measuring what it claims to. Worth flagging wherever those figures are still
quoted.