shuffle: detect and fail a Session deadlocked on disk back-pressure - #3497
Open
williamhbaker wants to merge 5 commits into
Open
williamhbaker wants to merge 5 commits into
williamhbaker wants to merge 5 commits into
Conversation
williamhbaker
force-pushed
the
wb/shuffle-disk-detector
branch
from
September 15, 2026 19:49
297851e to
9c8a00e
Compare
A Log stops draining Appends once its sealed segments reach the disk limit. That space is reclaimed by `Load` scans, which shards run only after the coordinator takes a checkpoint, and a checkpoint needs a flush cycle the parked Log may never complete. Nothing then advances. These messages let the Session observe that state. No actor sends or handles them yet.
`BackPressureState` must also record the flag last sent to each Slice, which is why the `serve` locals move into it. `DiskBackPressure` carries the current flag rather than the transition, so a Slice never sees a stale value, and a flag which engages and releases between two drains sends nothing at all. A Slice records what it is told and makes no use of it yet.
williamhbaker
force-pushed
the
wb/shuffle-disk-detector
branch
from
September 15, 2026 20:11
9c8a00e to
3f8153d
Compare
Each condition of the detector is ordinary on its own. Together they mean no shard will read, scan, or flush again, so the Log's disk backlog is never reclaimed. Failing the Session restarts the shards from their last checkpoint, which is the only way out. `on_blocked` resets the count as the response arrives rather than at the next tick, because a Slice can block and unblock within one tick interval. `would_emit` repeats the emission test of `take_ready` and must keep matching it. Nothing sends `Blocked` or `CaughtUp` yet, so the detector cannot fire.
A Slice holding flushed progress is never blocked, because the Session can still take a frontier from it. `try_slice_response_tx` therefore recomputes `blocked` after it sends `Progressed`: taking the flushed frontier is itself what can leave the Slice with nothing to report. `is_blocked` needs the index of the Log which parked the request stream, which is why the `Err` returns of the send helpers now carry it beside the sender. That part is mechanical.
`Head::Idle` is where a leader parks between transactions: it holds no ready frontier, waits on its checkpoint and nothing else, and no shard will reclaim Log disk space until the Session answers. Every other head state is driving IO of its own, so a leader in one hasn't yet exhausted its own work. A `CaughtUp` can cross its own frontier in flight, and the Session drops one which names a request it has already answered. Capture leaders hold no shuffle Session. This is the last input the detector needs.
williamhbaker
force-pushed
the
wb/shuffle-disk-detector
branch
from
September 15, 2026 20:25
3f8153d to
6c34848
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Disk back-pressure can deadlock a shuffle topology. A Log stops draining Appends once its sealed segments reach the disk limit. That space is reclaimed by
Loadscans, which shards run only after the coordinator takes a checkpoint, and a checkpoint needs a flush cycle the parked Log may never complete. The task then hangs until someone notices.A known source of this deadlock condition is sealed segments that are pinned as remainders due to a dangling producer CONTINUE_TXN message that was never acknowledged. A large number of these segments can fill the buffer.
This adds a detector which fails the Session instead, so that shards restart from their last checkpoint and read the backlog down. Three new messages carry the state the Session cannot otherwise see: a Log reports its back-pressure flag to each Slice, a Slice reports whether it has any progress left to report, and the coordinator reports that its outstanding checkpoint is the only thing it waits on. The Session fails once all of these hold together for five minutes, which is long enough to cover the Log's reclaim poll of up to sixty seconds.
Agentic Q/A completed on a local stack, testing various shard configurations under deadlock scenarios.
Closes #3498
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)