shuffle: use blocking reads for gapped-producer replays

[jgraettinger]

Summary

Gapped-producer replays read [offset, end_offset), where end_offset is the begin of a trigger document the main read already saw, so the whole range is known to exist. A broker that lags, though, may not have indexed it yet. A newly-assigned replica serves reads before it has listed a fragment still being persisted by its prior topology.

Replays used non-blocking reads, which fail against such a broker in one of two ways:

• OFFSET_NOT_YET_AVAILABLE: when the broker's write head is past the replay offset (e.g. it has newer local spool content). classify_read_failure treats this as terminal, so the task fails.
• A clean EOF short of end_offset: when the broker's write head is exactly where the replay has read to. The gazette client ends a non-blocking read at the write head, so the replay reports "complete" and silently skips [write_head, end_offset).

This switches replays to block: true. The broker waits until it has indexed the fragment, and the read still stops at end_offset.

[strix-security]

Strix Security Review

No security issues found.

Review summary

Reviewed the single-file diff in crates/shuffle/src/slice/replay.rs, which changes a bounded gapped-producer historical replay read from non-blocking to blocking (block: false → block: true) to prevent silent data skips and premature failures against lagging brokers. The change introduces no untrusted input, authorization, authentication, secret, or injection surface. The blocking read is bounded by end_offset, the range is guaranteed to exist, and the Gazette client surfaces stream/status errors through classify_read_failure for retry or terminal handling. No security issues were identified in the changed code.

_{Updated for f24c164.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}

[dgreer-dev]

LGTM.

Do you think there's any value in a regression test where the replay hits a stale write head, waits for a missing fragment to become available, and then verifies that all replayed documents are delivered exactly once? Should be red with the old non-blocking read and green with this change.

[jgraettinger]

Do you think there's any value in a regression test

Not especially. I think it would be a lot of test SLOC, and would predominantly be testing Gazette and client behavior -- behavior that's well covered under main-path reads. This change is fundamentally just making replay reads less special and like main-path blocking reads, but with an end offset, and we have existing coverage that the end-offset is respected.