Harden MothCDC for the 0.7 release - #5
Merged
Merged
Conversation
The shipped grouping-independence test only covered all-zeros. Add a differential that asserts MothReadChunker produces exactly the slice caterpillar's segment grouping (offset, len, count, unit bytes) across the full corpus, four size configs, and eight reader fragmentations (including 1-byte reads), plus a run crossing the 4 MiB buffer. Verified to fail with the singleton-carry fix reverted (16381 records vs 1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
check_all had dropped the Tier-5 reader-vs-slice differential, leaving the plain ReadChunker fuzzed against the oracle only on a single fixed input. Re-add it so the streaming buffer state machine is differentially checked on every corpus and proptest case at five reader step sizes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match MothReadChunker::next: state that Ok(0) is end of input, Interrupted is retried internally, and progress is preserved across other errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Readfragmentation by carrying undecidable singleton candidates across refillsInterrupted, preserves state across reader errors, replaces the EOF sentinel, and adds checked/fallible reader constructionu64Segmentvariants with a private validated representation and accessor APICdcsplitpoint contractstaticlibonly on explicit requestWhy
This is the release-hardening pass before announcing MothCDC publicly. The review found one hardware-specific panic, reader-fragmentation-dependent metadata behavior, unchecked 32-bit stream offsets, an unenforced extension-trait contract, avoidable downstream static-library codegen, and several untested error/configuration paths.
The streaming grouping bug came from emitting a singleton when its next boundary was not yet decidable. A
Cursorfilled enough buffer to discover the run, while a one-byte reader repeatedly emitted solos. The chunker now retains that singleton until the next refill proves whether the run continues.Public API impact
This intentionally targets 0.7.0 and contains breaking changes appropriate for the minor-version bump:
Chunk::offset()andSegment::offset()returnu64Segment::len()andSegment::chunk_count()returnu64Segmentaccessors such asis_caterpillar()anddedup_key()instead of matching public variantsReadChunkerandMothReadChunkeradd fallible constructors plusget_ref,get_mut, andinto_innerCdcimplementations that violate the documented splitpoint contract now panic immediatelyValidation
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo test --all-featurescargo test --release --all-featuresRUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-featurescargo +1.89.0 check --all-targets --all-featurescargo package --allow-dirtyThe SIMD prefetch provenance fix was submitted separately upstream as orlp#1, including an x86-64 regression test.
Follow-up
Add scalar-only Miri and cargo-fuzz coverage after measuring their CI runtime/stability. I attempted to file this separately, but GitHub issues are disabled for this repository.