Skip to content

Harden MothCDC for the 0.7 release - #5

Merged
russellromney merged 9 commits into
mainfrom
agent/harden-0.7-release
Jul 12, 2026
Merged

russellromney merged 9 commits into
mainfrom
agent/harden-0.7-release

Conversation

@russellromney

@russellromney russellromney commented Jul 12, 2026

Copy link
Copy Markdown
Owner

What changed

  • fixes the no-SSE4.1 scalar-dispatch tail panic and adds a forced-scalar regression
  • makes streaming caterpillar grouping independent of Read fragmentation by carrying undecidable singleton candidates across refills
  • retries Interrupted, preserves state across reader errors, replaces the EOF sentinel, and adds checked/fallible reader construction
  • moves chunk/segment offsets, logical lengths, and represented chunk counts to u64
  • replaces publicly constructible Segment variants with a private validated representation and accessor API
  • documents and enforces the public Cdc splitpoint contract
  • validates the C API before entering chunking logic and builds staticlib only on explicit request
  • adds all-feature, formatting, MSRV 1.89, and i686 CI coverage
  • refreshes 0.7 docs, examples, metadata, and the consolidated changelog

Why

This is the release-hardening pass before announcing MothCDC publicly. The review found one hardware-specific panic, reader-fragmentation-dependent metadata behavior, unchecked 32-bit stream offsets, an unenforced extension-trait contract, avoidable downstream static-library codegen, and several untested error/configuration paths.

The streaming grouping bug came from emitting a singleton when its next boundary was not yet decidable. A Cursor filled enough buffer to discover the run, while a one-byte reader repeatedly emitted solos. The chunker now retains that singleton until the next refill proves whether the run continues.

Public API impact

This intentionally targets 0.7.0 and contains breaking changes appropriate for the minor-version bump:

  • Chunk::offset() and Segment::offset() return u64
  • Segment::len() and Segment::chunk_count() return u64
  • callers use Segment accessors such as is_caterpillar() and dedup_key() instead of matching public variants
  • ReadChunker and MothReadChunker add fallible constructors plus get_ref, get_mut, and into_inner
  • custom Cdc implementations that violate the documented splitpoint contract now panic immediately

Validation

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all-features
  • cargo test --release --all-features
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features
  • cargo +1.89.0 check --all-targets --all-features
  • checks for i686, x86-64 Linux, aarch64 Linux, and wasm32
  • x86-64 forced-scalar regression under Rosetta
  • cargo package --allow-dirty
  • explicit C static-library build
  • downstream consumer build confirmed only the Rust library crate type is compiled
  • README image, benchmark branch, and corpus links return successfully

The SIMD prefetch provenance fix was submitted separately upstream as orlp#1, including an x86-64 regression test.

Follow-up

Add scalar-only Miri and cargo-fuzz coverage after measuring their CI runtime/stability. I attempted to file this separately, but GitHub issues are disabled for this repository.

russellromney and others added 9 commits July 12, 2026 01:11
The shipped grouping-independence test only covered all-zeros. Add a
differential that asserts MothReadChunker produces exactly the slice
caterpillar's segment grouping (offset, len, count, unit bytes) across the
full corpus, four size configs, and eight reader fragmentations (including
1-byte reads), plus a run crossing the 4 MiB buffer. Verified to fail with
the singleton-carry fix reverted (16381 records vs 1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
check_all had dropped the Tier-5 reader-vs-slice differential, leaving the
plain ReadChunker fuzzed against the oracle only on a single fixed input.
Re-add it so the streaming buffer state machine is differentially checked on
every corpus and proptest case at five reader step sizes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match MothReadChunker::next: state that Ok(0) is end of input, Interrupted is
retried internally, and progress is preserved across other errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@russellromney
russellromney marked this pull request as ready for review July 12, 2026 12:09
@russellromney
russellromney merged commit 1f78ac1 into main Jul 12, 2026
5 checks passed
@russellromney
russellromney deleted the agent/harden-0.7-release branch July 12, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant