fix(compaction): keep compact_now() Send + fix large-payload corruption (closes #3)#4
Merged
Merged
Conversation
compact_now() held a tracing::EnteredSpan guard across its .await points. EnteredSpan is !Send, so holding it across an await makes the returned future !Send — uncallable from a Send context (e.g. an #[async_trait] impl whose methods require Send). Replace the entered-span guard with tracing::Instrument so the span still covers the async work while the future stays Send. Behavior-preserving: same span name, fields, logging.
farhan-syah
force-pushed
the
fix/compact-now-send
branch
from
5453608 to
1312214
Compare
farhan-syah
changed the title
emanzx
added a commit
to emanzx/nodedb-lite
that referenced
this pull request
Jun 15, 2026
Add compact() to the StorageEngine trait (default no-op), overridden by the pagedb-backed engine to call pagedb::Db::compact_now() and map CompactStats to a lite-owned CompactionOutcome. Add a NodeDbLite::compact() forwarder. In-memory engine keeps the no-op default. Lets consumers reclaim deferred-free space / bound on-disk growth between writes. Requires pagedb compact_now() to be Send-callable (NodeDB-Lab/pagedb#4). Addresses NodeDB-Lab/pagedb#2 (compaction not exposed to consumers).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related compaction fixes:
compact_now()future staysSend(original PR) — swap theEnteredSpanguard fortracing::Instrument, so compaction can be called fromSendasync contexts (e.g. an#[async_trait]StorageEngineimpl).Closes #3 —
compact_now()corrupts store on large payloadsRoot cause: the dense repack rebuilt trees with
bulk_load, which could only inline values.collect_rangeresolves overflow chains back to inline bytes, so any value above the inline threshold (page_size/4) overflowed leaf capacity →payload too large, and the partial in-place rewrite then left pages whose AEAD tag wouldn't verify → store bricked.Fixes:
bulk_load— values past the inline threshold are spilled to overflow chains, exactly like theputpath (sharedoverflow::inline_value_threshold). A repack now reproduces the original storage shape instead of failing.main.db.compactand atomically renames it overmain.db. The live file is never modified until the rename, so a failure or crash before it leaves the original store fully intact (an orphaned scratch is cleaned up on the next open). This is the LMDB/SQLite-VACUUMmodel.compact_step— a full rewrite can't be safely chunked across writer-lock releases without dropping concurrent commits, socompact_stepnow performs the whole atomic compaction in one call and reports completion. The old fake-incremental relocation machinery (and its dead watermark codec) is removed. Sustained-write growth is already bounded continuously by the durable free-list, so reclaiming to the OS is a maintenance operation.Tests
Large/overflow-value compaction round-trips, a partial-compaction-then-commit reopen guard, and a crash-before-rename safety test. Full suite green; clippy and rustfmt clean.
Notes
main(which already carries the durable free-list work this depends on).