Repository navigation
Conversation
Add AccessIntent (Interactive, BulkScan, Maintenance) to ravel-cache and intent-taking variants of the S3-FIFO get and insert, the RAM Cache get, insert, peek and get_or_fetch paths, and the TieredCache read, admit and resolve paths. The existing methods are Interactive wrappers, so no caller outside the crate changes and none passes another intent yet; the SQL executor sets BulkScan in #3084. Interactive reads keep today's S3-FIFO policy. Maintenance reads go to the small queue and are never promoted by their own hits or by a ghost hit. BulkScan reads never raise freq and never enter main: they are admitted to a bulk region holding up to bulk_share (default 0.5) of max_bytes against interactive admissions, and more while the interactive queues leave bytes unused. Each key has a fixed hash rank, and a full region admits a key only by evicting a higher-ranked resident, so a repeated scan larger than the region converges to its lowest-ranked keys and serves the same subset on later passes, whatever the read order. A per-tenant bottom-k sketch estimates the scan's distinct bytes, which sets a rank threshold and the window after which entries no read touched are evicted; idle tenants are retired. Bulk admissions evict small-queue entries only while the region is below its share and never evict main; interactive admissions reclaim region bytes above the share first. The region's bookkeeping is charged inside max_bytes and capped at 1% of it. The spec's first-pass "FIFO within the region" is replaced by rank-ordered eviction: with FIFO eviction a loop whose admitted subset exceeds the region by any margin serves nothing on an identical-order pass and a changing set when the order changes. The disk tier keeps the interactive policy for every read; a disk hit is re-admitted to RAM under the request's intent. CacheMetrics gains per-intent hits and bytes served and a bookkeeping gauge, read through intent_snapshot(), since CacheMetricsSnapshot is built as a struct literal in ravel-server's tests. The caching guide now describes the caps as S3-FIFO rather than LRU (#2738) and documents the intent policy. Refs: #3083 Refs: #3072 Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
@claude-fleet review |
|
Review task queued. Task: 1559b0f9-eb7e-4866-88bc-06f740675b1a |
There was a problem hiding this comment.
Request changes: one should_fix finding on the admission path the feature is built for.
Findings:
- should_fix, crates/ravel-cache/src/cache.rs:514 — a declined bulk admission still pays
owned_exact, copying the whole payload and counting it as an admission copy - nit, crates/ravel-cache/src/s3fifo.rs:447 — a
Maintenance/BulkScanadmission leaves the key inghost_set, soghostandghost_setcan drift - nit, crates/ravel-cache/src/lib.rs:149 —
AccessIntent::as_strhas no caller in the workspace
Verified:
- Interactive behaviour is unchanged with an empty region:
charged_bytes()equalstotal_bytes()whenbookkeeping_bytes()is 0 — s3fifo.rs:338-345, 820;promote_to_mainadds a term that is true for the default intent — s3fifo.rs:447; the finalelseofevict_to_boundsreturnsStep::Emptyfromevict_bulk_topon an empty region — s3fifo.rs:832, 888-891. - Bookkeeping is charged inside
max_bytesand held to 1%: both admission paths test againstbookkeeping_budget = max_bytes / 100— s3fifo.rs:316, 525-534, 630-633. - A bulk admission never evicts main: the room loop calls only
evict_bulkandevict_from_small— s3fifo.rs:486-501. - Per-intent hits sum to the totals on a RAM tier:
record_hit_with_intentis the only caller ofrecord_hitoutside the disk tier — metrics.rs:52, disk.rs:811. - The max-age sweep keeps region accounting consistent:
drain_wheregoes throughremove, which clearsby_rankandregion.bytesfor a bulk entry — s3fifo.rs:775-782, 799-812. - The guide's "Probation is a tenth of the cap" matches
small_quota_bytes = (max_bytes / 10).max(1)— s3fifo.rs:280, docs/guides/caching.md:164-172. Cache::get_uncountedandCache::admitchanged signature but arepub(crate), so the "callers compile unchanged" claim does not rest on them — cache.rs:224, 275.
Checks run:
cargo test -p ravel-cache→ ok, 104 passed, 0 failed (includes all nine new s3fifo tests and the tiered disk-hit intent test).
Smaller things:
Inner::lookupnow callsbookkeeping_bytes()(twolen()reads) and an atomic store on every RAM lookup while holding the fifo lock — cache.rs:500-501. On the interactive path it stores 0 forever. A cheap guard (store only when the region is non-empty, or when the value changed) keeps the hot path as it was.- In
track_tenant,evict_from_smallis called without the!self.small.is_empty()guard the other call sites use — s3fifo.rs:642. It is safe today because an empty small queue returnsStep::Empty, which the match turns intoreturn false, but the asymmetry withevict_to_bounds's guarded calls invites a future edit that loops. - A bulk admission can promote a small-queue entry with
freq > 0into main viaevict_from_small'sStep::Skippedbranch — s3fifo.rs:495, 855-860. Not a defect (the entry earned the promotion), but "a bulk admission never evicts a main-queue interactive entry" in the description is about eviction only; main can still grow because of bulk traffic. limits.rs:71-78says values outside 0.0..=1.0 are clamped.f64::clampreturns NaN for a NaN input, andNaN as u64saturates to 0, so a NaNbulk_sharesilently disables the region rather than clamping. No caller can produce it today.
Not checked:
- That ravel-query, ravel-catalog and ravel-server still compile against the changed crate (only
cargo test -p ravel-cachewas run; the pub(crate) signature changes make a break unlikely, and CI covers it). - The sql / flight-sql / esql feature lanes and the workspace gate.
- ADR-3072 decision 3b and ADR-2677's decision 3 deferral themselves; the PR's claims about them were taken as written.
- The executor's mutant results and the quoted hit-rate figures beyond re-running the committed tests.
Findings outside the diff:
- crates/ravel-cache/src/cache.rs:514 (should_fix):
Inner::insertrunsowned_exactbefore the eviction structure decides, so a bulk admission the region declines still pays the copy and records it as an admission copy.
Failure: aBulkScanread of a 25 MB object whose rank is abovebulk_rank_bound, with fetchedBytesthat are shared or over-capacity (the ordinary object-store case), copies all 25 MB incopy_countedand theninsert_with_intentreturns(false, _)at s3fifo.rs:484 without storing anything. Under the workload this feature exists for -- a loop larger than the region, where the rank bound declines most keys by design -- that is a full memcpy per declined read, andadmission_copies/admission_copied_bytes(cache.rs:61-64) count copies for entries that were never admitted, so the metric no longer means what its name says.
Evidence: cache.rs:509-519 copies first and calls the fifo second; s3fifo.rs:483-485 and 500 are the two decline returns; the over-size decline at cache.rs:511-512 already skips the copy for exactly this reason, and the new decline path was not given the same exemption.
Fix: ask the fifo whether it would admit before copying (awould_admit(key, size, intent)that reproduces the rank-bound andbulk_fitschecks), or move the copy insideinsert_with_intentpast the decline points.
Task 1559b0f9-eb7e-4866-88bc-06f740675b1a on ravel-clickbench-ubuntu-2, model claude-opus-5, effort medium.
| } | ||
|
|
||
| let promote_to_main = self.ghost_set.remove(&key); | ||
| let promote_to_main = intent == AccessIntent::Interactive && self.ghost_set.remove(&key); |
There was a problem hiding this comment.
nit: A Maintenance admission short-circuits before ghost_set.remove(&key), so a key can be resident and still in the ghost queue at the same time; BulkScan returns at line 443-445 and skips the removal too.
Failure: a maintenance read admits key K to small (K stays in ghost_set and in the ghost deque), K is evicted from small again, and lines 863-864 push a second copy of K into ghost. The trim at 865-869 then pops the older copy and calls ghost_set.remove(&K) while the newer copy is still in the deque, so ghost and ghost_set disagree and a later interactive re-admission of K fails to promote even though the deque still remembers it. Before this change a re-admission always cleared the ghost entry, so a duplicate could not be created. No production impact, since nothing sets Maintenance or BulkScan yet.
Evidence: s3fifo.rs:443-450 (both early exits), 863-869 (push and trim).
Fix: remove the key from ghost_set/ghost for every intent and use the removal's result only to decide promote_to_main.
|
|
||
| impl AccessIntent { | ||
| /// The intent's metric label value. | ||
| pub fn as_str(self) -> &'static str { |
There was a problem hiding this comment.
nit: AccessIntent::as_str has no caller anywhere in the workspace.
Failure: none at runtime; it is a public method documented as "the intent's metric label value" while no metric emits a label, so the first reader to wire up labels has no way to tell whether these strings are already a published contract.
Evidence: grep -rn "as_str()" crates/ravel-cache/src returns nothing, and a workspace grep for IntentMetricsSnapshot/intent_snapshot outside crates/ravel-cache/src returns nothing. Either land it with the exporter in #3084 or say in the doc comment that it is unused until then.
A bulk read the region declined was still copied into an owned allocation and counted as an admission copy before the eviction structure decided. The S3-FIFO now takes the value as a closure, insert_with_intent_lazy, and builds it only at the point it stores an entry, so a declined bulk admission copies nothing and counts nothing. Interactive and maintenance values, which are declined only when over size or already resident, are still copied before the lock and counted only once stored. A closure was chosen over a would_admit predicate because insert_bulk evicts and rotates its pass window before it decides, so a separate predicate could disagree with the insert. Every admission that stores a key now drops it from the ghost queue, whatever its intent; only an interactive one promotes to main. The bookkeeping gauge is stored only when its value changed. track_tenant no longer evicts from an empty small queue. A NaN bulk_share reads as the default share instead of a zero region. The AccessIntent::as_str doc no longer names a caller that does not exist. The caching guide now says a bulk read never moves bytes into main, that an interactive read of a region entry can, and that a region entry lives one to two pass windows, so a loop larger than about 32 times the cache is served nothing. Refs: #3083 Refs: #3098 Refs: #3072 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ravel Fleet Executor <fleet-executor@nofire.ai>
A bulk read the region declined was still copied into an owned allocation and counted as an admission copy before the eviction structure decided. The S3-FIFO now takes the value as a closure, insert_with_intent_lazy, and builds it only at the point it stores an entry, so a declined bulk admission copies nothing and counts nothing. Interactive and maintenance values, which are declined only when over size or already resident, are still copied before the lock and counted only once stored. A closure was chosen over a would_admit predicate because insert_bulk evicts and rotates its pass window before it decides, so a separate predicate could disagree with the insert. Every admission that stores a key now drops it from the ghost queue, whatever its intent; only an interactive one promotes to main. The bookkeeping gauge is stored only when its value changed. track_tenant no longer evicts from an empty small queue. A NaN bulk_share reads as the default share instead of a zero region. The AccessIntent::as_str doc no longer names a caller that does not exist. The caching guide now says a bulk read never moves bytes into main, that an interactive read of a region entry can, and that a region entry lives one to two pass windows, so a loop larger than about 32 times the cache is served nothing. Refs: #3083 Refs: #3098 Refs: #3072 Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
Round-1 findings fixed in 6d6a2a6 (fleet fix round 7a538f81, re-authored here): the cache decides bulk admission before copying ( |
|
@claude-fleet review |
|
Review task queued. Task: 4b6b1fef-a631-40a9-a074-515a3b748f82 |
There was a problem hiding this comment.
Previous round at 06f8068: 3 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
crates/ravel-cache/src/cache.rs:514 (should_fix): Inner::insert runs owned_exact before the eviction structure decides, so a bulk admission the region declines still pays the copy and records it as an admission copy. Failure: a BulkScan read of a 25 MB object whose rank is above bulk_rank_bound, with fetched Bytes that are shared or over-capacity (the ordinary object-store case), copies all 25 MB in copy_counted and then insert_with_intent returns (false, _) at s3fifo.rs:484 without storing anything. Under the workload this feature exists for -- a loop larger than the region, where the rank bound declines most keys by design -- that is a full memcpy per declined read, and admission_copies/admission_copied_bytes (cache.rs:61-64) count copies for entries that were never admitted, so the metric no longer means what its name says. Evidence: cache.rs:509-519 copies first and calls the fifo second; s3fifo.rs:483-485 and 500 are the two decline returns; the over-size decline at cache.rs:511-512 already skips the copy for exactly this reason, and the new decline path was not given the same exemption. Fix: ask the fifo whether it would admit before copying (a would_admit(key, size, intent) that reproduces the rank-bound and bulk_fits checks), or move the copy inside insert_with_intent past the decline points. |
fixed | crates/ravel-cache/src/cache.rs:518 |
crates/ravel-cache/src/s3fifo.rs:447 (nit): A Maintenance admission short-circuits before ghost_set.remove(&key), so a key can be resident and still in the ghost queue at the same time; BulkScan returns at line 443-445 and skips the removal too. Failure: a maintenance read admits key K to small (K stays in ghost_set and in the ghost deque), K is evicted from small again, and lines 863-864 push a second copy of K into ghost. The trim at 865-869 then pops the older copy and calls ghost_set.remove(&K) while the newer copy is still in the deque, so ghost and ghost_set disagree and a later interactive re-admission of K fails to promote even though the deque still remembers it. Before this change a re-admission always cleared the ghost entry, so a duplicate could not be created. No production impact, since nothing sets Maintenance or BulkScan yet. Evidence: s3fifo.rs:443-450 (both early exits), 863-869 (push and trim). Fix: remove the key from ghost_set/ghost for every intent and use the removal's result only to decide promote_to_main. |
fixed | crates/ravel-cache/src/s3fifo.rs:463 |
crates/ravel-cache/src/lib.rs:149 (nit): AccessIntent::as_str has no caller anywhere in the workspace. Failure: none at runtime; it is a public method documented as "the intent's metric label value" while no metric emits a label, so the first reader to wire up labels has no way to tell whether these strings are already a published contract. Evidence: grep -rn "as_str()" crates/ravel-cache/src returns nothing, and a workspace grep for IntentMetricsSnapshot/intent_snapshot outside crates/ravel-cache/src returns nothing. Either land it with the exporter in #3084 or say in the doc comment that it is unused until then. |
fixed | crates/ravel-cache/src/lib.rs:148 |
Re-reviewed at 6d6a2a6. All three findings from the previous round are addressed; the new commit introduces one issue of its own.
Request changes: one should_fix finding.
Findings:
- should_fix, crates/ravel-cache/src/cache.rs:530 — the bulk admission's memcpy now runs while the fifo mutex is held
- nit, crates/ravel-cache/src/cache.rs:532 — a copy made for an already-resident interactive key is no longer counted
Previous round:
- 1 (bulk copy before the decision) — fixed.
Inner::insertno longer pre-copies aBulkScanvalue (cache.rs:517-523) and builds it in themakeclosure the fifo runs only when it stores (cache.rs:527-540, s3fifo.rs:442-455). The decline paths at s3fifo.rs:505-508 and 521-526 return beforemake(). - 2 (ghost entry left behind) — fixed.
forget_ghostruns on every storing admission: s3fifo.rs:463 for interactive/maintenance, s3fifo.rs:533 for bulk, with the promotion decision taken from its result rather than gating it. - 3 (
as_strhas no caller) — fixed as "say so in the doc comment", lib.rs:148-149.
Verified:
- "A bulk read the region declines is not copied into the cache" — cache.rs:518-523 skips the pre-copy for
BulkScan, and the onlyowned_exactfor that intent is inside the closure at cache.rs:530. - "an admission that stores a key, whatever its intent, drops it from the ghost queue" (s3fifo.rs:47-50) — s3fifo.rs:463, 482-491, 533.
- NaN
bulk_sharereads as the default rather than collapsing the region to zero — limits.rs:117-128, applied at s3fifo.rs:305. - The guide's new life-of-a-region-entry paragraph (docs/guides/caching.md:210-219) matches the code: window =
clamp(2 * pass_bytes, 2 * max_bytes, 16 * max_bytes)(s3fifo.rs:112-113, 628-629), an entry surviving one to two windows (s3fifo.rs:866-871), so 16-32x for a loop at or past 8x. - The added
!self.small.is_empty()guard intrack_tenant(s3fifo.rs:668-670) changes no outcome:evict_from_smallon an empty queue already returnedStep::Empty, which the match turns into the samereturn false.
Checks run:
cargo test -p ravel-cache --lib→ ok, 109 passed.- Reverted cache.rs:518 to the pre-fix order (copy every in-size value before the lock) →
a_declined_bulk_admission_copies_nothing_and_counts_no_admission_copyfails at cache.rs:1329, "the value itself". Restored. - Reverted s3fifo.rs:463 to
intent == AccessIntent::Interactive && self.ghost_set.remove(&key)→a_maintenance_or_bulk_admission_clears_the_ghost_entryfails at s3fifo.rs:1699, "Maintenance admission left the key in the ghost queue". Restored.
Smaller things:
- The FLIP note on the new cache.rs test (cache.rs:1277-1282) claims the pre-fix order makes "both declined admissions count a copy". It does not: the counting also moved into the closure, so under that one-line revert
admission_copies()stays 0 and only thereturned.as_ptr()assertion fails. That is still a real failure, so the test is non-vacuous, but the note names assertions that do not fire. Cache::admit's doc (cache.rs:273-275) still says it returnsvalueitself only "if it was over the size limit". It now also does so for a bulk admission the region declined, whichinsert_with_intent(cache.rs:267-268) does say.
Not checked: the feature lanes and the rest of the workspace (CI's job), and the parts of the PR that are unchanged since 06f8068 beyond re-reading the current files — the previous round covered them.
Not checked:
- the sql/flight-sql/esql feature lanes and the rest of the workspace build (CI's job)
- parts of the PR unchanged since 06f8068 beyond re-reading the current files
Task 4b6b1fef-a631-40a9-a074-515a3b748f82 on rp1, model claude-opus-5, effort medium.
| fifo.insert_with_intent_lazy(key, size, intent, &self.metrics, || { | ||
| let (bytes, copied) = match copied { | ||
| Some(copied) => (std::mem::take(&mut value), copied), | ||
| None => owned_exact(std::mem::take(&mut value)), |
There was a problem hiding this comment.
should_fix: Moving the bulk copy into the make closure fixes the wasted copy but runs that copy while self.fifo is locked, which is the same mutex every interactive read takes.
Failure: a BulkScan admission of a 25 MB object whose Bytes are shared or over-capacity (the ordinary object-store case) now runs Bytes::copy_from_slice of 25 MB at cache.rs:530 inside the fifo.insert_with_intent_lazy call at cache.rs:527, with the guard taken at cache.rs:526 still held. Inner::lookup takes that same lock (cache.rs:477), so every concurrent interactive get blocks for the duration of the memcpy, a few milliseconds per admitted bulk object. Before this commit the copy happened before the lock was taken, and the interactive path still does (cache.rs:517-523). That is a tail-latency regression on exactly the workload this feature exists to protect: interactive reads served while a bulk scan runs. No production impact until #3084 sets BulkScan.
Evidence: cache.rs:526 takes the lock, 527-540 calls the closure under it, 530 is the copy; cache.rs:477 and 526 are the same self.fifo. Pre-fix order at 06f8068 had the single owned_exact call above the lock() line.
Fix: keep the decision cheap and the copy outside the lock -- take the lock, run the rank-bound and room checks (would_admit(key, size, intent)), drop it, copy, then admit; a key that stops fitting between the two is just an ordinary decline.
| Some(copied) => (std::mem::take(&mut value), copied), | ||
| None => owned_exact(std::mem::take(&mut value)), | ||
| }; | ||
| if copied > 0 { |
There was a problem hiding this comment.
nit: record_admission_copy now runs only inside the closure, so a copy made for an admission the fifo declines as already-resident is performed and never counted.
Failure: an interactive admit of a key already in RAM with shared Bytes copies the whole value at cache.rs:519, then insert_with_intent_lazy returns at s3fifo.rs:457-460 without calling the closure, so admission_copies/admission_copied_bytes stay flat. The memcpy still happened, and Cache::insert's doc (cache.rs:255-259) tells the reader those copies are counted by CacheMetrics::admission_copies. Before this commit owned_exact counted at copy time, so this case was visible. Reachable today without #3084: TieredCache::insert_with_intent and the resolve_peeked_miss recheck race both re-admit resident keys.
Evidence: cache.rs:519 copies, cache.rs:532-534 counts only under the closure, s3fifo.rs:457-460 is the resident early return.
Fix: either count the pre-lock copy where it is made, or say in the insert doc that the counter tracks copies that were stored.
|
Round 2 acknowledged: the copy-under-the-lock finding is real (it was the trade-off the fix round named) and gets the fix you describe, a cheap |
A bulk admission used to copy its value inside the admission closure, with the fifo lock held, so every reader of the cache waited on the copy. The RAM tier now asks S3Fifo::would_admit under the lock, a dry run of insert_bulk that changes nothing, drops the lock to copy, and takes it again to admit. The admission re-runs every check, so a key that stopped fitting in between is declined like any other, and its copy is counted. A bulk read the first check declines is admitted, and so declined, in the same lock hold and is never copied, as before. would_admit runs insert_bulk's steps against a DryRun that records what tenant tracking, retirement and the room loop would free, through the same bound helpers insert_bulk now uses: region_entry_fits, region_capacity_of, tenants_full, rank_bound and TenantPass::observe. Its answer is exact unless the read closes its tenant's pass window, where it answers true and leaves the decision to the admission. admission_copies now counts every copy made, stored or not: an interactive insert of a key already resident copies its value and counts it, and the docs on Cache::insert and CacheMetrics say so. Refs: #3083 Refs: #3098 Refs: #3072 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ravel Fleet Executor <fleet-executor@nofire.ai>
A BulkScan admission made its owned copy inside the insert closure, while holding the same fifo mutex that every interactive lookup takes, so a large bulk copy stalled interactive reads behind it. The RAM tier now asks S3Fifo::would_admit under the lock, a check that mutates nothing and runs insert_bulk's own steps against the tenant's pass as the read would leave it: the rank collision, the rank bound, and the room the eviction loop would make. A read it declines is declined there and copies nothing. A read it accepts is copied with the lock released, then admitted under a fresh lock, where insert_bulk rechecks and may still decline. would_admit and insert_bulk share TenantPass::observe, window_full, the rank bound and the fit test, and bulk_room walks the eviction loop's steps in the same order on counts. The admission copy counters now count every copy made, where it is made, including a copy whose entry is not stored: an interactive admission of a resident key, or a bulk read declined on the recheck. The docs said only stored copies counted, which the interactive path already contradicted. Refs: #3083 Refs: #3098 Refs: #3072 Signed-off-by: Ravel Fleet Executor <fleet-executor@nofire.ai>
A BulkScan admission made its owned copy inside the insert closure, while holding the same fifo mutex that every interactive lookup takes, so a large bulk copy stalled interactive reads behind it. The RAM tier now asks S3Fifo::would_admit under the lock, a check that mutates nothing and runs insert_bulk's own steps against the tenant's pass as the read would leave it: the rank collision, the rank bound, and the room the eviction loop would make. A read it declines is declined there and copies nothing. A read it accepts is copied with the lock released, then admitted under a fresh lock, where insert_bulk rechecks and may still decline. would_admit and insert_bulk share TenantPass::observe, window_full, the rank bound and the fit test, and bulk_room walks the eviction loop's steps in the same order on counts. The admission copy counters now count every copy made, where it is made, including a copy whose entry is not stored: an interactive admission of a resident key, or a bulk read declined on the recheck. The docs said only stored copies counted, which the interactive path already contradicted. Refs: #3083 Refs: #3098 Refs: #3072 Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
Round 2's two findings are fixed in 7638e25 (fleet task 9ad6168b, folded onto the previous head 6d6a2a6 as one re-authored commit, tree identical to the task result).
Tests, each shown failing under a deliberate change and green after: This is review round 3 on this pull request. |
|
@claude-fleet review |
|
Review task queued. Task: a5109399-0dc8-4f06-bbe0-fd3c60d707cf |
There was a problem hiding this comment.
Previous round at 6d6a2a6: 5 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
crates/ravel-cache/src/cache.rs:518 (should_fix): Inner::insert runs owned_exact before the eviction structure decides, so a bulk admission the region declines still pays the copy and records it as an admission copy. Failure: a BulkScan read of a 25 MB object whose rank is above bulk_rank_bound, with fetched Bytes that are shared or over-capacity (the ordinary object-store case), copies all 25 MB in copy_counted and then insert_with_intent returns (false, _) at s3fifo.rs:484 without storing anything. Under the workload this feature exists for -- a loop larger than the region, where the rank bound declines most keys by design -- that is a full memcpy per declined read, and admission_copies/admission_copied_bytes (cache.rs:61-64) count copies for entries that were never admitted, so the metric no longer means what its name says. Evidence: cache.rs:509-519 copies first and calls the fifo second; s3fifo.rs:483-485 and 500 are the two decline returns; the over-size decline at cache.rs:511-512 already skips the copy for exactly this reason, and the new decline path was not given the same exemption. Fix: ask the fifo whether it would admit before copying (a would_admit(key, size, intent) that reproduces the rank-bound and bulk_fits checks), or move the copy inside insert_with_intent past the decline points. |
fixed | crates/ravel-cache/src/cache.rs:545 |
crates/ravel-cache/src/s3fifo.rs:463 (nit): A Maintenance admission short-circuits before ghost_set.remove(&key), so a key can be resident and still in the ghost queue at the same time; BulkScan returns at line 443-445 and skips the removal too. Failure: a maintenance read admits key K to small (K stays in ghost_set and in the ghost deque), K is evicted from small again, and lines 863-864 push a second copy of K into ghost. The trim at 865-869 then pops the older copy and calls ghost_set.remove(&K) while the newer copy is still in the deque, so ghost and ghost_set disagree and a later interactive re-admission of K fails to promote even though the deque still remembers it. Before this change a re-admission always cleared the ghost entry, so a duplicate could not be created. No production impact, since nothing sets Maintenance or BulkScan yet. Evidence: s3fifo.rs:443-450 (both early exits), 863-869 (push and trim). Fix: remove the key from ghost_set/ghost for every intent and use the removal's result only to decide promote_to_main. |
fixed | crates/ravel-cache/src/s3fifo.rs:592 |
crates/ravel-cache/src/lib.rs:148 (nit): AccessIntent::as_str has no caller anywhere in the workspace. Failure: none at runtime; it is a public method documented as "the intent's metric label value" while no metric emits a label, so the first reader to wire up labels has no way to tell whether these strings are already a published contract. Evidence: grep -rn "as_str()" crates/ravel-cache/src returns nothing, and a workspace grep for IntentMetricsSnapshot/intent_snapshot outside crates/ravel-cache/src returns nothing. Either land it with the exporter in #3084 or say in the doc comment that it is unused until then. |
fixed | crates/ravel-cache/src/lib.rs:148 |
crates/ravel-cache/src/cache.rs:530 (should_fix): Moving the bulk copy into the make closure fixes the wasted copy but runs that copy while self.fifo is locked, which is the same mutex every interactive read takes. Failure: a BulkScan admission of a 25 MB object whose Bytes are shared or over-capacity (the ordinary object-store case) now runs Bytes::copy_from_slice of 25 MB at cache.rs:530 inside the fifo.insert_with_intent_lazy call at cache.rs:527, with the guard taken at cache.rs:526 still held. Inner::lookup takes that same lock (cache.rs:477), so every concurrent interactive get blocks for the duration of the memcpy, a few milliseconds per admitted bulk object. Before this commit the copy happened before the lock was taken, and the interactive path still does (cache.rs:517-523). That is a tail-latency regression on exactly the workload this feature exists to protect: interactive reads served while a bulk scan runs. No production impact until #3084 sets BulkScan. Evidence: cache.rs:526 takes the lock, 527-540 calls the closure under it, 530 is the copy; cache.rs:477 and 526 are the same self.fifo. Pre-fix order at 06f8068 had the single owned_exact call above the lock() line. Fix: keep the decision cheap and the copy outside the lock -- take the lock, run the rank-bound and room checks (would_admit(key, size, intent)), drop it, copy, then admit; a key that stops fitting between the two is just an ordinary decline. |
fixed | crates/ravel-cache/src/cache.rs:551 |
crates/ravel-cache/src/cache.rs:532 (nit): record_admission_copy now runs only inside the closure, so a copy made for an admission the fifo declines as already-resident is performed and never counted. Failure: an interactive admit of a key already in RAM with shared Bytes copies the whole value at cache.rs:519, then insert_with_intent_lazy returns at s3fifo.rs:457-460 without calling the closure, so admission_copies/admission_copied_bytes stay flat. The memcpy still happened, and Cache::insert's doc (cache.rs:255-259) tells the reader those copies are counted by CacheMetrics::admission_copies. Before this commit owned_exact counted at copy time, so this case was visible. Reachable today without #3084: TieredCache::insert_with_intent and the resolve_peeked_miss recheck race both re-admit resident keys. Evidence: cache.rs:519 copies, cache.rs:532-534 counts only under the closure, s3fifo.rs:457-460 is the resident early return. Fix: either count the pre-lock copy where it is made, or say in the insert doc that the counter tracks copies that were stored. |
fixed | crates/ravel-cache/src/cache.rs:569 |
No longer carried to the next round, settled in rounds 2 and 3:
- crates/ravel-cache/src/cache.rs:518 (should_fix):
Inner::insertrunsowned_exactbefore the eviction structure decides, so a bulk admission the region declines still pays the copy and records it as an admission copy. Failure: aBulkScanread of a 25 MB object whose rank is abovebulk_rank_bound, with fetchedBytesthat are shared or over-capacity (the ordinary object-store case), copies all 25 MB incopy_countedand theninsert_with_intentreturns(false, _)at s3fifo.rs:484 without storing anything. Under the workload this feature exists for -- a loop larger than the region, where the rank bound declines most keys by design -- that is a full memcpy per declined read, andadmission_copies/admission_copied_bytes(cache.rs:61-64) count copies for entries that were never admitted, so the metric no longer means what its name says. Evidence: cache.rs:509-519 copies first and calls the fifo second; s3fifo.rs:483-485 and 500 are the two decline returns; the over-size decline at cache.rs:511-512 already skips the copy for exactly this reason, and the new decline path was not given the same exemption. Fix: ask the fifo whether it would admit before copying (awould_admit(key, size, intent)that reproduces the rank-bound andbulk_fitschecks), or move the copy insideinsert_with_intentpast the decline points. - crates/ravel-cache/src/s3fifo.rs:463 (nit): A
Maintenanceadmission short-circuits beforeghost_set.remove(&key), so a key can be resident and still in the ghost queue at the same time;BulkScanreturns at line 443-445 and skips the removal too. Failure: a maintenance read admits key K to small (K stays inghost_setand in theghostdeque), K is evicted from small again, and lines 863-864 push a second copy of K intoghost. The trim at 865-869 then pops the older copy and callsghost_set.remove(&K)while the newer copy is still in the deque, soghostandghost_setdisagree and a later interactive re-admission of K fails to promote even though the deque still remembers it. Before this change a re-admission always cleared the ghost entry, so a duplicate could not be created. No production impact, since nothing setsMaintenanceorBulkScanyet. Evidence: s3fifo.rs:443-450 (both early exits), 863-869 (push and trim). Fix: remove the key fromghost_set/ghostfor every intent and use the removal's result only to decidepromote_to_main. - crates/ravel-cache/src/lib.rs:148 (nit):
AccessIntent::as_strhas no caller anywhere in the workspace. Failure: none at runtime; it is a public method documented as "the intent's metric label value" while no metric emits a label, so the first reader to wire up labels has no way to tell whether these strings are already a published contract. Evidence:grep -rn "as_str()" crates/ravel-cache/srcreturns nothing, and a workspace grep forIntentMetricsSnapshot/intent_snapshotoutsidecrates/ravel-cache/srcreturns nothing. Either land it with the exporter in #3084 or say in the doc comment that it is unused until then.
Request changes: one should_fix finding on the bulk eviction loop, plus a stale line in the guide.
The previous round's five findings are all addressed: the bulk decision now runs before the copy, the copy runs with the fifo lock dropped, every copy is counted where it is made, forget_ghost runs for all three intents, and as_str says it is unused until #3084. would_admit is a faithful model of insert_bulk's checks as far as I could follow them, and would_admit_predicts_the_bulk_admission pins the direction that matters (never false where the admission stores).
Findings:
- should_fix, crates/ravel-cache/src/s3fifo.rs:576 — a bulk admission blocked by the bookkeeping budget drains the whole small queue and admits nothing
- nit, docs/guides/caching.md:220 — "a bulk read the region declines is not copied" is no longer true for a recheck decline
Verified:
- Previous finding 1 and 4: the bulk decision is made under the lock at cache.rs:544-550, the lock is dropped at 551 and the copy made at 553, so no admission copy runs under
self.fifo— the same mutexInner::lookuptakes at cache.rs:484. - Previous finding 5:
owned_copy(cache.rs:569-576) counts at the copy site, so a copy for a key the fifo finds resident is counted;Cache::insert's doc (cache.rs:254-259) now states exactly that. - Previous finding 2:
forget_ghostruns on the interactive/maintenance path (s3fifo.rs:484) and on the bulk path (s3fifo.rs:592), with the result used only to decide promotion. - "
Interactiveis exactly today's policy": every pre-existing entry point delegates withAccessIntent::Interactive(cache.rs:205-208, 231-234, 255-262; tiered.rs:287-288, 326-333, 475-482, 610-613, 652-656, 681-686, 722-728). - Per-intent hits sum to the totals:
record_hit_with_intentcallsrecord_hitfirst (metrics.rs:51-56), and the only RAM hit site isInner::get(cache.rs:468-480).
Smaller things:
insert_checked_then'sbetweenparameter exists only so one test can mutate the region between the check and the copy. It is cheap and documented, but it is a test seam in production code; a#[cfg(test)]variant would keepInner::insertthe single shape.bulk_roomuses non-saturating-=ontotal,regionandbookkeeping. The subtractions are bounded by the structures they iterate, so they cannot underflow as long as a key is never insmallandby_ranktwice; worth a saturating form anyway, since a debug build turns any future accounting drift into a panic on a read path.
Not checked:
- I ran no cargo command: no build, no tests, no revert check. The scenario in the finding is from reading
insert_bulkandbulk_fits_with, not from an executed test. - The executor's claimed mutant runs, the 11.5 GB geometry test figures, and the nextest/jemalloc result in the PR description.
crates/ravel-cache/src/disk.rsand the single-flight path beyond the intent plumbing shown in the diff.
Not checked:
- No cargo command run: no build, no test, no revert check; the should_fix finding is from reading insert_bulk and bulk_fits_with, not from an executed test
- The executor's mutant runs, the 11.5 GB geometry hot-set figures, and the nextest/jemalloc results claimed in the PR description
- crates/ravel-cache/src/disk.rs and the single-flight internals beyond the intent plumbing visible in the diff
Task a5109399-0dc8-4f06-bbe0-fd3c60d707cf on ci-16gb-fsn1-1, model claude-opus-5, effort medium.
| evicted.push(top_key); | ||
| continue; | ||
| } | ||
| if self.region.bytes + size <= self.region_share_bytes && !self.small.is_empty() { |
There was a problem hiding this comment.
should_fix: insert_bulk's eviction loop evicts small-queue entries to satisfy a bookkeeping-budget shortfall that those evictions cannot change, so it drains the entire probation queue and then declines anyway.
Failure: the region is at its bookkeeping cap (reached first whenever region entries are smaller than about 9.6 KB: the cap allows max_bytes/19200 entries while the byte share allows 0.5*max_bytes/size), the tenant's pass fits the region capacity so bulk_rank_bound is full and the rank check passes, and the arriving key's rank is above every resident region rank. Line 569-571 finds no higher-ranked victim, so each iteration falls to line 576: region.bytes + size <= region_share_bytes holds (the region is bookkeeping-bound, not byte-bound) and evict_from_small runs. But bulk_fits_with fails on its first term, bookkeeping + REGION_ENTRY_BOOKKEEPING_BYTES <= self.bookkeeping_budget (s3fifo.rs:623-624), which no small-queue eviction moves. The loop therefore evicts or promotes every entry in the small queue, returns (false, evicted) at line 582, and stores nothing. Concretely at max_bytes = 1 GB with 4 KB bulk objects: the region caps at 54,613 entries, and a bulk read whose rank happens to top the resident set evicts every once-read interactive entry in probation (up to the 10% small quota) for an admission that never happens. would_admit models the same loop and returns false (s3fifo.rs:643-666), but the decline path still calls insert_with_intent_lazy (cache.rs:546), which runs the real loop, so the new check avoids the copy and not the drain. No production impact until #3084 sets BulkScan.
Evidence: s3fifo.rs:568-583 (the loop), 611-627 (bulk_fits_with, the bookkeeping term), 996-1021 (evict_from_small), 357-360 (bookkeeping is tenants and region entries only), cache.rs:544-550 (the decline path still runs the loop).
Fix: before the small-queue step, decline when bookkeeping_bytes() + REGION_ENTRY_BOOKKEEPING_BYTES > bookkeeping_budget and no higher-ranked region entry is available, since only a region eviction can relieve that term.
| reads, a window being twice the tenant's pass bytes, held between 2 and | ||
| 16 times the cap. For a loop over 8 times the cap an entry therefore | ||
| lives for 16 to 32 times the cap of reads, and a loop larger than about | ||
| 32 times the cap is served nothing. A bulk read the region declines is |
There was a problem hiding this comment.
nit: The guide still says a declined bulk read is never copied, which this commit made untrue for the recheck decline.
Failure: an operator reading "A bulk read the region declines is not copied" concludes ravel_cache_admission_copies_total counts only stored entries, while a read would_admit accepts and the admission's own recheck then declines is copied and counted (cache.rs:551-562, 569-576).
Evidence: docs/guides/caching.md:220-221 against cache.rs:254-259, which the same commit updated to the new wording, and changelog.d/3083.added.md:7-9, which says "declines when it arrives". Mirror that qualifier in the guide.
|
Owner decision: a fourth round rather than merging with the finding. Fix round dispatched as fleet task c04ab7aa (opus) from this head 7638e25: decline before the small-queue step when the bookkeeping term cannot be relieved by a small-queue eviction (only a region eviction can), the |
insert_bulk's eviction loop fell through to the small-queue step whenever no region entry ranked above the arriving read, even when the term of bulk_fits that failed was the bookkeeping budget. Small-queue evictions do not change the bookkeeping, so on a region at its bookkeeping cap the loop evicted or promoted every once-read interactive entry in probation and then declined anyway. The loop now declines before the small-queue step when one more region entry's bookkeeping does not fit, and bulk_room, which would_admit uses to model the loop, makes the same check at the same point. Cache::insert_checked_then also stops running the admission for a bulk read would_admit declines. It used to call insert_with_intent_lazy with a value-less closure, which ran the real eviction loop, so the check avoided the copy but not the evictions. The new S3Fifo::decline_bulk counts the read against its tracked tenant's pass, as the admission did before declining, and evicts nothing. The accept path and its recheck are unchanged. AccessIntent::as_str is removed: nothing in the workspace called it and no metric is labelled by intent yet, so the exporter in #3084 can add the label values it publishes. The caching guide and the changelog fragment now say a declined bulk read evicts nothing, and the guide says a read accepted by the pre-copy check and declined by the admission's recheck is copied and counted. No production caller sets BulkScan until #3084; the crate's tests cover the behaviour. Refs: #3083 Refs: #3098 Refs: #3072 Signed-off-by: Ravel Fleet Executor <fleet-executor@nofire.ai>
insert_bulk's eviction loop fell through to the small-queue step whenever no region entry ranked above the arriving read, even when the term of bulk_fits that failed was the bookkeeping budget. Small-queue evictions do not change the bookkeeping, so on a region at its bookkeeping cap the loop evicted or promoted every once-read interactive entry in probation and then declined anyway. The loop now declines before the small-queue step when one more region entry's bookkeeping does not fit, and bulk_room, which would_admit uses to model the loop, makes the same check at the same point. Cache::insert_checked_then also stops running the admission for a bulk read would_admit declines. It used to call insert_with_intent_lazy with a value-less closure, which ran the real eviction loop, so the check avoided the copy but not the evictions. The new S3Fifo::decline_bulk counts the read against its tracked tenant's pass, as the admission did before declining, and evicts nothing. The accept path and its recheck are unchanged. AccessIntent::as_str is removed: nothing in the workspace called it and no metric is labelled by intent yet, so the exporter in #3084 can add the label values it publishes. The caching guide and the changelog fragment now say a declined bulk read evicts nothing, and the guide says a read accepted by the pre-copy check and declined by the admission's recheck is copied and counted. No production caller sets BulkScan until #3084; the crate's tests cover the behaviour. Refs: #3083 Refs: #3098 Refs: #3072 Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
Round 3's should_fix and both nits are fixed in the commit on top of 7638e25 (fleet task c04ab7aa, one re-authored commit, tree identical to the task result).
Mutants, each shown failing then reverted: the PR head logic (small queue 0 vs 20; resident 207 vs 227; region 30 vs 31), the check placed after one small-queue eviction (19 vs 20), the check placed before the region step (the higher-ranked region entry must still be evicted and the read admitted), the decline path still calling This is review round 4, requested on the owner's decision. |
|
@claude-fleet review |
|
Review task queued. Task: 5bf7685c-5f71-43ab-9384-920450ff54d8 |
There was a problem hiding this comment.
Previous round at 7638e25: 4 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
crates/ravel-cache/src/cache.rs:551 (should_fix): Moving the bulk copy into the make closure fixes the wasted copy but runs that copy while self.fifo is locked, which is the same mutex every interactive read takes. Failure: a BulkScan admission of a 25 MB object whose Bytes are shared or over-capacity (the ordinary object-store case) now runs Bytes::copy_from_slice of 25 MB at cache.rs:530 inside the fifo.insert_with_intent_lazy call at cache.rs:527, with the guard taken at cache.rs:526 still held. Inner::lookup takes that same lock (cache.rs:477), so every concurrent interactive get blocks for the duration of the memcpy, a few milliseconds per admitted bulk object. Before this commit the copy happened before the lock was taken, and the interactive path still does (cache.rs:517-523). That is a tail-latency regression on exactly the workload this feature exists to protect: interactive reads served while a bulk scan runs. No production impact until #3084 sets BulkScan. Evidence: cache.rs:526 takes the lock, 527-540 calls the closure under it, 530 is the copy; cache.rs:477 and 526 are the same self.fifo. Pre-fix order at 06f8068 had the single owned_exact call above the lock() line. Fix: keep the decision cheap and the copy outside the lock -- take the lock, run the rank-bound and room checks (would_admit(key, size, intent)), drop it, copy, then admit; a key that stops fitting between the two is just an ordinary decline. |
fixed | crates/ravel-cache/src/cache.rs:555 |
crates/ravel-cache/src/cache.rs:569 (nit): record_admission_copy now runs only inside the closure, so a copy made for an admission the fifo declines as already-resident is performed and never counted. Failure: an interactive admit of a key already in RAM with shared Bytes copies the whole value at cache.rs:519, then insert_with_intent_lazy returns at s3fifo.rs:457-460 without calling the closure, so admission_copies/admission_copied_bytes stay flat. The memcpy still happened, and Cache::insert's doc (cache.rs:255-259) tells the reader those copies are counted by CacheMetrics::admission_copies. Before this commit owned_exact counted at copy time, so this case was visible. Reachable today without #3084: TieredCache::insert_with_intent and the resolve_peeked_miss recheck race both re-admit resident keys. Evidence: cache.rs:519 copies, cache.rs:532-534 counts only under the closure, s3fifo.rs:457-460 is the resident early return. Fix: either count the pre-lock copy where it is made, or say in the insert doc that the counter tracks copies that were stored. |
fixed | crates/ravel-cache/src/cache.rs:576 |
crates/ravel-cache/src/s3fifo.rs:576 (should_fix): insert_bulk's eviction loop evicts small-queue entries to satisfy a bookkeeping-budget shortfall that those evictions cannot change, so it drains the entire probation queue and then declines anyway. Failure: the region is at its bookkeeping cap (reached first whenever region entries are smaller than about 9.6 KB: the cap allows max_bytes/19200 entries while the byte share allows 0.5*max_bytes/size), the tenant's pass fits the region capacity so bulk_rank_bound is full and the rank check passes, and the arriving key's rank is above every resident region rank. Line 569-571 finds no higher-ranked victim, so each iteration falls to line 576: region.bytes + size <= region_share_bytes holds (the region is bookkeeping-bound, not byte-bound) and evict_from_small runs. But bulk_fits_with fails on its first term, bookkeeping + REGION_ENTRY_BOOKKEEPING_BYTES <= self.bookkeeping_budget (s3fifo.rs:623-624), which no small-queue eviction moves. The loop therefore evicts or promotes every entry in the small queue, returns (false, evicted) at line 582, and stores nothing. Concretely at max_bytes = 1 GB with 4 KB bulk objects: the region caps at 54,613 entries, and a bulk read whose rank happens to top the resident set evicts every once-read interactive entry in probation (up to the 10% small quota) for an admission that never happens. would_admit models the same loop and returns false (s3fifo.rs:643-666), but the decline path still calls insert_with_intent_lazy (cache.rs:546), which runs the real loop, so the new check avoids the copy and not the drain. No production impact until #3084 sets BulkScan. Evidence: s3fifo.rs:568-583 (the loop), 611-627 (bulk_fits_with, the bookkeeping term), 996-1021 (evict_from_small), 357-360 (bookkeeping is tenants and region entries only), cache.rs:544-550 (the decline path still runs the loop). Fix: before the small-queue step, decline when bookkeeping_bytes() + REGION_ENTRY_BOOKKEEPING_BYTES > bookkeeping_budget and no higher-ranked region entry is available, since only a region eviction can relieve that term. |
fixed | crates/ravel-cache/src/s3fifo.rs:601 |
docs/guides/caching.md:220 (nit): The guide still says a declined bulk read is never copied, which this commit made untrue for the recheck decline. Failure: an operator reading "A bulk read the region declines is not copied" concludes ravel_cache_admission_copies_total counts only stored entries, while a read would_admit accepts and the admission's own recheck then declines is copied and counted (cache.rs:551-562, 569-576). Evidence: docs/guides/caching.md:220-221 against cache.rs:254-259, which the same commit updated to the new wording, and changelog.d/3083.added.md:7-9, which says "declines when it arrives". Mirror that qualifier in the guide. |
fixed | docs/guides/caching.md:220 |
No longer carried to the next round, settled in rounds 3 and 4:
- crates/ravel-cache/src/cache.rs:551 (should_fix): Moving the bulk copy into the
makeclosure fixes the wasted copy but runs that copy whileself.fifois locked, which is the same mutex every interactive read takes. Failure: aBulkScanadmission of a 25 MB object whoseBytesare shared or over-capacity (the ordinary object-store case) now runsBytes::copy_from_sliceof 25 MB at cache.rs:530 inside thefifo.insert_with_intent_lazycall at cache.rs:527, with the guard taken at cache.rs:526 still held.Inner::lookuptakes that same lock (cache.rs:477), so every concurrent interactivegetblocks for the duration of the memcpy, a few milliseconds per admitted bulk object. Before this commit the copy happened before the lock was taken, and the interactive path still does (cache.rs:517-523). That is a tail-latency regression on exactly the workload this feature exists to protect: interactive reads served while a bulk scan runs. No production impact until #3084 setsBulkScan. Evidence: cache.rs:526 takes the lock, 527-540 calls the closure under it, 530 is the copy; cache.rs:477 and 526 are the sameself.fifo. Pre-fix order at 06f8068 had the singleowned_exactcall above thelock()line. Fix: keep the decision cheap and the copy outside the lock -- take the lock, run the rank-bound and room checks (would_admit(key, size, intent)), drop it, copy, then admit; a key that stops fitting between the two is just an ordinary decline. - crates/ravel-cache/src/cache.rs:569 (nit):
record_admission_copynow runs only inside the closure, so a copy made for an admission the fifo declines as already-resident is performed and never counted. Failure: an interactiveadmitof a key already in RAM with sharedBytescopies the whole value at cache.rs:519, theninsert_with_intent_lazyreturns at s3fifo.rs:457-460 without calling the closure, soadmission_copies/admission_copied_bytesstay flat. The memcpy still happened, andCache::insert's doc (cache.rs:255-259) tells the reader those copies are counted byCacheMetrics::admission_copies. Before this commitowned_exactcounted at copy time, so this case was visible. Reachable today without #3084:TieredCache::insert_with_intentand theresolve_peeked_missrecheck race both re-admit resident keys. Evidence: cache.rs:519 copies, cache.rs:532-534 counts only under the closure, s3fifo.rs:457-460 is the resident early return. Fix: either count the pre-lock copy where it is made, or say in theinsertdoc that the counter tracks copies that were stored.
Approve: every previous finding is fixed, one new nit.
The two fixes both land where they should: the decline path no longer runs the admission at all (decline_bulk counts the read against the tenant pass and stops), and the admission loop refuses before the small-queue step when only a region eviction could relieve the bookkeeping term. I verified each with its own revert, and the invariant that would_admit is never false where the admission would store survives the change, since the new check sits at the same position in bulk_room as in insert_bulk.
Findings:
- nit, docs/guides/caching.md:208 — "up to half of the cap" is unreachable for bulk objects under ~9.6 KB
Verified:
- The accepted bulk copy now happens with the lock released — cache.rs:553-555 drops the guard, then
owned_copy; the admission retakes it at 563. - Every copy is counted where it is made, stored or not — cache.rs:571-578, with
Cache::insert's doc updated to match at cache.rs:252-262 andrecord_admission_copy's at metrics.rs:129-133. decline_bulkmakes none of the admission's evictions, and its window-rotation return is unreachable from the decline path:would_admitreturns true at s3fifo.rs:527-528 for a read that would close the window, so a read that reachesdecline_bulkcannot rotate — s3fifo.rs:542-556, 764-796.- The decline for an untracked tenant changing nothing is sound:
would_admitis false there only whenTENANT_BOOKKEEPING_BYTES + REGION_ENTRY_BOOKKEEPING_BYTES > bookkeeping_budget, i.e.max_bytesunder ~224 KB — s3fifo.rs:519-522. - The guide and changelog now carry the "when it arrives" qualifier and state that the counters count copies made, not entries stored — docs/guides/caching.md:220-226, changelog.d/3083.added.md:7-10.
AccessIntent::as_stris gone and nothing referenced it:grep -rn "as_str()" crates/ravel-cache/srcreturns nothing.
Checks run:
cargo test -p ravel-cache --lib→ ok, 118 passed.- Removed the bookkeeping guard at s3fifo.rs:599-603 →
a_bulk_read_at_the_bookkeeping_cap_leaves_the_small_queue_alonefails on "small-queue population", left 0 right 20;a_bulk_read_at_the_bookkeeping_cap_evicts_a_higher_ranked_entrystill passes, so the guard is not placed too early. Restored. - Reverted cache.rs:548 to
insert_with_intent_lazy(..., || None)→a_declined_bulk_read_evicts_no_region_entryfails on "no region entry evicted", left 30 right 31. Restored;git status --porcelainclean.
Smaller things:
decline_bulk's return value is documented as the keys a window rotation evicted, and the one caller discards it with a comment about cleanup. Given the point above it is always empty on that path; adebug_assertor a sentence saying so would save the next reader the derivation.copy_probe::record(self.fifo.is_locked())(cache.rs:573) reads whether any thread holds the lock, not this one. Fine for the single-threaded tests that read it, worth a word if it ever gets used from a concurrent test.
Not checked:
- Clippy and the feature lanes (CI's job), and the crate's doctests and integration tests — only
--libran here. - The 11.5 GB server-geometry and 2,000-key loop results quoted in the description; I read the tests that encode them but did not re-run the executor's mutant set.
Not checked:
- clippy and the sql/flight-sql/esql/ravel-bench feature lanes (CI's gate)
- ravel-cache doctests and integration tests; only
cargo test -p ravel-cache --libran - the 11.5 GB server-geometry and 2,000-key loop figures quoted in the description, and the executor's mutant set
Task 5bf7685c-5f71-43ab-9384-920450ff54d8 on rp1, model claude-opus-5, effort medium.
| - `interactive`: the S3-FIFO policy above. | ||
| - `bulk_scan`, for a statement that reads every object of a tenant: a hit | ||
| does not count toward promotion, and a bulk read never moves bytes into | ||
| the main queue. The bytes go to a bulk region, which holds up to half of |
There was a problem hiding this comment.
nit: The guide says the bulk region "holds up to half of the cap", but the 1% bookkeeping budget caps the region by entry count, which binds first for any bulk object under about 9.6 KB.
Failure: an operator sizing --cache-max-bytes for a tenant whose bulk objects are 4 KiB expects half the cap available to the scan and gets about a fifth. The region admits at most (max_bytes/100 - 2048*tenants)/192 entries (s3fifo.rs:95, 108, 652), so at max_bytes = 4 MiB with one tenant and 4 KiB objects it stalls at 207 entries, 848 KB, against a 2 MiB share. The crossover is at an entry size of 9600 bytes; above it the byte share binds and the guide is right. This is now a hard decline rather than a drain, which is the right behaviour, but the cap it declines against is not the one the guide names.
Evidence: docs/guides/caching.md:208-210 and 228-229 against s3fifo.rs:95 (192 bytes per region entry), 108 (budget is max_bytes/100), 650-655 (bulk_fits_with's bookkeeping term), and this PR's own fixture at s3fifo.rs:2126-2135, which asserts the cap is reached while region.bytes + CAP_ENTRY <= region_share_bytes.
Fix: one clause saying the region also holds at most about max_bytes/19200 entries, so the half-the-cap figure is reached only for objects above roughly 10 KB.
The S3-FIFO read cache gains
AccessIntent { Interactive, BulkScan, Maintenance }and intent-aware entry points onS3Fifo,CacheandTieredCache(ADR-3072 decision 3b, the cache side of the shape ADR-2677's deferral of decision 3 named).Interactiveis exactly today's policy and the old method names stay asInteractivewrappers, so every caller in ravel-query and ravel-catalog compiles unchanged and production behaviour is unchanged until #3084 setsBulkScanfrom the SQL executor.BulkScanreads do not raisefreqor promote on a ghost hit and are admitted into a bulk region bounded toCacheLimits.bulk_share(default 0.5) ofmax_bytesas a deterministic stable subset by hash rank, so a loop larger than the region serves the same subset on every later pass; a bulk admission never evicts amain-queue interactive entry.Maintenancereads stay in the small queue and never promote. The region's bookkeeping (a per-tenant bottom-k sketch of distinct bytes over a traffic window standing in for pass boundaries, since the cache cannot see them) is charged insidemax_bytesand capped at 1 %. The disk tier is untouched; a disk hit re-admits to RAM under the request's intent. Per-intent hits and bytes served are on a newIntentMetricsSnapshot.docs/guides/caching.mdnow describes both caps as S3-FIFO (part of #2738; the flag help and reference still say LRU, so #2738 stays open).Fleet task 6503eae7 (epic #3072 track P5), landed by hand after the Stage 4 checkpoint review passed with no blocker. Tests: a 2,000-key loop over 1,000 entries serves 997 touches on passes 2 and 3 with the same set; the 11.5 GB server-geometry hot-set cases (33 MB of 16 KiB interactive ranges against 25 MB bulk objects, and the hot set first touched mid-loop) keep at least 99 % hits; bookkeeping stays inside the cap; the three existing scan-resistance tests are unchanged; every mutant the executor ran (intent ignored, bulk promoted to main, FIFO in the region, bulk evicting main, maintenance promoted, bookkeeping outside the cap) fails its test. Executor gates: clippy over the 15-crate closure, doctests and every feature lane green; closure nextest 8,320 of 8,323 with the 3 failures being ravel-server jemalloc background-thread tests that fail identically on the dispatch ref on that macOS executor. No local gate ran here; CI is the gate.
Three review items are recorded on #3083 as the gate for #3084, not changed here: the region is reclaimed by interactive admissions only when both other queues are empty, so after bulk traffic stops the share stays held until the age sweep; an entry's life in the region is bounded at 16 to 32 x
max_bytesof traffic, so a loop beyond that serves nothing, which the guide does not yet say; the small-queue eviction added for a cache full of interactive entries has no test for a hot set first read while the region fills.Refs: #3083, #3072, #2738