Spool disk usage is unbounded: no eviction for accepted body blobs; in-flight multipart retains whole objects

[frrist]

Current behavior

Ingot's local disk story is split across two very different pipelines:

Catalog blocks (manifests, MST nodes) — bounded. They flow through the logstore catalog plane: segments seal on size/age (SealBytes/SealAge), ship to the network as CAR shards, and the retention sweep retires local segments beyond Retain (logstore/planelog.go: flushOne → runRetention). Reads of retired blocks resolve through the recorded shard inclusions (#44). Local footprint is bounded and self-maintaining.

Object-body blobs (nearly all the volume) — unbounded. Bodies are not journaled: logstore/store.go — "Object-body blobs are not journaled — they are spooled and uploaded per-blob (see blockstore.Spool), so the data plane is gone." The write path spools every body blob to <data_dir>/spool (content-addressed, streamed) and uploads it to its provider (single-shot PUT: add + accept; multipart: parked at UploadPart, accepted at Complete). The spool then doubles as the read-after-write floor and read cache — the layered read path is spool → log → network (blockstore/layered.go), and the network fallback for a missing spool blob is proven by TestForgeReadAfterEviction (wipe /data/spool, GETs re-fetch from piri via blob_locations + /content/retrieve).

But nothing ever evicts. blockstore.Spool explicitly leaves the lifecycle to callers ("the upload_intents state machine, eviction policy is owned by the caller"), and no caller implements a policy. The only Spool.Remove call sites are multipart abort / part-supersede / stale-session sweeping (s3frontend/multipart.go: cleanupPartBlobs). Accepted blobs stay on disk forever.

Drawbacks

1. Spool disk grows with every body byte ever written — not just live data. Every accepted body blob is retained locally in addition to its durable, on-chain-proven copy on piri, and DeleteObject/overwrite never reclaim local disk: releaseBlobs (s3frontend/object.go) sends the network-side /blob/remove only — it does not remove the spool file or the intent row. A bucket that was filled and emptied still occupies its full size in /data/spool. On an appliance running ingot and piri side by side, live data costs ~2× its size and deleted data keeps costing 1×, forever, with no knob. (This looks deferred rather than forgotten: releaseBlobs' own comment ends “crash recovery reconciles upload_intents × blob_refs (a later phase)”.)
2. In-flight multipart uploads need whole-object headroom. Part blobs are parked (durable on the provider) during UploadPart, but their spool files are retained until Complete/Abort — even though concluding a parked blob needs no local bytes (the blob_parks row carries the task CIDs and the sealed put invocation) and parts of an in-flight upload aren't readable via S3 anyway. A large multipart upload therefore requires spool space ≈ the full object size, when it could require ≈ one part.
3. The config misleads. SealBytes/SealAge/Retain read like "local storage is bounded", but they only bound the (comparatively tiny) catalog plane; logstore/config.go still describes planes as "data or catalog" though the data plane is gone.
4. The only mitigation is undocumented. Manually wiping /data/spool is safe for accepted blobs (the eviction itest proves the read path), but that's tribal knowledge, not an operator procedure — and it's unsafe for spooled-but-not-yet-parked bytes, which an operator can't distinguish.

Proposed changes (for discussion)

1. Park-time spool drop (multipart). Delete the spool file once a part blob's park succeeds. Conclude doesn't need local bytes; the crash fallback (spooled-but-never-parked) keeps its file — which is exactly the state where the blob isn't durable yet. In-flight footprint drops from whole-object to ~one part. Small, self-contained change in parkBlobs.
2. Delete-driven spool cleanup. releaseBlobs already computes exactly the digests whose last claim dropped; extend it to also remove the spool file and intent row, with the same shared-content guards cleanupPartBlobs uses (another in-flight session's part may reference the same content-addressed blob). Probably the cheapest win: it turns DeleteObject into an operation that actually frees local disk.
3. Spool budget + eviction sweeper (accepted blobs). A spool_bytes (and/or spool_max_age) knob with a periodic sweeper over upload_intents rows in state accepted that have a recorded blob_locations row: evict oldest/least-recently-read first until under budget. Reads fall back to the network tier — the mechanism is already proven; only the policy is missing. This is the body-blob analogue of the catalog plane's Retain.
   • Safety invariant: never evict a blob that is not (accepted + located) or (parked with a live park row — see 1).
   • Policy questions: LRU vs pure age; a minimum residency window to preserve the read-after-write floor; global vs per-bucket budgets; interaction with ReadCacheBytes (the in-memory block cache fronting the network tier).
4. Optional write-through mode. For thin appliances: don't retain accepted bytes at all (spool as pure staging, evict at accept). The default stays keep-hot-until-budget.
5. Operator visibility. Metrics for spool size / eviction activity, and documentation that describes the disk model (spool = bounded cache, piri = durable store) and the safe manual-wipe procedure until the sweeper lands.
6. Housekeeping. Fix the stale "data or catalog" wording in logstore/config.go; consider renaming or scoping the top-level seal/retain knobs to make their catalog-only reach obvious.

The architecture doc already specifies this

None of the proposals above are new design — docs/architecture.md specifies them and records the gap:

• §5 (the local store): read-after-write and read-cache roles "may use distinct eviction policies over a shared, bounded, size-configurable store"; the "near-stateless" mode (every read via the network tier) is named as supported — i.e. the write-through option above.
• Appendix C (upload_intents): "cache eviction deletes the row and the file" — the delete mechanics and safety pairing proposed here.
• §11 open questions: "Local cache vs near-stateless — cache sizing/eviction, or commit to near-stateless."
• §12 status: the local-table read tier "is only exercised after spool eviction (also not built)". That bullet is now stale in the good direction — the read tier is wired and proven (TestForgeReadAfterEviction, fix(read): resolve retention-retired catalog blocks via local shard inclusions #44) — so the prerequisite eviction was waiting on has landed.
• §9 contract table: the piri-side twin, "Parked-allocation GC + honor Expires (to-build)" — filed as No automatic cleanup for allocated-but-never-accepted blobs; expired allocations block removals piri#44.
• §10: the far end of the design space — allocate-by-size + bind-digest-after is captioned "remove the spool" (same-rack), i.e. even the staging role is contemplated as removable.

So this issue is about scheduling and ratifying a specified phase (and picking the policy knobs), not introducing a new mechanism.

Prior art / references

• Catalog plane seal → ship → retire: logstore/planelog.go, logstore/store.go.
• Retired-block read fallback: fix(read): resolve retention-retired catalog blocks via local shard inclusions #44.
• Body-blob network fallback: TestForgeReadAfterEviction (itest/forge_eviction_test.go).
• Deferred-park multipart mechanics (park rows, conclude-without-local-bytes): feat: deferred-accept multipart (FIL-520) + network blob removal on DeleteObject (FIL-588) #40 (s3frontend/multipart.go, uploader/blob.go).

🤖 Generated with Claude Code