TreeDB alloc loop: reduce Ironbird getEntrySlice flush run allocations

[snissn]

Parent tracker: #3515
Previous loop: #3529 / PR #3531
Related older generic issue: #536

Goal

Reduce the Ironbird-visible TreeDB allocation attributed to TreeDB/caching.getEntrySlice in the accepted plain-send and small-multisend TreeDB rows, without weakening flush/run ownership semantics or retaining unbounded backing arrays.

Done means one of two outcomes:

• a focused PR lands with before/after evidence reducing the reproduced getEntrySlice allocation and no material Ironbird runtime/TPS regression; or
• the issue records a concrete classification showing the allocation is required by the current flush/run ownership model, and links the next actionable allocation blocker.

Why This Exists

After #3531, the accepted Ironbird rows still show getEntrySlice as the largest shared TreeDB-owned flat allocator in both target workloads. This is now a better next target than zstd compression, command-WAL payload pieces, or lower-volume adapter paths.

This loop should be driven by the actual Ironbird heap path. Package benchmarks and unified-bench are useful for iteration, but a local microbench-only win is not enough if accepted Ironbird heap allocation does not improve or is only re-attributed elsewhere.

Current Evidence

Candidate artifact root after #3531:

• /mnt/fast4tb/ironbird-3531-memtable-allocs-20260706T094142Z

Accepted rows:

Workload	Heap profile	Total alloc_space	getEntrySlice flat alloc_space	Runtime context
plain-send	plain-send/treedb/attempt-1/pprof/simapp-treedb-all-validator-0-heap.pprof	122.12 GiB	8.35 GiB / 6.83%	199,534 tx over 340.53s, 585.96 TPS
small-multisend	small-multisend/treedb/attempt-1/pprof/simapp-treedb-all-validator-0-heap.pprof	145.72 GiB	9.84 GiB / 6.76%	239,166 tx over 405.02s, 590.50 TPS

Current top TreeDB-relevant flat allocators from those accepted profiles:

Workload	Site	Flat alloc_space
plain-send	TreeDB/caching.getEntrySlice	8.35 GiB
plain-send	TreeDB/zipper.(*ReadOnlyPrepareResult).cloneKey	7.65 GiB
plain-send	TreeDB/internal/memtable.getAppendOnlyEntries	5.60 GiB
plain-send	compress/zstd.(*fastBase).ensureHist	3.77 GiB
small-multisend	TreeDB/caching.getEntrySlice	9.84 GiB
small-multisend	TreeDB/internal/memtable.getAppendOnlyEntries	7.25 GiB
small-multisend	compress/zstd.(*fastBase).ensureHist	4.40 GiB
small-multisend	TreeDB/caching.(*Batch).SetViewValidatedWithRevision	3.41 GiB

Initial pprof stack classification from the coordinator/explorer pass:

• dominant getEntrySlice path appears to be caching.buildOpRuns under canonical point-unit flush/run materialization, especially buildCanonicalUnitRuns / flushCanonicalPointUnitsStreamed.
• small checkpoint-frontier paths exist but are secondary.

Scope

Included:

• TreeDB/caching.getEntrySlice and putEntrySlice pooling behavior.
• TreeDB/caching.buildOpRuns and canonical point-unit flush/run materialization paths.
• Entry-slice pool sizing, admission/drop behavior, chunk classes, reuse across buildCanonicalUnitRuns, and duplicate []batch.Entry materialization if proven by profiles.
• Counters or focused benchmarks that explain requested slice lengths, hits/misses, drops, oversize classes, and retained bytes.

Excluded:

• Append-only memtable key/value ownership work from TreeDB alloc loop: classify public Set clone object churn #3529/treedb: reduce append-only memtable allocations #3531.
• SetView / SetWithRevision / CometBFT adapter work from TreeDB alloc loop: minimize Ironbird Batch.SetWithRevision allocations #3526/TreeDB alloc loop: route CometBFT DB readonly Batch.Set to TreeDB SetView #3527/TreeDB/caching: reserve pointer batch index metadata #3528.
• Zipper split/fence work from TreeDB alloc loop: reduce zipper merge/split object churn #3530/treedb: reduce zipper split allocation churn #3532 unless line-level profiles prove the same getEntrySlice contract is shared.
• Cross-repo snissn/compress zstd work.
• Command-WAL frame rewrites or RawKV payload arena work unless this loop first proves getEntrySlice is no longer the dominant blocker.
• WAL, value-log, on-disk format, durability, or public API semantics changes.

North-Star Gates

Gate	Current	Target	Required evidence	If the gate fails
Plain-send getEntrySlice alloc_space	8.35 GiB	reduce by at least 10%, or prove required/source-shape limited	accepted Ironbird plain-send heap profile plus focused before/after benchmark/profile	keep issue open or close only with explicit classification and next blocker
Small-multisend getEntrySlice alloc_space	9.84 GiB	reduce by at least 10%, or prove required/source-shape limited	accepted Ironbird small-multisend heap profile plus focused before/after benchmark/profile	keep issue open or close only with explicit classification and next blocker
Runtime/TPS	plain 585.96 TPS; small 590.50 TPS from post-#3531 rows	no material regression; default blocker threshold worse than 2% unless repeat evidence shows noise	accepted Ironbird rows with >=300s load windows when macro validation is run	profile/fix before merge or document explicit waiver
Memory retention	bounded existing pool behavior	no unbounded retained backing arrays and no pathological RSS/high-water growth	tests/counters around class caps, drops, retained bytes, and reset behavior	do not merge
Ownership semantics	flush/run materialization must own mutable data as needed	no aliasing or lifetime regression	focused tests around mutation/lifetime contracts affected by any reuse change	do not merge

Root-Cause Classification Gate

Before implementation, classify getEntrySlice allocation into one or more of:

Candidate limiter	Evidence to collect	Action if confirmed
pool miss/admission drop	counters for requested length, class, hit/miss/drop, retained bytes	tune bounded pool classes/admission with tests
oversize slice class	requested vs granted cap and excess bytes by path	adjust classing or avoid over-rounding
duplicate materialization	line-level profile showing []batch.Entry copied/allocated more than once per run/unit	remove duplicate copy only where ownership is safe
required ownership copy	code/path proof that source entries cannot be safely borrowed	record classification and move to next blocker
benchmark/harness shape	repeated profiles disagree or counters show low local reproducibility	do not merge a claimed optimization without stable evidence

Required Workflow

Start phase:

• Use go tool pprof terminal mode on the two accepted Ironbird heap profiles to record top, top -cum, and line/trace evidence for getEntrySlice.
• Add or use focused counters/benchmarks for getEntrySlice request length/class/hit/miss/drop/retained-byte behavior if the current code does not expose enough path proof.
• Reproduce the allocation locally with a focused package benchmark or unified-bench profile-dir run before implementation.

Implementation phase:

• Make one scoped change at a time.
• Add deterministic tests for any pooling, ownership, reset, or retention-contract change.
• Keep retention bounded and fail-closed under memory pressure.

Close phase:

• Re-run focused package tests for changed packages.
• Re-run the focused before/after benchmark/profile and record exact commands, commits, and artifact paths.
• Update the PR with benchmark/profile tables and the gate result.
• Run accepted Ironbird plain-send and small-multisend TreeDB rows using the candidate gomap commit if focused evidence predicts macro-visible allocation/GC/runtime impact.
• Latest-head CI and mature review gates are required before merge.

Branch And PR Policy

• Work on a topic branch from current origin/main.
• Do not push directly to main.
• PR must be mergeable before merge: latest-head CI, focused tests, benchmark evidence, and review findings resolved or explicitly rejected with rationale.
• Do not request review-credit-consuming AI reviews until the PR is mature enough to avoid churn.

[snissn]

Worker update for #3533: opened PR #3534: #3534

Classification:

• getEntrySlice flat alloc_space is primarily hot entry-slice pool miss/GC churn in canonical point-unit flush/run materialization, not value-log/WAL behavior and not per-entry unstable iterator copying.
• Accepted Ironbird pprof caller split:
  • plain-send: total 122.12 GiB, getEntrySlice flat 8.35 GiB / 6.83%; buildOpRuns caller 7902.33 MiB / 92.44%, direct flushCanonicalPointUnitsStreamed 645.85 MiB / 7.56%.
  • small-multisend: total 145.72 GiB, getEntrySlice flat 9.84 GiB / 6.76%; buildOpRuns caller 9237.89 MiB / 91.63%, direct flushCanonicalPointUnitsStreamed 844.28 MiB / 8.37%.
• Trace sample sizes were mostly 704 KiB and 1.38 MiB, matching 8192/16384 []batch.Entry backing arrays. Warmed local buildOpRuns benchmarks were already near-zero allocation, so the avoidable part is retaining hot chunk classes across GC instead of relying on sync.Pool beyond the small strong lease bucket.

Change:

• Raise bounded maxEntrySliceLeasesPerBucket from 128 to 512.
• Add BenchmarkEntrySliceLeaseHotClassAfterGC to reproduce the GC-churn path.
• Existing entrySlicePoolBudgetBytes and pool-pressure trimming remain the retention bounds; high pressure trims leases and critical pressure drops them.

Measured local gate:

GOWORK=off TMPDIR=/mnt/fast4tb/tmp go test ./TreeDB/caching -run '^$' -bench 'BenchmarkEntrySliceLeaseHotClassAfterGC$' -benchmem -count=5

• before: about 46,148,700 B/op, 145 allocs/op
• after: 299-362 B/op, 1 alloc/op

Additional after smoke:

GOWORK=off TMPDIR=/mnt/fast4tb/tmp go test ./TreeDB/caching -run '^$' -bench 'BenchmarkEntrySliceLeaseHotClassAfterGC$' -benchmem -benchtime=20x -memprofile /mnt/fast4tb/gomap-3533-profiles/entryslice-hot-after.mem.pprof

Result: 10555103 ns/op, 646 B/op, 1 allocs/op.

Warmed build-run benchmark check:

GOWORK=off TMPDIR=/mnt/fast4tb/tmp go test ./TreeDB/caching -run '^$' -bench 'BenchmarkBuildOpRuns(Allocs|_ModeCompare)$' -benchmem -count=3

BenchmarkBuildOpRunsAllocs remained ~95-98 B/op and 2 allocs/op after the change; mode-compare cases stayed in the existing low-allocation bands.

Tests:

GOWORK=off TMPDIR=/mnt/fast4tb/tmp go test ./TreeDB/caching -run 'Test(EntrySlice|PutEntrySlice|GetEntrySlice|MaybeResetEntrySlice|PoolPressure.*EntrySlice|BuildOpRunsChunking)'
GOWORK=off TMPDIR=/mnt/fast4tb/tmp go test ./TreeDB/caching

Both passed.

No long Ironbird row was run in this worker loop, per instructions. No AI review requested yet.

[snissn]

Resolved as a measured non-merge/rejection.

The worker correctly identified as a hot allocation site and produced PR #3534, but the accepted Ironbird validation rows showed the candidate is not a net win for this objective.

Accepted rows against the candidate gomap ref :

• plain-send: alloc_space improved 8546.18MB -> 7555.58MB (-11.59%), but total alloc_space worsened +6.76%, TreeDB flat allocation worsened +13.04%, TPS moved 585.96 -> 572.41 (-2.31%), commit +25.35%, finalize +15.39%.
• small-multisend: alloc_space improved 10079.66MB -> 8923.69MB (-11.47%), but total alloc_space worsened +2.72%, TreeDB flat allocation worsened +7.16%, TPS moved 590.50 -> 585.76 (-0.80%), commit +16.39%, finalize +18.77%.

Artifacts:

• baseline:
• candidate:
• validation pin in ironbird: branch , commit

Outcome: close this loop without merge. The next loop should target a larger actual-Ironbird TreeDB allocator rather than trying to keep this cap increase.

[snissn]

Corrected evidence note for the measured non-merge/rejection.

The worker correctly identified TreeDB/caching.getEntrySlice as a hot allocation site and produced PR #3534, but the accepted Ironbird validation rows showed the candidate is not a net win for this objective.

Accepted rows against the candidate gomap ref e932a6022ac1296173e9cfb75dd17da17a7e0091:

• plain-send: getEntrySlice alloc_space improved 8546.18MB -> 7555.58MB (-11.59%), but total alloc_space worsened +6.76%, TreeDB flat allocation worsened +13.04%, TPS moved 585.96 -> 572.41 (-2.31%), commit +25.35%, finalize +15.39%.
• small-multisend: getEntrySlice alloc_space improved 10079.66MB -> 8923.69MB (-11.47%), but total alloc_space worsened +2.72%, TreeDB flat allocation worsened +7.16%, TPS moved 590.50 -> 585.76 (-0.80%), commit +16.39%, finalize +18.77%.

Artifacts:

• baseline: /mnt/fast4tb/ironbird-3531-memtable-allocs-20260706T094142Z
• candidate: /mnt/fast4tb/ironbird-3534-getentryslice-allocs-20260706T102500Z
• validation pin in ironbird: branch codex/3533-getentryslice-ironbird-validation, commit 17f2e88

Outcome: close this loop without merge. The next loop should target a larger actual-Ironbird TreeDB allocator rather than trying to keep this cap increase.