TreeDB alloc loop: minimize Ironbird execution allocation

[snissn]

Goal

Minimize allocation from TreeDB code paths during actual Ironbird execution, with plain-send as the first priority and small-multisend as the second priority.

Done means the active loop process has either landed every reasonable PR-sized allocation reduction visible in accepted Ironbird TreeDB heap profiles, or recorded measured rejections/blockers for the remaining allocator families with enough evidence that continuing would be speculative churn.

Why This Exists

The useful optimization target is not a generic microbenchmark allocation number. The source of truth is accepted Ironbird execution: full-stack TreeDB rows, real Cosmos transaction workloads, heap profiles captured from the validator, and runtime/storage timing from the same run.

Package benchmarks and unified-bench are still required, but only as repro and iteration tools. A loop is not successful unless it reduces the targeted Ironbird TreeDB allocation family, keeps total/TreeDB-focused allocation neutral or better, and does not create a material throughput or ABCI timing regression.

Current Operating Model

Each allocation loop should have its own child issue and scoped PR or measured rejection.

1. Start from an accepted Ironbird heap profile and rank TreeDB-owned flat alloc_space sites.
2. Create or update one child issue for the selected allocator family.
3. Use codex-issue-graph-executor for the child graph state, implementation branch, PR, evidence, and merge gate.
4. Add or use a focused local repro only after the Ironbird allocator target is confirmed.
5. Run focused tests and local allocation benchmarks/profiles before macro validation.
6. Run accepted Ironbird plain-send validation for plain-send targets.
7. Run accepted Ironbird small-multisend guard when the changed path is shared or when small-multisend is the target.
8. Merge only if latest-head CI/review gates pass and no material performance regression remains.
9. If the candidate fails the macro gate, close or mark the child rejected with exact artifacts and move to the next Ironbird-ranked allocator family.

Current Graph State

Active child:

Issue	PR	State	Branch/head	Contract surface	Current gate
#3566	#3567 draft	macro validation running	codex/3566-readonly-prepare-clonekey / 3848fabeb9a1fc1ef570bb1fc39a7fe5c6932c98	TreeDB/zipper/read_only_prepare.go, focused tests/benchmarks in TreeDB/zipper	plain-send passed allocator gate; small-multisend guard running before PR readiness

Current #3566 target:

Workload	Baseline artifact	Baseline gomap	Target allocator	Baseline alloc_space
plain-send TreeDB	/mnt/fast4tb/ironbird-3566-current-main-plain-20260706T162213Z/plain-send/treedb/attempt-1	6f2d94def23551ccb1c6bdb35514bc72733d88e0	zipper.(*ReadOnlyPrepareResult).cloneKey	9218.10 MB

Candidate #3567 plain-send result:

Metric	Baseline	Candidate	Delta	Decision
total alloc_space	118380.47 MB	110324.47 MB	-6.8%	pass
TreeDB-focused flat alloc_space	26753.94 MB	18948.87 MB	-29.2%	pass
ReadOnlyPrepareResult.cloneKey	9218.10 MB	no focused samples	effectively removed	pass
load-window TPS	583.29	578.45	-0.83%	neutral within default gate

Recent accepted/rejected loop policy:

• PR-sized wins should be landed only when accepted Ironbird rows support them.
• Microbench wins that worsen total Ironbird allocation, TreeDB flat allocation, commit/finalize/check timing, or throughput should be rejected even if the named symbol improves.
• Prior rejected retention/pool-only loops stay rejected unless a fresh Ironbird profile and a different implementation strategy justify a new child issue.

North-Star Gates

Gate	Current standard	Required evidence	If the gate fails
Target allocator reduction	selected TreeDB flat site should materially drop, normally at least 10% unless the issue is explicitly instrumentation/safety-only	before/after accepted Ironbird heap profiles with same workload shape and gomap commits	fix, reject candidate, or open/link the next measured allocator issue
Total allocation	total validator alloc_space should be neutral or lower	same heap profiles and total pprof top	reject or profile/fix before merge
TreeDB-focused allocation	TreeDB-focused flat allocation should be neutral or lower	TreeDB-focused pprof summaries	reject or profile/fix before merge
Runtime throughput	no material TPS regression; default guard is not worse than 1% without explicit acceptance	accepted Ironbird load-window metrics with at least a 300s window	rerun if noisy; otherwise fix/reject
ABCI/storage timing	no material commit/finalize/check regression attributable to the change	Ironbird storage_signal_summary and CPU/heap profiles	profile/fix or reject before merge
Correctness/CI	focused tests, affected package tests, latest-head CI and review gates pass	local test logs plus GitHub latest-head checks/reviews	do not merge

Scope

Included:

• TreeDB allocation families visible in accepted Ironbird plain-send and small-multisend heap profiles.
• Child issues and PRs for allocator families such as read-only prepare key cloning, memtable append-only entry backing, command-WAL payload build/read/decode allocation, batch Set/SetView allocation, value-log compression/buffer allocation, and caching/flush materialization allocation when they are visible in Ironbird profiles.
• Focused local repros and package benchmarks that explain and validate an Ironbird-observed allocation site.
• Full-stack TreeDB Ironbird rows for macro-visible candidates.

Excluded:

• Optimizing microbench-only sites that do not show up in accepted Ironbird execution.
• Broad rewrites without a ranked Ironbird allocator target.
• Public API changes, on-disk format changes, durability-policy changes, or legacy slab/value-store concepts unless a future child issue explicitly scopes and justifies them.
• Opening broad PRs that mix unrelated allocator families.

Branch And PR Policy

• Work on topic branches.
• Do not push directly to main.
• One PR should map to one child allocator loop unless a coordinator comment explicitly changes the graph.
• PRs must be mergeable before merge: latest-head CI, focused tests, benchmark/profile evidence, and review findings resolved or explicitly rejected with rationale.
• Codex, Copilot, CodeRabbit, or other review-credit-consuming AI reviews should be requested only after the PR is mature: coherent code, focused tests, required benchmark evidence or rationale, current PR body/status, no known local blockers, and latest-head CI running or green.
• Material performance regressions are blocking unless minimized and explicitly accepted by the coordinator with evidence.

Required Child Issue Shape

Every child allocation loop must include:

• The exact Ironbird artifact path and gomap commit that selected the target.
• The selected TreeDB flat allocator, alloc_space, total alloc_space, TreeDB-focused allocation, TPS, and ABCI timing baseline.
• The local reproducer or clear rationale if no local reproducer exists.
• The exact tests and benchmarks required before macro validation.
• The accepted Ironbird validation command/shape and artifact path.
• A clear closeout: merged PR with evidence, or rejected candidate with artifacts and next allocator issue.

[snissn]

Codex Issue Graph State

Coordinator state after correcting the allocation loop framing. The source of truth for this workstream is Ironbird TreeDB heap alloc_space from accepted plain-send and small-multisend rows. Package benchmarks and unified-bench are iteration tools; they are not sufficient by themselves for macro optimization claims.

Graph policy:

• Rank TreeDB-owned allocation sites from accepted Ironbird plain-send and small-multisend heap profiles.
• Open/use one child issue per loop, with one scoped branch/PR candidate or a measured rejection.
• Validate each candidate with focused tests/benchmarks first, then accepted Ironbird rows when macro-visible allocation/GC/runtime impact is plausible.
• Do not run broad matrices before a candidate has profile evidence that could plausibly move Ironbird TreeDB allocation.
• Treat disappearing pprof symbol attribution as insufficient when total TreeDB allocation, GC, or hot-path time worsens.
• Merge only through latest-head CI/review/performance gates; record caveats on the PR/tracker before merge.

Resolved latest loop:

Node	State	Branch/PR	Contract surface	Evidence / closeout
#3529 public Set clone object churn	closed	PR #3531 merged, commit 7d7a968fdcb64d5ddebaea45d086487fc6cee0aa	TreeDB/internal/memtable append-only small non-inline key ownership and bounded entry backing pool retention	Focused package tests/benchmarks passed; latest-head CI green; accepted Ironbird rows had no observed accepted-row TPS regression. Closeout is scoped allocation hygiene, not a full parent allocation win.

Current running loop:

Node	State	Branch/PR	Contract surface	Current evidence / gate
#3533 Ironbird getEntrySlice flush run allocations	running	branch/PR pending	TreeDB/caching.getEntrySlice, putEntrySlice, buildOpRuns, canonical point-unit flush/run materialization	Baseline from accepted post-#3531 profiles: plain-send getEntrySlice 8.35 GiB of 122.12 GiB; small-multisend getEntrySlice 9.84 GiB of 145.72 GiB. Gate: reduce by at least 10% in both accepted Ironbird rows, or record a concrete required-allocation classification and next blocker.

Latest accepted post-#3531 top TreeDB-relevant flat alloc_space:

Workload	Site	Flat alloc_space
plain-send	TreeDB/caching.getEntrySlice	8.35 GiB
plain-send	TreeDB/zipper.(*ReadOnlyPrepareResult).cloneKey	7.65 GiB
plain-send	TreeDB/internal/memtable.getAppendOnlyEntries	5.60 GiB
plain-send	compress/zstd.(*fastBase).ensureHist	3.77 GiB
small-multisend	TreeDB/caching.getEntrySlice	9.84 GiB
small-multisend	TreeDB/internal/memtable.getAppendOnlyEntries	7.25 GiB
small-multisend	compress/zstd.(*fastBase).ensureHist	4.40 GiB
small-multisend	TreeDB/caching.(*Batch).SetViewValidatedWithRevision	3.41 GiB

Current graph inventory:

Node	State	Branch/PR	Contract surface	Notes
#3516 value-log append-buffer retention	merged	PR #3516, merge commit 914b7f1d028de1b03b25b5d0df4555467d102e32	TreeDB/internal/valuelog bounded append-buffer retention	Merged as scoped allocation hygiene.
#3517 append-only reserve-growth candidate	closed, not planned	no PR	rejected candidate under getAppendOnlyEntries/getEntrySlice	Candidate failed local allocation gate; parent allocation site remains open.
#3522 RawKV payload allocation candidate	closed, not planned	no PR	one adapter-arena replacement candidate	Removed named site locally but regressed total allocation/time; closed as rejected candidate.
#3525 append-only allocation classification	closed, not planned	no PR	getAppendOnlyEntries backing-array growth/replacement	Classified, no safe bounded PR-sized optimization from that attempt.
#3528 SetWithRevision metadata/slice candidate	open	PR #3528, branch codex/3526-setwithrevision	TreeDB/caching.(*Batch).SetWithRevision metadata/slice path	Existing PR; not the dominant current accepted Ironbird site. Re-evaluate after #3533/#3527 ordering.
#3527 / cometbft-db#5 SetView forwarding	open	cometbft-db PR #5, branch codex/3527-treedb-setview	external CometBFT readonly batch path into TreeDB SetView	Potentially relevant to Batch.SetViewValidatedWithRevision, but smaller than getEntrySlice/getAppendOnlyEntries in latest accepted rows.
#3532 zipper allocation candidate	open	PR #3532, branch codex/3524-3530-zipper-allocs	zipper split/fence allocation churn	Plain-send still shows cloneKey as large; verify exact current ownership before advancing.
successor loop	pending	create/select after #3533 resolves	next dominant Ironbird allocation site	Current candidates after #3533: memtable.getAppendOnlyEntries, zipper cloneKey, or Batch.SetViewValidatedWithRevision, depending on refreshed accepted Ironbird profiles.

#3533 execution gates:

• Work from current origin/main after treedb: reduce append-only memtable allocations #3531 merge.
• Start with line/trace evidence from accepted Ironbird heap profiles and a local focused reproducer/counter path.
• Optimize only bounded getEntrySlice/flush-run allocation behavior; do not touch memtable, SetView, zipper, zstd, command-WAL, durability, value-log, or on-disk behavior in this loop.
• Add tests for any pooling/retention/ownership/lifetime contract changed.
• Run focused tests and before/after benchmark/profile evidence before requesting review.
• Run accepted Ironbird plain-send and small-multisend rows only after focused evidence predicts macro-visible improvement.

[snissn]

Update: opened focused allocation-hygiene PR #3516 for the value-log append-buffer pool candidate.

What changed:

• clean branch codex/3515-vlogpool16-allocs, one commit over current main: 3226376e2e0d09b60d3f24ac6f3681fc621640d8
• increases the bounded default value-log append-buffer pool from 8 to 16 entries
• adds BenchmarkWriterAppendBufPoolConcurrentSyncCycles to reproduce the 16-writer sync-cycle churn
• adds BenchmarkAppendOnlyBatchReserveGrowth as a guardrail for the rejected memtable reserve-growth lane

Exact-branch evidence:

• go test ./TreeDB/internal/memtable ./TreeDB/internal/valuelog ./TreeDB/caching -count=1 passed
• BenchmarkWriterAppendBufPoolConcurrentSyncCycles, 100x x3, reports 0 appendbuf_drops/op and only first-cycle warmup allocation amortized across the run

Directional Ironbird evidence only:

• artifact root /mnt/fast4tb/ironbird-normal-workload-sweep-gomap-76878f-20260706T064704Z
• same value-log pool diff on the prior stacked validation branch removed getWriterAppendBuf from the top alloc_space sites in both plain-send and small-multisend
• total alloc_space moved from 115,880 MB -> 111,336 MB on plain and 144,442 MB -> 138,571 MB on small, with neutral/slightly lower TPS
• not exact-branch merge evidence because that run used stacked gomap commit 76878fbc; exact branch is 3226376e

Process adjustment:

• stopped the duplicate exact-branch Ironbird rerun before completion to avoid spending more wall time on a small hygiene PR
• next optimization should not start until the next candidate plausibly moves total allocation/GC by >10% or has a clear correctness/operational reason