[comparison draft] unsafe update-combiner throughput ceiling PoC

[snissn]

Summary

Comparison-only draft throughput proof-of-concept for issue #1500: sharded update-combiner lane workers now prepare direct buffered update plans concurrently by document-ID shard, then merge compatible prepared plans back into one global buffered/publish path.

This intentionally validates throughput headroom first and is not a merge-to-main candidate as-is. It is stacked on codex/update-combine-bsonset-id-fastpath and should remain draft/profiling-only until the production conflict protocol is implemented.

Comparison branch boundary

• Keep this PR as a draft comparison branch.
• Use its benchmark numbers as the unsafe upper-bound target for later safe implementations.
• Do not treat the lane-worker, stale-plan, or unsafe async-ack modes as production semantics.
• Before landing equivalent behavior on main, add explicit conflict handling, read visibility, backpressure, and durability semantics.

What changed

• Added profiling-only update-combiner lane workers gated by CollectionManager.SetUpdateCombineLaneWorkersForProfiling.
• Added a prepared-plan merger that combines compatible direct buffered update plans and stages them as one global direct-buffer batch.
• Preserved final publish economics in the benchmark shape: post-run rows still report indexed_flush_calls=1 and indexed_flush_docs/batch=100000.
• Added profiling env knobs in cmd/mongo_gateway_bench:
  • MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_SHARDS=N
  • MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_LANE_WORKERS=true
  • MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_STALE_DIRECT_PLANS=true
• Added docs and a stats regression test so merged prepared-plan batches retain per-index detail counters.

Safety boundary

This is not a production correctness contract yet.

The MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_STALE_DIRECT_PLANS=true flag allows controlled benchmark runs to stage direct buffered update plans that were built before another lane advanced the buffered generation. Use it only for shapes with non-overlapping document IDs and unchanged unique secondary indexes.

Before making this production, the next PR needs explicit conflict/hot-key handling: per-document serialization, duplicate-key fallback, unique-secondary-change fallback, and transaction/coordinator boundaries.

Benchmark evidence

Command family used for the post-cleanup comparison:

GOWORK=off \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_DOCUMENTS=100000 \
MONGO_GATEWAY_PROFILE_BENCH_WRITERS=<32|64> \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_SHARDS=<1|8> \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_LANE_WORKERS=<false|true> \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_STALE_DIRECT_PLANS=<false|true> \
go test ./cmd/mongo_gateway_bench \
  -run '^$' \
  -bench '^BenchmarkDirectCollectionConcurrentUpdateBSONIndexes2$' \
  -benchtime=100000x \
  -count=3 \
  -benchmem

Artifacts:

• Full writer sweep: /tmp/gomap_lane_merge_poc_sweep_20260513_100859/sweep.txt
• Post-cleanup 32/64 comparison: /tmp/gomap_lane_merge_poc_post_stats_compare_20260513_101424/compare.txt

Post-cleanup 3-run averages:

mode	writers	docs/sec	ns/op	update_combine_wait_ns/doc	queue_wait_ns/doc	run_ns/doc	buffer_lock_hold_ns/doc	buffer_stage_ns/doc	indexed_flush_calls	indexed_flush_docs/batch	allocs/op
base	32	189,888	5,267	132,908	35,271	3,804	308.4	321.3	1.0	100,000	1.0
lane-merge PoC	32	334,393	2,997	55,764	5,017	5,562	162.4	187.4	1.0	100,000	7.0
base	64	192,928	5,188	254,441	42,863	3,614	264.5	270.6	1.0	100,000	1.0
lane-merge PoC	64	369,188	2,714	87,186	7,543	5,434	105.6	119.9	1.0	100,000	5.0

Relative throughput:

• 32 writers: 334,393 / 189,888 = +76.1%
• 64 writers: 369,188 / 192,928 = +91.4%

Earlier full writer sweep showed the same scaling shape across concurrency:

• w1: +2.5%
• w4: +12.9%
• w8: +41.3%
• w16: +67.4%
• w32: +80.1%
• w64: +113.6%

Validation

GOWORK=off go test ./TreeDB/collections ./cmd/mongo_gateway_bench
git diff --check

Profile pass: lane-batching tune

Additional commit: fef3c11a tune sharded update combiner lane batching.

I profiled the current PoC at 64 writers with CPU, heap/alloc, block, and mutex profiles.

Artifacts:

• CPU-only profiles: /tmp/gomap_update_combiner_poc_cpuonly_20260513_103021
• Heavy alloc/mutex/block profiles: /tmp/gomap_update_combiner_poc_profiles_20260513_102904
• Lane drain experiments:
  • /tmp/gomap_lane_drain16_compare_20260513_103140
  • /tmp/gomap_lane_drain4_compare_20260513_103243
  • /tmp/gomap_lane_drain2_compare_20260513_103325
  • /tmp/gomap_lane_pool_drain4_compare_20260513_103510

Profile summary:

• The current PoC is no longer primarily blocked on one global combiner/publish lane.
• CPU-only pprof for lane mode shows the largest timed-update cumulative stack under runShardWorker -> prepareBatchWithScratch -> buildUpdateBatchPlan.
• Residual CPU is mostly current-document read / leaf lookup / BSON $set replacement work, not publish lock contention.
• Mutex profile is dominated by value-log leaf append during root publish, which is final flush/root-apply work rather than the combiner ingress wall.
• Block profile is dominated by expected writer waits on per-update completion plus background writer/maintenance loops; it did not expose a new single hot lock in lane ingress.
• Allocation profile showed the lane path still pays for prepared-plan batch objects and direct buffered primary entries. I tested pooling the prepared-batch request slice, but it did not improve throughput and was not kept.

Tuning kept:

• Lane-worker drain patience increased from the global default 1 scheduler yield to a lane-only 4 yields.
• This is PoC-only tuning for sharded lane workers; the original single-combiner path still uses the old drain behavior.

Drain-tuning evidence, 100k docs, 3-run averages:

mode	writers	docs/sec	ns/op	items/batch	wait ns/doc	queue wait ns/doc	drain ns/doc	allocs/op
lane drain=4	32	328,255	3,046	1.74	59,796	4,076	3,949	6.0
lane drain=4	64	395,682	2,528	2.14	87,013	6,590	2,871	5.0

For comparison, the prior post-cleanup lane-merge run averaged 369,188 docs/sec at 64 writers. The drain=4 tune improved the high-concurrency average in this sample while keeping the same safety boundary and one final indexed flush.

Experiment rejected:

• defaultCollectionUpdateCombineLaneDrainYields=16 increased batch size but added too much drain delay; 64-writer throughput fell to 317,384 docs/sec.
• Pooling prepared-batch request slices reduced some bytes/op but hurt throughput and did not reduce alloc count, so it was reverted.

Next obvious walls after this PoC:

• Reduce current-document read / leaf lookup cost inside buildUpdateBatchPlan for non-overlapping $set updates.
• Reduce direct buffered primary-entry allocation/copy cost.
• Consider a benchmark-only direct prepared replacement path only if we intentionally want an unsafe upper ceiling distinct from the current semantic $set path.

Profile pass: unsafe async foreground-ack ceiling

Additional commit: 76c84849 add unsafe async update ack profiling mode.

This adds a profiling-only MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_ASYNC_ACK=true mode. In that mode, the foreground update path returns after the request is admitted to the sharded combiner queue. Lane workers still drain the queued requests, prepare real BSON $set update plans, stage them, and the benchmark then times DrainUpdateCombinersForProfiling() plus FlushAll() separately inside the measured window.

This is intentionally not a durability or visibility contract. It exists to measure the gap between foreground admission throughput and background update/root-publish drain throughput.

Command shape:

GOWORK=off \
MONGO_GATEWAY_PROFILE_BENCH_WRITERS=<1|64> \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_SHARDS=8 \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_LANE_WORKERS=true \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_STALE_DIRECT_PLANS=true \
MONGO_GATEWAY_PROFILE_BENCH_UPDATE_COMBINE_UNSAFE_ASYNC_ACK=<false|true> \
MONGO_GATEWAY_PROFILE_BENCH_BUFFERED_INDEXED_ASYNC_FLUSH=true \
MONGO_GATEWAY_PROFILE_BENCH_BUFFERED_INDEXED_ASYNC_FLUSH_MAX_QUEUED_UNITS=4 \
MONGO_GATEWAY_PROFILE_BENCH_BUFFERED_INDEXED_WRITE_MAX_DOCUMENTS=256000 \
MONGO_GATEWAY_PROFILE_BENCH_BUFFERED_INDEXED_WRITE_MAX_ROOT_RUNS=256000 \
go test ./cmd/mongo_gateway_bench \
  -run '^$' \
  -bench '^BenchmarkDirectCollectionConcurrentUpdateBSONIndexes2$' \
  -benchtime=300000x \
  -count=1 \
  -benchmem

Artifacts:

• Low/high sync vs async comparison: /tmp/gomap_update_async_ack_poc_20260513_115800
• Async-ack CPU/alloc/mutex/block profiles: /tmp/gomap_update_async_ack_profiles_20260513_115830
• Rejected lockless enqueue experiment: /tmp/gomap_update_async_ack_nolock_20260513_120032

Representative results, 300k docs:

mode	writers	total docs/sec	total ns/doc	foreground ack docs/sec	foreground ack ns/doc	combiner drain ns/doc	final drain ns/doc	items/batch	update current read ns/doc	allocs/op
sync lane	1	152,936	6,539	n/a	n/a	n/a	n/a	1.0	641	10
unsafe async ack	1	472,494	2,116	1,202,795	831	272	1,285	29.9	2,075	2
sync lane	64	411,679	2,429	n/a	n/a	n/a	n/a	26.8	2,202	5
unsafe async ack	64	552,016	1,812	2,132,248	469	559	1,343	255.5	2,056	1

Profile conclusion:

• The foreground admission ceiling is now above 2M updates/sec at 64 writers on this host.
• The drain-included end-to-end row is still around 0.5M updates/sec because the background workers still perform real buildUpdateBatchPlan, current-document read, BSON $set replacement, staging, and final root publish.
• CPU profile for async-ack mode still lands primarily in runShardWorker -> prepareBatchWithScratch -> buildUpdateBatchPlan, with readUpdateBatchCurrentDocument / Tree.GetAppend as the main storage-read component.
• Prepared merge/stage is small relative to lane planning. Root publish remains visible in mutex profiles through value-log leaf append, but it is not foreground admission work.
• A lockless enqueue experiment was rejected: it did not improve foreground admission in the sample and weakened the profiling mode more than necessary.