Skip to content

perf: batch outbound writes with whole-buffer encryption and atomic bundles - #545

Open
carabistouflette wants to merge 5 commits into
Steel-Foundation:masterfrom
carabistouflette:pr1-batched-outbound-writes
Open

perf: batch outbound writes with whole-buffer encryption and atomic bundles#545
carabistouflette wants to merge 5 commits into
Steel-Foundation:masterfrom
carabistouflette:pr1-batched-outbound-writes

Conversation

@carabistouflette

@carabistouflette carabistouflette commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Type of change

  • Block implementation
  • Item implementation
  • Command implementation
  • Entity implementation
  • Bug fix
  • New feature
  • Breaking change
  • Refactor / code cleanup
  • Performance improvement

Description

Overhauls the Java outbound write path. It also adds the criterion benchmarks used for the
measurements below (steel-protocol/benches/packet_codec.rs: whole-buffer encryption
throughput, and per-packet-flush vs batched-single-flush socket write patterns), kept in-tree
for future regression tracking:

  • StreamEncryptor encrypts whole buffers at once (mirroring vanilla CipherBase.encipher) instead of running one async poll_write per byte, and retains partially written encrypted bytes across polls so a partial socket write can never desync the CFB8 stream.
  • New OutboundPacket::Bundle variant: bundle delimiters and sub-packets are queued as a single channel message, eliminating the multi-producer interleaving race where another packet could land between a bundle's delimiters.
  • TCPNetworkEncoder::write_packets writes a batch of packets with a single flush; both sender tasks (pre-play and play) drain up to 128 queued packets per writer-lock acquisition via recv_many.
  • A batch write that fails partway drops the encoder instead of returning it to the shared writer slot: mid-packet CFB8 cipher state must never be reused. write_outbound_batch/OUTBOUND_BATCH_SIZE are shared between the play and pre-play sender loops in player::connection.

How this was tested

  • cargo test -p steel-core --lib connection::java (16 tests, incl. new send_encoded_bundle_sends_one_atomic_bundle_message), cargo test -p steel-core --lib player::connection (adds failed_batch_write_discards_the_encoder and successful_batch_write_restores_the_encoder for the poison-on-failure behavior), cargo test -p steel-protocol --lib (new encrypts_whole_buffers_like_vanilla_cipher and retains_unwritten_encrypted_bytes_across_partial_writes), cargo test -p steel-login, cargo clippy -r --all-targets clean.
  • Benchmarks: numbers and replication steps in Measurements below.

Measurements

Run back-to-back on the same idle machine (Linux x86_64 laptop, Intel Core Ultra 7 258V, Rust
nightly, criterion defaults: 3 s warmup, 5 s measurement, 100 samples; loopback TCP with
TCP_NODELAY and a concurrent drain task). Absolute numbers are machine-specific; the ratios
are what matter.

Outbound write pattern, 128 encrypted CKeepAlive packets per batch over loopback:

sender write path encryptor mean time / batch throughput
master (36156e9): write + flush per packet byte-at-a-time 238.6 µs 0.54 Melem/s
this PR: write + flush per packet whole-buffer 223.1 µs 0.57 Melem/s
this PR: batched, single flush whole-buffer 27.9 µs 4.59 Melem/s

Batching alone gives ~8.6x more outbound throughput per writer-lock acquisition vs the
master send path — one write syscall per batch instead of one per packet. The encryptor
change contributes ~0: the per-packet rows show it is within noise of master.

Whole-buffer CFB8 encryption throughput into a non-backpressuring sink (isolates the encryptor
from the socket):

buffer size master byte-at-a-time this PR whole-buffer
64 B 44.8 MiB/s 45.0 MiB/s
512 B 45.6 MiB/s 47.8 MiB/s
4096 B 48.1 MiB/s 47.4 MiB/s
64 KiB 48.1 MiB/s 48.0 MiB/s

Throughput parity (AES dominates; ≤ run-to-run noise). The encryption win is elsewhere: one
task reschedule per buffer instead of per byte when the socket backpressures, and no dropped
bytes when the inner writer returns Ok(0).

How to replicate

On this branch (all three arms above except the master row):

cargo bench -p steel-protocol --bench packet_codec

For the master baseline row, run the same bench against the commit this PR branches from.
The bench needs write_packets, which only exists on this branch, so copy it onto a master
worktree and drop the batched arm:

git worktree add --detach /tmp/steel-master 36156e984
mkdir -p /tmp/steel-master/steel-protocol/benches
git show pr1-batched-outbound-writes:steel-protocol/benches/packet_codec.rs \
  > /tmp/steel-master/steel-protocol/benches/packet_codec.rs
cat >> /tmp/steel-master/steel-protocol/Cargo.toml <<'EOF'

[dev-dependencies]
criterion = { workspace = true, features = ["async_tokio"] }

[[bench]]
name = "packet_codec"
harness = false
EOF
# delete the `group.bench_function("batched_single_flush", ..)` block from
# /tmp/steel-master/steel-protocol/benches/packet_codec.rs (`write_packets` does not exist on master)
cd /tmp/steel-master && cargo bench -p steel-protocol --bench packet_codec

Screenshots / logs

N/A (no visual surface).

Checklist

  • Code builds w/o errors or warnings
  • Self-reviewed the diff
  • Docs updated (if applicable) — N/A, no docs cover this area yet
  • No leftover debug code / comments

Additional notes

  • Stack: pr2-outbound-byte-budget is stacked on this branch.
  • Env: Linux, Rust nightly, Minecraft protocol 776 (0.15.2+mc26.2).

…undles

- StreamEncryptor encrypts whole buffers at once (vanilla CipherBase.encipher
  model) instead of one async write per byte, retaining partially written
  encrypted bytes across partial writes and pending polls
- OutboundPacket::Bundle makes bundle sends atomic: delimiters and sub-packets
  are queued as a single message and written under one writer lock, without
  interleaving other packets
- TCPNetworkEncoder::write_packets writes a batch with a single flush; sender
  tasks drain up to 128 queued packets per lock acquisition via recv_many

Measured with criterion on a loopback socket (128 encrypted packets):
- batched single flush: 4.59 Melem/s vs 0.57 Melem/s per-packet flush (~8x)
- whole-buffer encryption: parity within noise vs per-byte loop on
  non-backpressuring sinks; the win is one pending-reschedule per buffer
  instead of per byte under backpressure
- regression tests cover bundle atomicity and encryptor buffer retention
@carabistouflette
carabistouflette force-pushed the pr1-batched-outbound-writes branch from 639b3aa to 4b0b9e9 Compare August 29, 2026 10:29
Batch writes took the encoder by shared reference, so a mid-batch failure
left an encoder with mid-packet CFB8 state installed; the queued-disconnect
and immediate-write paths would then append to a partially written packet.
Hoist write_outbound_batch and OUTBOUND_BATCH_SIZE into player::connection
and take the encoder out of the shared slot, restoring it only after the
batch and flush succeed. Covers the steel-core and steel-login sender loops.
State the caller contract precisely (continuation retry after a partial
write, whole-buffer retry after Pending) and note that poll_flush does not
push bytes retained from a partial write.
Comment thread steel-core/src/player/connection/java.rs
Comment thread steel-protocol/src/utils.rs Outdated
- StreamEncryptor now uses commit-all buffering (BufWriter semantics): every
  written buffer is reported fully consumed and its ciphertext is retained
  until the inner writer accepts it. A sink that accepts partially, or drops
  bytes past the reported count, can no longer desync the CFB8 stream.
  poll_flush and poll_shutdown drain retained ciphertext before resolving.
- write_outbound_batch bounds the write with OUTBOUND_WRITE_TIMEOUT (30 s):
  a client that stops reading can no longer wedge the sender task while
  holding the writer lock. On timeout the encoder is discarded, poisoning
  the writer slot the same way a failed batch does.

Covers the two CHANGES_REQUESTED threads on Steel-Foundation#545: the abc/XY repro reported
by kdcokenny now produces abcXY, and a stalled client cannot block kicks or
shutdown indefinitely.
@coco875

coco875 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

I would like an confirmation that it don't do something similar to with #473

@github-actions

Copy link
Copy Markdown

This pull request has conflicts with the base branch "master". Please resolve those so we can test out your changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants