Skip to content

fix: reduce loro-js snapshot memory - #1051

Merged
zxch3n merged 2 commits into
mainfrom
codex/loro-js-lazy-snapshot-memory
Jul 21, 2026
Merged

fix: reduce loro-js snapshot memory#1051
zxch3n merged 2 commits into
mainfrom
codex/loro-js-lazy-snapshot-memory

Conversation

@zxch3n

@zxch3n zxch3n commented Jul 21, 2026

Copy link
Copy Markdown
Member

Summary

  • keep imported latest-state SSTables encoded and hydrate containers one block at a time
  • retain snapshot history as an encoded base plus a small materialized overlay for local and imported changes
  • rewrite only dirty state blocks during snapshot export and remove redundant output-buffer copies
  • add regression coverage, a real-document memory benchmark, performance notes, and a patch changeset

Why

Importing the 11.4 MB test document eagerly decoded 115,147 containers and materialized the full 423,797-operation history. A local edit then duplicated more history/state structures, and snapshot export allocated another set of encoded buffers. This amplified the document to roughly 703 MiB heap after import, over 860 MiB after one edit, and about 1.66 GiB peak RSS during snapshot export.

The new representation keeps the immutable snapshot bytes as the base, materializes only accessed or dirty state, and appends new history to an overlay until an API genuinely needs arbitrary historical traversal.

Impact

On the supplied 11,387,982-byte test document:

  • snapshot import: about 0.90 s
  • local commit: about 1.9 ms
  • full update export: about 6.6 ms
  • snapshot export: about 57 ms
  • process baseline after loading input: 70.92 MiB RSS
  • process peak after snapshot export: 160.70 MiB RSS
  • incremental peak: 89.78 MiB
  • peak used JS heap: 8.58 MiB

The ordinary fully materialized snapshot path did not regress. After three warmups, two 15-sample B4 A/B runs measured 110.5/107.0 ms medians at the parent revision and 107.7/107.7 ms on this branch. Both emitted identical 309,780-byte snapshots.

Validation

  • pnpm --dir loro-js check
    • format check
    • lint
    • typecheck
    • 229 tests across 16 files
    • package build
  • git diff --check
  • real-document snapshot/update version-vector and operation-count validation
  • real-document snapshot state comparison across all 115,147 entries
  • local and remote edit snapshot round trips

@zxch3n
zxch3n marked this pull request as ready for review July 21, 2026 02:39
@zxch3n
zxch3n merged commit 4e663a1 into main Jul 21, 2026
2 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

WASM Size Report

  • Original size: 3053.22 KB
  • Gzipped size: 1009.02 KB
  • Brotli size: 707.57 KB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant