Skip to content

test(planner): avoid materializing zero-filled shard payloads - #1439

Open
bherald wants to merge 1 commit into
JustVugg:devfrom
bherald:test/resource-plan-sparse-fixtures-20260911
Open

test(planner): avoid materializing zero-filled shard payloads#1439
bherald wants to merge 1 commit into
JustVugg:devfrom
bherald:test/resource-plan-sparse-fixtures-20260911

Conversation

@bherald

@bherald bherald commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

The planner tests use safetensors fixtures whose tensor payloads are all zero. One disk-bound model case creates four 3 GB payloads, materializing those bytes in Python and writing them to temporary storage even though the planner only needs the headers and file sizes.

This changes the fixture writer to write the header and extend the file to its expected length. It preserves tensor offsets, logical file size and zero-valued payload bytes without allocating the payload in Python. Filesystems supporting sparse extension also avoid the large disk writes.

A fixture integrity test verifies offsets, final length and zero-valued reads. Runtime code is unchanged.

Verification: python3 -m unittest tests.test_env_defaults tests.test_resource_plan passes all 90 tests. Before the change, the same selection ran 89 tests and failed the large-model fixture with OSError: [Errno 28] No space left on device on a checkout with limited temporary storage.

@bherald
bherald marked this pull request as ready for review September 11, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant