Skip to content

fix(fileutils): drain pipe buffers after process exits#124

Open
steps-re wants to merge 1 commit into
watts-dev:developmentfrom
steps-re:fix/run-drain-on-exit
Open

fix(fileutils): drain pipe buffers after process exits#124
steps-re wants to merge 1 commit into
watts-dev:developmentfrom
steps-re:fix/run-drain-on-exit

Conversation

@steps-re

@steps-re steps-re commented Jul 8, 2026

Copy link
Copy Markdown

Problem

fileutils.run() uses O_NONBLOCK + read_async() to avoid the classic pipe deadlock (write-end full while parent blocks reading the wrong fd). However, when the subprocess exits run() breaks out of the read loop as soon as p.poll() returns non-None — potentially before all data the process wrote has been drained from the kernel pipe buffer.

With O_NONBLOCK a single read() syscall returns only what the kernel has ready at that instant; a large burst of output written just before process exit (common in nuclear simulation and optimization codes) can silently disappear. This is the underlying cause of the hang/truncation behaviour reported in #49.

Fix

Add a post-exit drain loop for both stdout and stderr (2 + the while True loop in fileutils.py). Once the write end of the pipe is closed, read_async() returns b"" (EOF) rather than EAGAIN, so the loop terminates immediately after all buffered bytes are consumed.

Test

Added test_run_captures_all_output which emits 160 000 bytes of output — well past a typical 64 KB pipe-buffer read — and asserts every byte is captured. The test is skipif win32 (same as the rest of the non-blocking path).

tests/test_fileutils.py::test_run_captures_all_output PASSED

All 4 test_fileutils tests pass.

Fixes #49


Signed-off-by: Mike German mike@stepsventures.com

When run() breaks out of the read loop (p.poll() returns non-None),
bytes that the subprocess wrote just before exiting may still be
sitting in the kernel pipe buffer.  With O_NONBLOCK a single read()
call returns only what fits in one syscall, so a large burst of final
output (common in nuclear simulation codes) could be silently dropped.

Add a post-exit drain loop for both stdout and stderr.  Once the write
end of the pipe is closed (process exited), read_async() returns b''
(EOF) rather than raising EAGAIN, so the drain loop terminates cleanly.

Add a regression test that emits 160 000 bytes of output—well past a
typical 64 KB pipe buffer—and asserts every byte is captured.

Closes watts-dev#49

Signed-off-by: Mike German <mike@stepsventures.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Using communicate() rather than read() in fileutils.run()

1 participant