Skip to content

executor: filter command output lines instead of buffering all of them - #365

Merged
umputun merged 3 commits into
umputun:masterfrom
paskal:feat/keepline-output-filter
Aug 24, 2026
Merged

executor: filter command output lines instead of buffering all of them#365
umputun merged 3 commits into
umputun:masterfrom
paskal:feat/keepline-output-filter

Conversation

@paskal

@paskal paskal commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Previously, Local.Run and Remote.sshRun streamed stdout to the log and at the same time retained every byte in a bytes.Buffer, then handed the whole buffer to splitOutputLines, which copied it again. A command printing a large log therefore cost roughly two copies of its output, including at the ten call sites that discard the result.

After this change, RunOpts carries KeepLine func(line string) bool, a predicate deciding which lines Run returns; a nil predicate keeps every line. Every line still reaches the log in full, so this bounds only what the caller retains. Script passes the same setvar test its parse loop uses, so the two cannot drift; Echo keeps the non-empty lines it reports; the sites that ignore the result share a discardOutput value.

The capture is now a streaming writer in place of the buffer, so output is never fully retained. It reproduces the previous splitOutputLines(buffer.String()) exactly, which the tests pin across five write chunk sizes so that a line split across writes, a \r\n pair split down the middle and a run of newlines all give the same answer as the batch split. A line of any length still arrives whole, and an unterminated final line is returned like a terminated one.

Measured with the reproduction from #363, three runs each:

peak RSS
before 215.9, 216.0, 217.3 MB
after 27.8, 28.0, 27.2 MB

The issue's control run, with the same output sent to /dev/null, sat at 21.5 to 22.4 MB.

Verbose is dropped from RunOpts, dead since 6c872a3: all three executors took it as _ and verbosity comes from MakeLogs. As noted in the issue, KeepLine makes RunOpts non-comparable, which nothing in the tree depends on.

A line spanning several writes has to be staged, and the staging buffer is released once it grows past 64KiB rather than being kept for reuse. Holding it would pin the longest line for the rest of the command, which is the retention this change removes elsewhere.

The last commit is the maskSecrets point raised at the end of the issue: it compiled a regexp per secret on every call, and it is called for every line of every command. Patterns are now compiled once in MakeLogs and shared by every writer. On a typical log line with four secrets configured, masking drops from 6031 to 2190 ns/op. It sits on its own commit and can be split out if you would rather keep it separate.

Resolves #363.

paskal added 3 commits August 20, 2026 19:45
RunOpts gains KeepLine, a predicate deciding which stdout lines Run
returns; a nil predicate keeps every line. Every line still reaches the
log in full, so this bounds only what the caller retains.

Previously, Local.Run and Remote.sshRun kept every stdout byte in a
bytes.Buffer and handed the whole thing to splitOutputLines, which
copied it again, so a command printing a large log cost roughly two
copies of its output even where the result was discarded. Script now
keeps only the setvar lines it parses, Echo only the lines it reports,
and the sites that ignore the result keep nothing.

Verbose is dropped from RunOpts, dead since 6c872a3: all three
executors took it as _ and verbosity comes from MakeLogs. Note KeepLine
makes RunOpts non-comparable, which nothing depends on today.

Resolves umputun#363
Previously, maskSecrets compiled a regexp per secret on every call, and
it is called for every line of every command, so on a noisy command with
secrets configured it was the dominant cost of the output path.

Patterns are now compiled once in MakeLogs and shared by every writer.
On a typical log line with four secrets configured, masking drops from
6031 to 2190 ns/op.
A line spanning several writes is staged in a buffer that was truncated
for reuse, keeping its backing array reachable until the command ended.
One long line therefore stayed pinned for the rest of the run, which is
the retention this filtering removes elsewhere. Buffers up to 64KiB are
still reused to spare an allocation per line, larger ones are released.
@paskal
paskal requested a review from umputun as a code owner August 20, 2026 19:24
@umputun
umputun merged commit bd2fd80 into umputun:master Aug 24, 2026
1 check passed
@paskal
paskal deleted the feat/keepline-output-filter branch August 24, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

executor: whole command output is buffered in memory for every command

2 participants