fix(cli): keep a live preview's ownership record and stop past a bad one - #3308
Conversation
d982ef5 to
f68c19a
Compare
0bbdf8a to
1077c7c
Compare
f68c19a to
f8020d2
Compare
A missed liveness probe is not proof the preview is gone — a server blocked on a Puppeteer capture answers nothing for a second or two — but any miss retired the session record, and the record carries the only PID-reuse guard `--stop` has. Reproduced by SIGSTOPping a managed preview and running `--status`: the record was deleted and never came back, leaving every later stop to fall through to an unauthenticated port scan with no ownership proof at all. Only a wrapper process that is provably gone now retires a record. That record gains a process-birth token so a recycled PID reads as a different process, and it is written through a temp file and renamed — every reader deletes it when it fails to parse, so a torn read would otherwise destroy a live server's proof of ownership. Two failure-propagation bugs in the stop path: `--kill-all` collected the first unprovable record's exception and abandoned every server after it, so they were left running AND unreported; and a replacement refused to launch when the server it was replacing had already exited on its own, which is the goal state rather than a failure. `--list` now shows managed sessions ahead of whatever else answers the scan.
f8020d2 to
9c7fd17
Compare
terencecho
left a comment
There was a problem hiding this comment.
LGTM.
Verified at 9c7fd170:
- Missed-probe fix:
wrapperProcessIsAlivekeeps the record when a scan misses but the wrapper's birth token still matches (previewLifecycle.ts:281-297). Testkeeps a live preview's record when a single probe misses itpins it. - PID-reuse guard:
processIdentityreturns a birth token (linux tick 22, maclstart, Windows CIMCreationDateFileTime).ownedStopTargetPidrequiresidentity(saved.pid) === saved.wrapperIdentitybefore treating saved.pid as a kill target and, when the live server pid ≠ saved.pid, additionally requiresisProcessDescendant(bounded depth 64, visited-set, fail-closed on null parent). Tests pin both the descendant-happy path and the recycled-PID divergence. - Fail-closed on server.pid=null:
stopBackgroundPreviewnow throwspreview ownership could not be proveninstead of the priorsaved.pidtrust-degradation fallthrough. Testrefuses to stop when the live server cannot prove its own PIDpins it. - Atomic session write: temp-file +
renameSyncinwritePreviewSession— readers only see the pre-existing.jsonwhile the.pid.tmpis being filled, so a torn read can't destroy live ownership proof.listBackgroundPreviewStatusesfilters.jsononly, ignoring temp orphans. Testnever leaves a partial session record for a concurrent readerpins it. --kill-allper-record propagation:handlePreviewKillAllcollectsfailures[]per session and continues, then sweeps viakillActiveServers. Testkeeps stopping after a record whose ownership cannot be provenpins it.- Replacement-when-owned-exited:
stopOwnedPreviewBeforeReplacementtreats "nothing left to stop" as goal state, not fatal. Testlaunches the replacement when the owned server died on its ownpins it. --listdedup: managed sessions prefix the scan, and the scan is filtered by${resolve(projectDir)}\0${port}. Testprefers the managed record over the same server's own self-reportpins it.- Dead helpers retired:
processIdentity+isProcessDescendantnow have production callers. - CI: all required green. Only marketplace
WIPremains IN_PROGRESS (not required).
Prior comment on #3307 (COMMENT 4956803130) is closed by this PR: server.pid → saved.pid trust-degradation loop → replaced by fail-closed throw + birth-token gate. The Windows non-English netstat fail-open I flagged (portUtils.ts:223) is not directly patched here, but its harmful consequence in --kill-all is now fenced by pidSource !== "os" skip-and-warn in killActiveServers, and its consequence in --stop is fenced by the wrapperIdentity guard.
jrusso1020
left a comment
There was a problem hiding this comment.
Reviewed at 9c7fd170. All 8 required contexts are green, including both windows jobs — UNSTABLE is failing optional checks only, so CI is not the constraint here.
Audited end-to-end: previewLifecycle.ts (+249/-28) and its test file (+414/-16), preview.ts (+99/-29), preview.test.ts, plus utils/orphanCleanup.ts at this head (not in the diff, but it supplies the two helpers this PR makes load-bearing). Not executed: no suites run locally.
Strengths
- The two helpers that were dead at #3307's head are now genuinely load-bearing —
processIdentityandisProcessDescendantfeedreadyPreviewSession,wrapperProcessIsAliveandownedStopTargetPid. That was tai's finding on #3307; this closes it rather than deleting the helpers, which was the better of the two options since the birth token is what makes PID reuse detectable at all. - The atomic write earns its comment.
writePreviewSessiongoes through${path}.${pid}.tmp+renameSync(:82-84), and the comment states the actual hazard: every reader deletes this record when it fails to parse, so a concurrent reader catching a half-written file would destroy a live server's only ownership proof. Same-directory rename, so it is genuinely atomic, andreaddirSyncfilters on.jsonso the temp file is invisible to the lister. stopOwnedPreviewBeforeReplacementtreats "nothing left to stop" as the goal state, not a failure, and says why — the previous behaviour refused to start any preview until the user hand-deleted the record. That is the right direction for a cleanup path.- Test-to-source ratio is 486:348, and the added cases name real states rather than mechanics: "keeps a live preview's record when a single probe misses it", "retires the record once the wrapper PID has been recycled", "never leaves a partial session record for a concurrent reader".
blocker — a failed identity lookup retires a live preview's record, which is the one thing this PR exists to prevent
The invariant is stated on the line immediately above the call (:274-275):
wrapperIdentity… is the only PID-reuse guard--stophas, and it never comes back. Only a wrapper process that is provably gone retires a record.
But the check cannot distinguish gone from unanswered (:290-296):
if (!saved.wrapperIdentity) return false;
const identity = dependencies.identity ?? processIdentity;
return identity(saved.pid) === saved.wrapperIdentity;processIdentity returns null on failure, not only on absence — and on two of three platforms the failure is a subprocess timeout on a live process: win32 shells out to powershell.exe for a Win32_Process CIM query with timeout: 2000, and the non-Linux POSIX branch runs ps -o lstart= with the same 2s budget. Either throws on a slow spawn and is caught into return null. Only Linux reads /proc/<pid>/stat directly and is reliable. A null then compares unequal to a real token, wrapperProcessIsAlive returns false, and :277 deletes the record.
Failure scenario: a Windows or macOS preview is alive but momentarily blocked (the PR's own comment cites a Puppeteer thumbnail capture), so the HTTP probe misses; PowerShell is slow to start under that same load and the identity call times out; the record is deleted. wrapperIdentity is now gone permanently, so the next --stop has no PID-reuse guard — the exact loss the comment says must not happen, reached through a different door than the one this PR closed.
Three things make me call it blocker rather than important:
- It hits the PR's stated purpose. The title is keep a live preview's ownership record; on 2 of 3 platforms, under the load that causes the missed probe in the first place, it still does not.
- It is untested in the direction that matters. The added cases pin
identityreturning a matching token (() => "posix:birth"→ kept) and a different one (() => "posix:someone-else"→ retired). Nothing makes it returnnullin the status path, so the suite proves the comparison discriminates same-from-different and never same-from-unknown — which is the distinctionwrapperProcessIsAlive's own docstring claims to make. - The same ambiguity is already handled the safe way 60 lines down, which shows the two cases were not meant to behave alike.
ownedStopTargetPid:356also comparesidentity(saved.pid) !== saved.wrapperIdentity, and anullthere falls back toliveServerPid— it kills less than it might, and nothing is destroyed. At:296the samenulldestroys state that never regenerates.
To be fair on direction: this is an incomplete fix, not a regression. Before this PR the record was deleted on any missed probe, so every case this now keeps is a strict improvement. The fix looks like two lines — have the caller distinguish "no answer" from "different answer" (keep the record when the lookup yields null but the process is still signalable, e.g. kill(pid, 0)), or let processIdentity report failure distinguishably from absence.
note — processIdentity is synchronous, so --list pays for it serially
listBackgroundPreviewStatuses maps records through Promise.all, but processIdentity uses execFileSync. On Windows and macOS that is one blocking subprocess spawn per record with a 2s ceiling each, so preview --list with several stale records blocks the CLI for up to N × 2s — and every one of those spawns is an independent chance to hit the timeout above, so the two findings compound: the more records, the likelier at least one live record is wrongly retired. On Linux this is a cheap /proc read and does not apply.
nit — a failed rename leaves the temp file behind
writePreviewSession:82-84 has no cleanup if renameSync throws — the .tmp is orphaned in the session directory. Harmless to correctness (the .json filter hides it from the lister) but it accumulates, and a try/finally with rmSync(temporary, { force: true }) is one line.
Verdict: REQUEST CHANGES
Reasoning: The ownership model is a solid rework and strictly better than what it replaces, but the keep-alive path it adds still retires a live record when the identity lookup times out on Windows or macOS — the PR's own stated invariant, unpinned by any test. I recognise this blocks a PR that already has an approval at this head; I think a two-line distinction between "no answer" and "different answer" is worth that, and I will convert as soon as it lands.
— Rames Jusso
…ifferent one Review blocker. The keep-alive path this PR adds could still retire a LIVE record — through a different door than the one it closed. `processIdentity` catches every failure into `null`, and on two of three platforms that failure is a subprocess timeout on a live process: the win32 `Win32_Process` CIM query and the POSIX `ps -o lstart=` both run on a 2 s budget, under exactly the load that made the HTTP probe miss in the first place. A `null` compared unequal to the saved token, so the record was deleted and `wrapperIdentity` — the only PID-reuse guard `--stop` has — was gone for good. Only Linux, reading /proc directly, was reliable. No answer is now distinguished from a different answer: the PID is checked with `kill(pid, 0)` first, which asks the kernel without signalling and treats EPERM as alive. A PID nothing can signal is gone and retires the record with no subprocess at all; a signalable PID whose token cannot be read keeps it. Only a token that comes back and differs retires it. That ordering also answers the `--list` note: the identity subprocess no longer runs for the stale records that made it slow, so the N x 2 s worst case is gone along with the timeouts that fed the bug. Verified by mutation: restoring the old "no answer means gone" behaviour reds the new case. Also clean up the temp file when a rename fails, rather than orphaning it in the session directory.
|
Blocker confirmed and fixed at I verified it in source before touching anything, because the claim is subtle: The distinction is now explicit, and I took your
Two things fell out of putting the cheap check first, which I hadn't planned:
On your point 2 — that the suite proved same-vs-different but never same-vs-unknown — both directions are pinned now: identity The temp-file nit is fixed too — Thanks for holding this one despite the approval at that head. The two-line distinction was the right call, and the reordering it forced made |
jrusso1020
left a comment
There was a problem hiding this comment.
Re-reviewed at cc93fee0. Blocker resolved. All 8 required contexts green at this head, including both windows jobs.
wrapperProcessIsAlive now separates the three states instead of two, and does it in the right order:
if (!signalable(saved.pid)) return false; // gone — no token needed to say so
const identity = (dependencies.identity ?? processIdentity)(saved.pid);
if (identity === null) return true; // no answer ≠ different answer
return identity === saved.wrapperIdentity; // recycled PIDPutting processIsSignalable (kill(pid, 0)) first is better than what I asked for. It makes the cheap, decisive check authoritative for the case that actually matters, and it keeps the subprocess off the path entirely for stale records — which also retires the --list cost note I raised, since the records that made it slow are exactly the ones that now short-circuit before any spawn.
Test coverage closes the gap that made this a blocker rather than a nit. All three states are pinned now, not two: identity: () => null, isSignalable: () => true → "keeps a live preview's record when the identity lookup gives no answer"; isSignalable: () => false → "retires the record when the PID cannot be signalled at all"; and the recycled-token case still retires. A null return can no longer be mistaken for a mismatch without a test going red.
note — the underlying 2s budget is real, and main is currently proving it
Not this PR's to fix, but worth having on the record while it is fresh. main is red on Tests on windows-latest right now, and the failing assertion is orphanCleanup.test.ts:22, the test #3307 added:
const first = processIdentity(process.pid);
expect(first).toMatch(/^(?:linux|posix|windows):/);
It fails with .toMatch() expects to receive a string, but got object — which is null reaching the matcher, since typeof null === "object". So processIdentity returned null for the current, provably alive process, and the case took 2181 ms against that function's 2000 ms PowerShell budget. Main's windows job was green at 3e4b08cd (pre-#3307) and red at b31dde35 (post-#3307), and this PR's own run passed on the same code — so it is a flaky timeout, not a deterministic break.
That is the premise of the finding this PR just fixed, reproducing in CI rather than in argument: the win32 identity lookup does time out on live processes under load. This PR now handles that safely, which is the important half. The remaining half is that the lookup itself is unreliable at 2 s and it is currently breaking a required context on main — raising the budget, or making that test tolerate a null, looks like the next thing rather than anything here.
Verdict: APPROVE
Reasoning: The blocker is fixed in the shape that matters and better than proposed — liveness first, null treated as unknown rather than recycled — with all three states pinned by tests, and every required context is green.
— Rames Jusso
`captures a stable birth token for the current process` made two assertions that a lookup allowed to fail cannot support. `processIdentity` returns null whenever the lookup cannot be completed — not only when the process is absent — and on Windows and macOS it shells out to PowerShell or `ps` on a 2 s budget that a cold CI runner routinely outruns. Both failed on windows-latest, in sequence: first `.toMatch()` received null, and once that was guarded, `expect(second).toBe(first)` compared a null from the cold first spawn against a token from the warm second one. Two lookups can disagree for exactly one reason — one of them failed — so stability is only assertable across two successful ones. The token itself cannot change between calls; it is a birth timestamp and the process did not restart. `processIdentity(-1)` stays unconditional: the guard rejects it before any subprocess runs. The strict shape assertion moves to a Linux-only case, where /proc is read directly with no subprocess and null is genuinely not allowed — keeping the guarantee on the one platform that can honour it rather than dropping it everywhere. Callers already depend on this contract: `wrapperProcessIsAlive` treats null as "no answer" rather than "gone" precisely because it is reachable.
82b2fdd to
ce61db7
Compare
…x bridge (#3349) Authoring a clip above unity gain throws at runtime today. ## What breaks `MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element: ```ts el.volume = clipVolume * volume; // init.ts, onSetVolume ``` `HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees: ``` el.volume = 2 → IndexSizeError: Failed to set the 'volume' property... ``` The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it. ## The fix Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too: - `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold. - `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`. Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp. ## Tests - `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands. - `media.test.ts` — the transport receives the authored gain while the element stays legal. - `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity. All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others. ## Provenance This is the last unlanded piece of #3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (#3308, #3309, #3333, #3339) or duplicated by the open #3306 and #3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant.
…x bridge (heygen-com#3349) Authoring a clip above unity gain throws at runtime today. ## What breaks `MAX_AUDIO_GAIN_DB = 12` makes `data-volume` legal up to ~3.98. The sandbox runtime's volume bridge assigns the product straight to the element: ```ts el.volume = clipVolume * volume; // init.ts, onSetVolume ``` `HTMLMediaElement.volume` is spec-pinned to [0,1] and **throws `IndexSizeError`** outside it — verified in Chrome, and the test DOM agrees: ``` el.volume = 2 → IndexSizeError: Failed to set the 'volume' property... ``` The throw lands inside a `for` loop over every media element, so it takes the rest of the loop with it: every clip after the boosted one keeps whatever volume it already had, while `state.bridgeVolume` says the change was applied. A composition with one boosted clip stops responding to the volume control for every clip authored after it. ## The fix Clamp what the element receives. That is not lossy, because the element was never where the boost lived — the transport gets the authored gain unclamped, and this PR pins that half too: - `syncRuntimeMedia` hands `onElementVolume` both the element's clamped volume **and** the authored gain, so the transport can have the boost the element cannot hold. - `setElementVolume` keeps that gain on the per-element node, clamped only to `MAX_AUDIO_GAIN`. Those two paths already worked; they were untested, and they are the reason clamping the element is the right half to clamp. ## Tests - `init.test.ts` — a boosted clip followed by a quieter one, both seeded with sentinels, then the real `set-volume` control message. Asserts the boosted element lands at 1 **and** that the clip after it still gets its own volume, which is what a throw mid-loop strands. - `media.test.ts` — the transport receives the authored gain while the element stays legal. - `webAudioTransport.test.ts` — the per-element gain node keeps a boost above unity. All three mutation-checked: removing the clamp reds the first, and clamping the gain at either transport seam reds the others. ## Provenance This is the last unlanded piece of heygen-com#3280. That PR was rebased onto current `main` and collapsed from +3050 to +944, of which everything except these lines is either already merged (heygen-com#3308, heygen-com#3309, heygen-com#3333, heygen-com#3339) or duplicated by the open heygen-com#3306 and heygen-com#3310. Cutting it out separately because the throw is live on `main` now and shouldn't wait behind a PR that is otherwise redundant. (cherry picked from commit 9140c0e)
A missed liveness probe is not proof the preview is gone — a server blocked on a Puppeteer capture answers nothing for a second or two — but any miss retired the session record, and that record carries the only PID-reuse guard
--stophas.Reproduced
SIGSTOP a managed preview, then run
preview --status. Before this change the record is deleted and never comes back, leaving every later stop to fall through to an unauthenticated port scan with no ownership proof at all. After it, the record survives and the preview is reachable again once resumed.Only a wrapper process that is provably gone now retires a record. That record gains a process-birth token so a recycled PID reads as a different process, and it is written through a temp file and renamed — every reader deletes it when it fails to parse, so a torn read would otherwise destroy a live server's proof of ownership.
Two failure-propagation bugs in the stop path
--kill-allcollected the first unprovable record's exception and abandoned every server after it, so they were left running and unreported. Per-record failures are now collected and reported.--listnow shows managed sessions ahead of whatever else answers the scan, so the same server is not listed twice from its own self-report.Stack
Based on
u4a-process-kill-trust, whose OS-verified listener PID this relies on. Retarget tomainbefore merging.