Repository navigation
Split CI into named phases so a hang identifies itself - #28
Merged
Merged
Conversation
CI wedged again on 2026-08-04 (run 30877370518): the Test step ran past
15 minutes and the job timeout cancelled it. A rerun of the identical
commit passed in 21 seconds, so the tree is not the cause -- this is the
same intermittent hang seen through July.
It has gone undiagnosed because each occurrence destroys its own
evidence:
- `dotnet test` folds restore, build, and run into one step, so the
wedged phase is unknowable from the outside.
- --blame-hang only watches test execution. A wedge in restore or
build writes no dump and names no test, which matches what we saw:
8+ minutes elapsed, past the hang threshold, no dump artifact.
- The job-level timeout cancels the run, and GitHub retains no logs
for a cancelled run. Both this run's logs and the artifact list came
back empty.
Split restore/build/test into separate steps with individual budgets
sized to fire before the job timeout (5+5+8 = 18 < 20), so a wedge fails
as a *step* -- the job then finishes and its logs survive -- and the
guilty phase is named in the UI even if nothing else does. blame-hang
moves to 4m so its dump lands inside the test step's 8m budget instead
of being lost to the step timeout.
This does not fix the hang; it is the instrumentation needed to find it,
since the test-side suspects are already ruled out (every poll loop in
the UI tests is time-bounded, and both process calls use
WaitForExitAsync with a cancellation token).
Deliberately not caching ~/.nuget/packages: restore is the leading
suspect and a warm cache would mask the phase we need to observe.
Verified locally -- the --no-restore/--no-build chain runs clean and all
136 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
CI wedged again this morning on #27 (run 30877370518) — the Test step ran past 15 minutes and the job timeout cancelled it.
Rerunning the identical commit passed in 21 seconds:
654f0f6654f0f6So the tree isn't the cause. This is the same intermittent hang that ran through July.
Why it keeps going undiagnosed
Each occurrence destroys its own evidence:
dotnet testfolds restore → build → run into one step, so the wedged phase is unknowable from outside.--blame-hangonly watches test execution. A wedge in restore or build writes no dump and names no test — which matches exactly what we saw: 8+ minutes elapsed, past the hang threshold, no dump artifact produced.gh run view --logand the artifacts endpoint came back empty.That's three independent reasons a hang leaves nothing behind, which is why July's hangs were never root-caused either.
What this changes
Splits restore/build/test into separate steps with individual budgets, sized so a step timeout fires before the job timeout:
The ordering is the point: a step timeout fails that step and lets the job finish, so logs are retained. A job timeout cancels the run and the logs are gone. Next hang will name its phase in the UI even if nothing else survives.
Normal total is ~40 s, so the budgets are loose enough that only a real wedge trips them.
What this does not do
It does not fix the hang. No root cause is proven, and I'd rather say so than ship a guess. What I could rule out:
MainWindowFlowTests,SigningRefreshTests,RenderHarness) is time-bounded — fixedPumpdurations and a 15 s packaging deadline. None can spin forever.ProcessRunner.cs:44,UpdateService.cs:471) useWaitForExitAsync(ct)with a cancellation token.That leaves restore/build — the phase nothing currently observes. Hence instrumenting rather than patching.
Deliberately not caching
~/.nuget/packagesin this PR: restore is the leading suspect, and a warm cache would mask the very phase we're trying to observe. Worth adding once a hang has been caught in a named step.Verified
dotnet restore→build --no-restore→test --no-buildchain runs clean locally; 136/136 tests pass.${{ }}interpolation added (andgrep -rn '\${{ *}}'over.github/workflowsis clean — an empty expression anywhere, even a comment, fails the whole file).🤖 Generated with Claude Code