Skip to content

Split CI into named phases so a hang identifies itself - #28

Merged
thefinder808 merged 1 commit into
mainfrom
fix/ci-phase-visibility
Aug 4, 2026
Merged

thefinder808 merged 1 commit into
mainfrom
fix/ci-phase-visibility

Conversation

@thefinder808

Copy link
Copy Markdown
Owner

What happened

CI wedged again this morning on #27 (run 30877370518) — the Test step ran past 15 minutes and the job timeout cancelled it.

Rerunning the identical commit passed in 21 seconds:

Attempt Tree Test step
First 654f0f6 wedged >15 min, cancelled
Rerun 654f0f6 21 s, green

So the tree isn't the cause. This is the same intermittent hang that ran through July.

Why it keeps going undiagnosed

Each occurrence destroys its own evidence:

  • dotnet test folds restore → build → run into one step, so the wedged phase is unknowable from outside.
  • --blame-hang only watches test execution. A wedge in restore or build writes no dump and names no test — which matches exactly what we saw: 8+ minutes elapsed, past the hang threshold, no dump artifact produced.
  • The job-level timeout cancels the run, and GitHub retains no logs for a cancelled run. Both gh run view --log and the artifacts endpoint came back empty.

That's three independent reasons a hang leaves nothing behind, which is why July's hangs were never root-caused either.

What this changes

Splits restore/build/test into separate steps with individual budgets, sized so a step timeout fires before the job timeout:

Restore  5m
Build    5m
Test     8m   (blame-hang 4m, so the dump lands inside the budget)
         --
         18m  <  job 20m

The ordering is the point: a step timeout fails that step and lets the job finish, so logs are retained. A job timeout cancels the run and the logs are gone. Next hang will name its phase in the UI even if nothing else survives.

Normal total is ~40 s, so the budgets are loose enough that only a real wedge trips them.

What this does not do

It does not fix the hang. No root cause is proven, and I'd rather say so than ship a guess. What I could rule out:

  • Every poll loop in the UI tests (MainWindowFlowTests, SigningRefreshTests, RenderHarness) is time-bounded — fixed Pump durations and a 15 s packaging deadline. None can spin forever.
  • Both process calls (ProcessRunner.cs:44, UpdateService.cs:471) use WaitForExitAsync(ct) with a cancellation token.

That leaves restore/build — the phase nothing currently observes. Hence instrumenting rather than patching.

Deliberately not caching ~/.nuget/packages in this PR: restore is the leading suspect, and a warm cache would mask the very phase we're trying to observe. Worth adding once a hang has been caught in a named step.

Verified

  • dotnet restore → build --no-restore → test --no-build chain runs clean locally; 136/136 tests pass.
  • YAML parses; step budgets sum to 18 ≤ job 20, so steps fail before the job is cancelled.
  • No ${{ }} interpolation added (and grep -rn '\${{ *}}' over .github/workflows is clean — an empty expression anywhere, even a comment, fails the whole file).

🤖 Generated with Claude Code

CI wedged again on 2026-08-04 (run 30877370518): the Test step ran past
15 minutes and the job timeout cancelled it. A rerun of the identical
commit passed in 21 seconds, so the tree is not the cause -- this is the
same intermittent hang seen through July.

It has gone undiagnosed because each occurrence destroys its own
evidence:

  - `dotnet test` folds restore, build, and run into one step, so the
    wedged phase is unknowable from the outside.
  - --blame-hang only watches test execution. A wedge in restore or
    build writes no dump and names no test, which matches what we saw:
    8+ minutes elapsed, past the hang threshold, no dump artifact.
  - The job-level timeout cancels the run, and GitHub retains no logs
    for a cancelled run. Both this run's logs and the artifact list came
    back empty.

Split restore/build/test into separate steps with individual budgets
sized to fire before the job timeout (5+5+8 = 18 < 20), so a wedge fails
as a *step* -- the job then finishes and its logs survive -- and the
guilty phase is named in the UI even if nothing else does. blame-hang
moves to 4m so its dump lands inside the test step's 8m budget instead
of being lost to the step timeout.

This does not fix the hang; it is the instrumentation needed to find it,
since the test-side suspects are already ruled out (every poll loop in
the UI tests is time-bounded, and both process calls use
WaitForExitAsync with a cancellation token).

Deliberately not caching ~/.nuget/packages: restore is the leading
suspect and a warm cache would mask the phase we need to observe.

Verified locally -- the --no-restore/--no-build chain runs clean and all
136 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thefinder808
thefinder808 merged commit 1e82225 into main Aug 4, 2026
1 check passed
@thefinder808
thefinder808 deleted the fix/ci-phase-visibility branch August 4, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant