Skip to content

fix(ci-serialisation): report every outcome, and quieten the baseline builds - #10

Merged
sakanni merged 2 commits into
developfrom
fix/serialisation-summary-coverage
Aug 25, 2026
Merged

fix(ci-serialisation): report every outcome, and quieten the baseline builds#10
sakanni merged 2 commits into
developfrom
fix/serialisation-summary-coverage

Conversation

@sakanni

@sakanni sakanni commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Problem

Three defects in ci-serialisation, none of which produces a failure anywhere. Each is a
property of the action as a whole rather than of any one step, which is why none of them was
visible from reading a single place in the file.

A red check can have an empty job summary. The summary was written by five
mutually-exclusive steps, each gated on a different combination of step conclusions and
outputs. Those conditions had to enumerate the states worth reporting, and between them they
named five of the action's eighteen functional steps:

covered:   Compute changed files, Resolve dependencies (branch),
           Build primary repo (branch), Run serialisation tests (branch), Compare
uncovered: the other thirteen

A failure in any of the thirteen matched no condition and wrote nothing, so the check went red
and said nothing at all. Seven of them run before serialisation is attempted — locating and
building the verification solution, inferring its configuration, publishing the runner — which
is exactly the case a reader is least equipped to diagnose, because the natural reading of a red
serialisation check is that serialisation found something.

Two of the four builds annotate the check with the wrong warnings. setup-dotnet registers
the csc problem matcher, which turns every MSBuild diagnostic line into a check-run
annotation. The branch-leg builds pass -clp:ErrorsOnly to suppress that; the two baseline
builds did not. Those two compile the base branch, so their warnings belong to code the
author did not write and cannot act on, and they consume a per-step annotation cap that the
caller's own diagnostics need.

The assemblies directory is reset twice. The base-checkout step removed the contents of
ProgramData\BHoM\Assemblies. The very next step, resolve-dependencies, removes and recreates
that directory and ProgramData\BHoM\Upgrades in its own Prepare folders step, which runs
unconditionally. The earlier line was strictly weaker and strictly redundant — and it is the
line a reader looking for where the two legs are separated finds first.

What changed

One summary step instead of five, always-gated, with the decision extracted to
.github/scripts/Write-SerialisationSummary.ps1 so it can be tested without running a
serialisation check. The five existing messages are unchanged. What is new is the last branch:
a catch-all that names the phase the check stopped in and states that a stop is a CI failure
rather than a finding about the pull request.

The catch-all names the phase, not the step, deliberately. Naming the step would mean giving
all eighteen an id and threading each conclusion into the condition — which is the enumeration
that failed here in the first place. The phase is derived from the signals already available, so
adding a step cannot reopen the hole.

-clp:ErrorsOnly on the two baseline builds. Console output only: no effect on build
behaviour, exit codes, or how errors are reported. Errors still print and still annotate.

The redundant reset removed, with a comment recording where the reset actually happens so
the next reader does not go looking for it here.

Verification

19 new tests, and the three that matter failed before the fix. The first commit adds them
alone; on that commit all three fail, each naming the offending lines rather than describing
them:

[-] writes the summary from a single step that always runs
    Expected 1, but got 5.
[-] passes -clp:ErrorsOnly on every build it runs
    dotnet build "...solution..." --no-restore -c Release --nologo -m
    dotnet build "...ver_sln..." --no-restore -c ... --nologo
[-] leaves the assemblies reset to resolve-dependencies
    Remove-Item "C:\ProgramData\BHoM\Assemblies\*" -Recurse -Force ...

The second commit makes them pass. Sixteen further tests cover the summary decision directly,
including the states that previously produced nothing — a failure before the branch leg ran, a
failure in the baseline leg, and the case where nothing can be identified at all.

Full Pester suite green in CI. The structural tests are text assertions rather than a YAML
parse: the PowerShell test job installs Pester and nothing else, and all three facts are
line-shaped, so a YAML dependency would cost more than it returns.

What is not verified

No end-to-end run exercises the paths this changes. That is a property of the changes, not
a gap in the testing:

  • The summary change only alters what is written on paths where the check fails. A passing
    run produces the same summary as before.
  • -clp:ErrorsOnly changes console verbosity. Nothing observable in a green run changes.
  • The removed reset is duplicated by the following step, so removing it is inert by
    construction — nothing runs between the two lines.

Reaching the baseline leg for real needs a repository that genuinely fails serialisation, which
is not something the available test repositories provide. It would not prove anything additional
here, and I would rather say so than imply coverage this does not have.

What to look at

  • The catch-all branch in Write-SerialisationSummary.ps1. It is the point of the change, and
    it is the branch that will be wrong first if the action gains a phase.
  • ci-serialisation-action.Tests.ps1 asserts one always-gated summary step. If a second is
    ever added, that test fails — which is intended, and is the guard against the old shape
    returning.
  • The two baseline builds, to confirm the flag is the only difference from their branch-leg
    counterparts.

Not in scope

ci-versioning has the same build-flag asymmetry: its caller build passes -clp:ErrorsOnly
and its verification-solution build does not. Left alone here to keep this to one action.

This does not change which checks are required anywhere.

Three properties of ci-serialisation's action.yml that no test asserted, each
broken in a way that produces no failure anywhere:

  The job summary is written by five mutually-exclusive steps whose conditions
  between them name five of the eighteen functional steps. A failure in any of
  the other thirteen matches no condition and writes nothing, so the check goes
  red with an empty summary. Seven of the thirteen run before serialisation is
  attempted.

  Two of the four builds pass -clp:ErrorsOnly and two do not. The two without it
  compile the base branch, so their warnings annotate the check with diagnostics
  from code the author did not write.

  The action resets the assemblies directory itself, immediately before the step
  that resets it properly.

These tests fail on this commit. That is the point: each names the offending
lines rather than describing them, so the fix that follows is verifiable rather
than asserted.

Text assertions rather than a YAML parse, because the powershell-tests job
installs Pester and nothing else and all three facts are line-shaped.
… builds

Three changes to one action, each closing a defect the preceding commit captured.

The job summary was written by five mutually-exclusive steps. Their conditions had
to enumerate the states worth reporting, and between them they named five of the
action's eighteen functional steps, so a failure anywhere else matched nothing and
the check went red with an empty summary. Seven of the unnamed steps run before
serialisation is attempted, which is the case a reader is least able to diagnose.

Replaced by one always-gated step whose decision lives in a script, so the branch
that handles "none of the above" is a branch in code rather than a condition
nobody wrote. That branch names the phase the check stopped in, derived from the
signals already available, and says a stop is a CI failure rather than a finding
about the pull request. Naming the step instead would mean giving every step an id
and threading each conclusion through, which is the enumeration that failed here.

The five existing messages are unchanged, and coverage figures are still emitted
only when the branch leg measured them, so a summary never claims coverage a run
did not produce.

The two baseline builds now pass -clp:ErrorsOnly, as the branch-leg builds already
did. setup-dotnet registers the csc problem matcher, which turns each MSBuild
diagnostic into an annotation; these two compile the base branch, so their warnings
belong to code the author did not write and consume the per-step annotation cap the
caller's own diagnostics need. Console output only: no effect on build behaviour,
exit codes, or how errors are reported.

The assemblies reset in the base-checkout step is removed. resolve-dependencies
removes and recreates that directory, and Upgrades alongside it, in its own Prepare
folders step, which is the next step and runs unconditionally. The removed line
covered one of the two directories and was redundant with the step after it, while
reading as the mechanism that separates the two legs.

No change to which checks are required anywhere.
@sakanni
sakanni merged commit 83fa71e into develop Aug 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant