Skip to content

RFC: CLI-managed regression baselines (committed + native bundles)#5

Merged
lewisjared merged 8 commits into
mainfrom
rfc/regression-baselines
Jun 17, 2026
Merged

RFC: CLI-managed regression baselines (committed + native bundles)#5
lewisjared merged 8 commits into
mainfrom
rfc/regression-baselines

Conversation

@lewisjared

@lewisjared lewisjared commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Adds an RFC for a more structured approach to tracking regression baselines per test-case.

The output of each test case consists of two bundels:

  • Committed bundle (series.json + CMEC bundles) - tracked in git and part of the PR diff.
  • Native bundle (raw outputs) - minted via the CI and stored as publicaly accessible results.

Key decisions

  • test_case_version (int in manifest.json) — an explicit bump is the re-baseline trigger; CI hard-fails a changed bundle with no bump.
  • Trying to avoid the use of a shared-runner on PRs to allow PRs from forks. The fallback is that the nightly check will run on a shared-runner so should be able the catch any drifts for diagnostics that can't run on public runners.
  • NativeStore Protocol — public read / gated write; backend deferred (R2 to be used for a prototype).
  • Consumes execution.json from #3 for baseline provenance once it lands.

@lewisjared lewisjared changed the title RFC: CLI-managed regression baselines (golden-in-git + native fixtures) RFC: CLI-managed regression baselines (committed + native bundles) Jun 4, 2026
@lewisjared

Copy link
Copy Markdown
Contributor Author

@mikapfl @fuchsi-huber Interested in your thoughts on this too following our discussions about regression outputs.

@bouweandela The regression outputs need some more rework to be fit for purpose. The gist of the changes for a diagnostic dev is that we track a smaller set of the diagnostic output (no logs, no images) to avoid bloating the git repo. We can do this now.

The rest of the workflow can come later and is more helpful from the view of a reviewer/maintainer.

@lewisjared lewisjared merged commit 2478e97 into main Jun 17, 2026
@bouweandela

Copy link
Copy Markdown

Looks like a great plan @lewisjared! I noticed you mentioned a heavy esmvaltool ocean diagnostic that is causing some trouble. Which one is it? I may be able to make it less heavy..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants