Skip to content

Fix colliding ccache-rolling-ci keys in CI#3776

Merged
nbbrooks merged 1 commit into
mainfrom
nbbrooks/ci-ccache-key-fix
Jun 24, 2026
Merged

Fix colliding ccache-rolling-ci keys in CI#3776
nbbrooks merged 1 commit into
mainfrom
nbbrooks/ci-ccache-key-fix

Conversation

@nbbrooks

Copy link
Copy Markdown
Collaborator

Description

Fix long standing issue where both the rolling + clang-tidy and rolling + deprecation jobs use [read: clobber] the same ccache-rolling-ci key.

This splits them into dedicated keys and should significantly speed up rolling job CI times.

Checklist

  • Required by CI: Code is auto formatted using clang-format
  • Extend the tutorials / documentation reference
  • Document API changes relevant to the user in the MIGRATION.md notes
  • Create tests, which fail without this PR reference
  • Include a screenshot if changing a GUI
  • While waiting for someone to review your request, please help review another open pull request to support the maintainers

@nbbrooks nbbrooks changed the title Fix colliding ccache-rolling-ci keys Fix colliding ccache-rolling-ci keys in CI Jun 24, 2026
The ccache and target_ws cache prefixes were nested as string prefixes:
the clang-tidy and deprecation jobs used prefix `ccache-rolling-ci`, which
GHA restore-keys prefix-matched against the ccov job's `ccache-rolling-ci-ccov`
cache. Both jobs therefore restored the Debug+coverage (and wrong-compiler)
ccov cache and missed on nearly every compile (clang-tidy hit rate ~7%,
deprecation ~60%), causing 33-34 min full rebuilds.

Give each build variant a distinct, non-nested token (ccov/tidy/default) in
both the ccache and target_ws cache prefixes so each job restores only its
own cache.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nbbrooks nbbrooks force-pushed the nbbrooks/ci-ccache-key-fix branch from eaead28 to bcdf00f Compare June 24, 2026 03:10
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.29%. Comparing base (d98fc1f) to head (bcdf00f).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3776      +/-   ##
==========================================
+ Coverage   46.28%   46.29%   +0.02%     
==========================================
  Files         726      726              
  Lines       59507    59509       +2     
  Branches     7624     7623       -1     
==========================================
+ Hits        27536    27544       +8     
+ Misses      31805    31798       -7     
- Partials      166      167       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nbbrooks

Copy link
Copy Markdown
Collaborator Author

Benchmarked the cache split. Measuring colcon build-phase time only (from the Summary: N packages finished lines for upstream + target builds), since the test phase that follows is unaffected by ccache and any wall-clock difference there would be flake-driven noise.

Job Baseline (main, old shared key) 1st run on this PR (new keys, cold migration) 2nd run on this PR (new keys, fully warm)
rolling-ci + ccov 49m 13s 47m 16s 1m 45s
rolling-ci + clang-tidy (delta) 41m 24s 40m 50s 1m 35s
rolling-ci + deprecation check 35m 37s 36m 35s 1m 08s

What each column tells us:

  • 1st run vs baseline: ~equal. The key migration is free — no penalty for switching.
  • 2nd run on same SHA: each job's dedicated cache fully restores its own compile artifacts. Confirms the split keys work in isolation (no warm-up regressions).

A caveat on interpretation: this same-SHA re-run would also be fast under the old shared-key setup, but only for whichever job won the write race last time. The actual payoff of the split — that the next PR's 1st CI run can restore all three caches independently instead of seeing two clobbered ones — will only be measurable after this lands on main and a subsequent PR runs against it.

So: a no-cost change that unlocks a hit-rate improvement which becomes observable on future PRs.


(Failures on this PR are pre-existing flakes — _with_move_group family + osqp_vendor on Resolute + Sonar setup — none caused by the cache change.)

@nbbrooks nbbrooks merged commit 4a766fd into main Jun 24, 2026
17 of 24 checks passed
@nbbrooks nbbrooks deleted the nbbrooks/ci-ccache-key-fix branch June 24, 2026 05:32
@github-project-automation github-project-automation Bot moved this to ✅ Done in MoveIt Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant