Flaky cache integration tests: BuildKit GC can prune persistent cache mounts between solves

[cpuguy83]

Summary

TestNoble/artifact_build_cache_dir (test/cache_test.go) failed rarely in CI (run 28251196280, job 83702856304). The populateCache solve wrote echo noble 0..5 into all six caches, but the later checkCacheContents solve failed reading cache dir #2:

grep: /tmp/cache/<randKey>3/hello: No such file or directory

Cache #2 is the only bare-key (NoAutoNamespace: true) CacheDir — its buildkit cache-mount id is the raw randKey.

Root cause (deterministically reproduced)

BuildKit's GC pruned the idle persistent cache-mount snapshot between the populate and check solves; the cache-mount reuse path then silently substituted a fresh empty mount, so grep found nothing. Persistent cache mounts are best-effort, not durable, and the test assumes data written in one solve survives into a later solve.

Reproduced locally on the same image (moby/buildkit:v0.31.0):

• Wrote a file to a cache id in one build; buildctl du -v showed the exec.cachemount record.
• Ran plain buildctl prune (the same non---all path automatic GC uses) → cache mounts dropped 10 → 0.
• Re-read the cache id in a new build → empty mount, cat: exit code 1 — exact CI signature.
• Control: two RUN steps sharing the cache id within one build read the value fine — only cross-solve persistence is at GC's mercy.

Why it's prunable (buildkit v0.31.0):

• exec.cachemount is not exempt from non---all GC — cache/manager.go:1159-1163 exempts only Internal/Frontend/shared.
• DefaultGCPolicy (cmd/buildkitd/config/gcpolicy.go:71) has a rule explicitly filtering type==exec.cachemount, plus a rule with no keep-duration that evicts any idle record under the space cap.
• A throttled GC is scheduled ~1s after every solve (control/control.go), capped to once/min.
• CachePolicyRetain (cache/refs.go:1648) only prevents eager removal on release; it does not exempt records from GC.

Why it's rare in CI: GC only prunes when the runner disk crosses the threshold and a GC tick lands in the narrow populate→check window. The determinism checker on PR #1120 inserts an extra re-solve between those two solves, widening that window — an amplifier, not the cause.

Classification

• Not a dalec product bug. dalec correctly maps CacheDir → llb.AsPersistentCacheDir. Cache mounts are optimizations; losing their contents only forces a rebuild, never a correctness issue.
• Not a buildkit bug. Persistent cache mounts are documented as best-effort and subject to GC.
• Test-robustness bug. testArtifactBuildCacheDir (and similarly the gobuild/sccache/bazel cache tests) assume cache-mount data persists across independent solves, which buildkit does not guarantee under GC.

Recommended fix

Make the integration-test buildkit instance not GC during the test run, since the cache tests legitimately depend on persistent cache mounts to validate sharing/namespacing semantics:

# buildkitd.toml
[worker.oci]
  gc = false
[worker.containerd]
  gc = false

Create the CI builder with docker buildx create ... --buildkitd-config buildkitd.toml (--config on older buildx). CI currently creates the builder with no buildkitd config (.github/workflows/ci.yml:290-292, 502-504), so default GC is active. Runner disk is per-job, so disabling GC for the test window is safe. This removes the whole class of cache-persistence test flakes.

References

• Test: test/cache_test.go testArtifactBuildCacheDir
• Mapping: cache.go (*CacheDir).ToRunOption
• BuildKit v0.31.0: solver/llbsolver/mounts/mount.go, cache/manager.go:1117, cache/refs.go:1648, control/control.go, cmd/buildkitd/config/gcpolicy.go:71
• CI: run 28251196280, job 83702856304