Skip to content

fix(gitextractor): disable git auto gc/maintenance while cloning - #9218

Open
DoDiODev wants to merge 1 commit into
apache:mainfrom
DoDiODev:pr/gitextractor-no-auto-gc
Open

DoDiODev wants to merge 1 commit into
apache:mainfrom
DoDiODev:pr/gitextractor-no-auto-gc

Conversation

@DoDiODev

Copy link
Copy Markdown
Contributor

Summary

Incremental git collection fails sporadically in Collect Commits with:

could not find '/tmp/gitextractorXXXX/objects/pack/.tmp-26800-pack-<sha>.idx' to open: No such file or directory

Root cause: since git 2.29, git fetch ends with git maintenance run --auto
(older versions: git gc --auto), and that runs detached in the background
by default. For an incremental collection GitcliCloner runs several fetches
(shallow fetch, --shallow-since, --deepen rounds). The last one may leave a
background maintenance or gc process repacking the repository while libgit2 is
already scanning objects/pack/. libgit2 sees the temporary
.tmp-<pid>-pack-*.idx file, which git renames or removes a moment later, and
the subtask aborts. Because this is a race, the failure is intermittent.

This PR passes
-c gc.auto=0 -c gc.autoDetach=false -c maintenance.auto=false to every git
command run by GitcliCloner. The clone is a short-lived temporary directory
that is removed after extraction, so automatic housekeeping has no benefit
there. The explicit git repack -d in deepen() is still executed
synchronously as before.

A regression test runs an incremental clone with GIT_TRACE and asserts that
git does not start maintenance run / gc --auto. Without the fix the test
fails with
run_command: git maintenance run --auto --no-quiet --detach after each fetch.

Does this close any open issues?

N/A

Screenshots

N/A

Other Information

  • Unknown config keys are ignored by git, so maintenance.auto is safe on git
    versions older than 2.29; gc.auto=0 covers those versions.
  • Checked on a local instance (git 2.56.0): before the change an incremental
    run failed in Collect Commits with the error above; with the change, 47
    gitextractor tasks across two further pipeline runs completed without it.
  • Full CI mirror on my fork, including unit tests (the new test passes there),
    lint and e2e on MySQL and PostgreSQL:
    run

git fetch ends with `git maintenance run --auto` (older git: `git gc
--auto`), which runs detached by default. During an incremental
collection the last fetch can leave a background repack running while
libgit2 already scans objects/pack/, so a temporary
.tmp-<pid>-pack-*.idx disappears under it and Collect Commits fails
intermittently with "could not find ... to open".

Pass -c gc.auto=0 -c gc.autoDetach=false -c maintenance.auto=false to
every git command run by GitcliCloner. The clone is a temporary
directory, so automatic housekeeping has no benefit there; the
explicit `git repack -d` in deepen() still runs synchronously.

Add a regression test that traces an incremental clone with GIT_TRACE
and asserts that git starts neither `maintenance run` nor `gc --auto`.

Signed-off-by: DoDiODev <DoDiDev@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant