Skip to content

[#2079] Backfill commits from checkouts #2091

Description

@alanpeixinho

Parent

#2079
Schema: #2089
Fetch library: #2090

Management command that fills commits (and parent storage from #2089) using the fetch library from #2090. Incremental: only hashes we already see in checkouts and do not yet have metadata for.

In scope

  • Select distinct checkouts.git_commit_hash missing from commits (or stub rows still missing author/metadata).
  • Call the fetch library; upsert commits and parent links per the schema chosen in #2089 (stubs if using commit_parents FKs).
  • Optionally set checkouts.git_commit_message when it is null and git returned a message.
  • Skip and log failures (bad URL, missing object, full pack). The job must not die on one hash.
  • Dry-run flag.
  • Wire ephemeral git: Compose tmpfs on the service that runs the command/cron; pass the scratch path into the library. Do not write mirrors under BACKEND_VOLUME_DIR.

Job

  1. Batch missing hashes (batching left to the implementer).
  2. Resolve URL if needed (library).
  3. Fetch + parse (library).
  4. Upsert; continue on error.

Idempotent: re-runs only work on what’s still missing.

Tests

Command upserts from mocked fetch results. Failures skip the hash and continue. Dry-run writes nothing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BackendMost or all of the changes for this issue will be in the backend code.Data QualityDatabaseIssue that alters only configs of a database itselfenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions