Skip to content

fix(targethasher): collapse apparent-name bzlmod external files via repo mapping - #322

Draft
yushan8 wants to merge 4 commits into
mainfrom
yushan/bzlmod-apparent-repo-collapse
Draft

yushan8 wants to merge 4 commits into
mainfrom
yushan/bzlmod-apparent-repo-collapse

Conversation

@yushan8

@yushan8 yushan8 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Why?

bazel query prints repos visible to the main module by apparent name (@name//...), but bzlmod marker hashes are keyed by canonical name and the collapse only recognized @@canonical//... labels. Files in such repos missed both the git known-hash map and the marker collapse, so the hasher read them from disk. On the java repo that was 26k files and about 40 GB (large jars, zips and container images), making targethasher.FromProto the largest part of nativeGraphRunner.Compute.

What?

  • Add bazel.RepoMapping, which runs bazel mod dump_repo_mapping "" to get the apparent-to-canonical repo map, and HashConfig.RepoMapping to carry it.
  • Resolve @name//... labels through the mapping in HashExternalTargetsBzlmod, so they collapse to the same marker-based hash as the equivalent @@canonical//... label. A mapped repo with no marker is left alone instead of erroring; fullHashRepos matches either name.
  • nativeGraphRunner.Compute fetches the mapping for bzlmod repos and emits repo_mapping_duration. A failure fails the compute, like a marker read failure, so hashes stay deterministic.
  • query-bench mirrors Compute for Bzlmod repos (--bzlmod, on by default) and logs the duration of each phase through zap: query, git file hashes, marker read, repo mapping and hasher. It also gains --cpuprofile for the hasher phase, and the graph JSON dump is opt-in via --dump.

Compatibility: collapsed external files now hash from their marker instead of their content, so their hashes (and the graph hashes above them) change. Cached graphs computed before this change will report those external targets as changed once. Cache keys do not encode an algorithm version, so existing entries need to be invalidated or the one-time diff accepted.

Issue

None.

🤖 Generated with Claude Code

Test Plan

Measured with example/cmd/query-bench on a 1.3M-target bzlmod monorepo (about 640k files, 18.8k repos) with a warm Bazel server. The target count is unchanged at 1,321,910.

  • targethasher.FromProto: 65-68s before, 10.6-12.7s after.
  • Total Compute: about 2m05s before, about 1m05s after.
  • The repo mapping adds about 3.5s.

Issues

Stack

  1. @ fix(targethasher): collapse apparent-name bzlmod external files via repo mapping #322
  2. fix(itg): wire bzlmod external-target collapse into incremental graph updates #324

…epo mapping

Summary:
Intent:
- `bazel query` prints repos visible to the main module by apparent name (`@name//...`), but bzlmod marker hashes are keyed by canonical name and the collapse only recognized `@@canonical//...` labels. Files in such repos missed both the git known-hash map and the marker collapse, so the hasher read them from disk. On fievel that was 26k files and about 40 GB (large jars, zips and container images), making `targethasher.FromProto` the largest part of `nativeGraphRunner.Compute`.

Changes:
- Add `bazel.RepoMapping`, which runs `bazel mod dump_repo_mapping ""` to get the apparent-to-canonical repo map, and `HashConfig.RepoMapping` to carry it.
- Resolve `@name//...` labels through the mapping in `HashExternalTargetsBzlmod`, so they collapse to the same marker-based hash as the equivalent `@@canonical//...` label. A mapped repo with no marker (for example the built-in `@bazel_tools`) is left alone instead of erroring; `fullHashRepos` matches either name.
- `nativeGraphRunner.Compute` fetches the mapping for bzlmod repos and emits `repo_mapping_duration`. A failure fails the compute, like a marker read failure, so hashes stay deterministic.

Compatibility:
- Collapsed external files now hash from their marker instead of their content, so their hashes (and the graph hashes above them) change. Cached graphs computed before this change will report those external targets as changed once. Cache keys do not encode an algorithm version, so existing entries need to be invalidated or the one-time diff accepted.

Test Plan:
Measured with a local benchmark harness (not part of this PR) on a 1.3M-target bzlmod monorepo (about 640k files, 18.8k repos) with a warm Bazel server. The target count is unchanged at 1,321,910.
- `targethasher.FromProto`: 65-68s before, 10.6-12.7s after.
- Total `Compute`: about 2m05s before, about 1m05s after.
- The repo mapping adds about 3.5s.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@yushan8
yushan8 force-pushed the yushan/bzlmod-apparent-repo-collapse branch from c08dbb6 to ad4048c Compare October 2, 2026 22:28
…collapse

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
yushan8 and others added 2 commits October 2, 2026 16:03
… repos

query-bench only timed the bazel query and the hasher, and hashed with an
empty HashConfig, so it failed on Bzlmod repos (it looked for //external).
Add a --bzlmod flag (default true) that mirrors nativeGraphRunner.Compute:
drop //external from the query, then time git file hashes, repo marker reads
and the repo mapping, and hash with UseBzlmod and the real known-source,
marker and mapping inputs.

Each phase is logged through zap with its run, duration and counts. The
hasher phase can be profiled with --cpuprofile, and the graph JSON dump is
now opt-in via --dump so it no longer floods stdout on large repos.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Avoid panicking when the logger cannot be created, and make main the only
caller of os.Exit with deferred cleanup still running. Give each run its own
timeout context instead of deferring cancel inside the loop, and split the
run into per-phase helpers with a small timer for the repeated
start/elapsed/log bookkeeping, which removes the deep nesting. Replace the
flag locals with an options struct and stop ignoring the profile file's
Close error.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@yushan8
yushan8 marked this pull request as ready for review October 2, 2026 23:55
@yushan8
yushan8 requested review from a team as code owners October 2, 2026 23:55
@yushan8
yushan8 marked this pull request as draft October 5, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant