Repository navigation
Conversation
…epo mapping Summary: Intent: - `bazel query` prints repos visible to the main module by apparent name (`@name//...`), but bzlmod marker hashes are keyed by canonical name and the collapse only recognized `@@canonical//...` labels. Files in such repos missed both the git known-hash map and the marker collapse, so the hasher read them from disk. On fievel that was 26k files and about 40 GB (large jars, zips and container images), making `targethasher.FromProto` the largest part of `nativeGraphRunner.Compute`. Changes: - Add `bazel.RepoMapping`, which runs `bazel mod dump_repo_mapping ""` to get the apparent-to-canonical repo map, and `HashConfig.RepoMapping` to carry it. - Resolve `@name//...` labels through the mapping in `HashExternalTargetsBzlmod`, so they collapse to the same marker-based hash as the equivalent `@@canonical//...` label. A mapped repo with no marker (for example the built-in `@bazel_tools`) is left alone instead of erroring; `fullHashRepos` matches either name. - `nativeGraphRunner.Compute` fetches the mapping for bzlmod repos and emits `repo_mapping_duration`. A failure fails the compute, like a marker read failure, so hashes stay deterministic. Compatibility: - Collapsed external files now hash from their marker instead of their content, so their hashes (and the graph hashes above them) change. Cached graphs computed before this change will report those external targets as changed once. Cache keys do not encode an algorithm version, so existing entries need to be invalidated or the one-time diff accepted. Test Plan: Measured with a local benchmark harness (not part of this PR) on a 1.3M-target bzlmod monorepo (about 640k files, 18.8k repos) with a warm Bazel server. The target count is unchanged at 1,321,910. - `targethasher.FromProto`: 65-68s before, 10.6-12.7s after. - Total `Compute`: about 2m05s before, about 1m05s after. - The repo mapping adds about 3.5s. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
yushan8
force-pushed
the
yushan/bzlmod-apparent-repo-collapse
branch
from
October 2, 2026 22:28
c08dbb6 to
ad4048c
Compare
…collapse Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
3 tasks done
yushan8
force-pushed
the
yushan/bzlmod-apparent-repo-collapse
branch
from
October 2, 2026 22:48
0d14495 to
7788840
Compare
… repos query-bench only timed the bazel query and the hasher, and hashed with an empty HashConfig, so it failed on Bzlmod repos (it looked for //external). Add a --bzlmod flag (default true) that mirrors nativeGraphRunner.Compute: drop //external from the query, then time git file hashes, repo marker reads and the repo mapping, and hash with UseBzlmod and the real known-source, marker and mapping inputs. Each phase is logged through zap with its run, duration and counts. The hasher phase can be profiled with --cpuprofile, and the graph JSON dump is now opt-in via --dump so it no longer floods stdout on large repos. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Avoid panicking when the logger cannot be created, and make main the only caller of os.Exit with deferred cleanup still running. Give each run its own timeout context instead of deferring cancel inside the loop, and split the run into per-phase helpers with a small timer for the repeated start/elapsed/log bookkeeping, which removes the deep nesting. Replace the flag locals with an options struct and stop ignoring the profile file's Close error. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
yushan8
marked this pull request as ready for review
October 2, 2026 23:55
yushan8
marked this pull request as draft
October 5, 2026 20:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
bazel queryprints repos visible to the main module by apparent name (@name//...), but bzlmod marker hashes are keyed by canonical name and the collapse only recognized@@canonical//...labels. Files in such repos missed both the git known-hash map and the marker collapse, so the hasher read them from disk. On the java repo that was 26k files and about 40 GB (large jars, zips and container images), makingtargethasher.FromProtothe largest part ofnativeGraphRunner.Compute.What?
bazel.RepoMapping, which runsbazel mod dump_repo_mapping ""to get the apparent-to-canonical repo map, andHashConfig.RepoMappingto carry it.@name//...labels through the mapping inHashExternalTargetsBzlmod, so they collapse to the same marker-based hash as the equivalent@@canonical//...label. A mapped repo with no marker is left alone instead of erroring;fullHashReposmatches either name.nativeGraphRunner.Computefetches the mapping for bzlmod repos and emitsrepo_mapping_duration. A failure fails the compute, like a marker read failure, so hashes stay deterministic.query-benchmirrorsComputefor Bzlmod repos (--bzlmod, on by default) and logs the duration of each phase through zap: query, git file hashes, marker read, repo mapping and hasher. It also gains--cpuprofilefor the hasher phase, and the graph JSON dump is opt-in via--dump.Compatibility: collapsed external files now hash from their marker instead of their content, so their hashes (and the graph hashes above them) change. Cached graphs computed before this change will report those external targets as changed once. Cache keys do not encode an algorithm version, so existing entries need to be invalidated or the one-time diff accepted.
Issue
None.
🤖 Generated with Claude Code
Test Plan
Measured with
example/cmd/query-benchon a 1.3M-target bzlmod monorepo (about 640k files, 18.8k repos) with a warm Bazel server. The target count is unchanged at 1,321,910.targethasher.FromProto: 65-68s before, 10.6-12.7s after.Compute: about 2m05s before, about 1m05s after.Issues
Stack