Skip to content

orc pgo/02 skeleton counter layout - #725

Closed
makslevental wants to merge 5 commits into
mainfrom
orc-pgo/02-skeleton-counter-layout
Closed

makslevental wants to merge 5 commits into
mainfrom
orc-pgo/02-skeleton-counter-layout

Conversation

@makslevental

@makslevental makslevental commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Second PR of the orc-pgo stack. It adds the project skeleton, the coverage gate and CI, plus the first library piece, CounterLayout.

Project (projects/orc-pgo)

  • OrcPgo is a static library. It links the static LLVM component libraries from llvm_map_components_to_libnames, never libLLVM.dylib, so every process has exactly one copy of LLVM. On macOS, otool -L on the test binary shows only libSystem and libc++.
  • Unit tests use GoogleTest v1.15.2 via FetchContent. unittests/TestSupport has helpers for parsing IR, getting a host TargetMachine and JITting a module, which later PRs use.

CounterLayout

  • Assigns positional counter indices: one counter per function entry, plus one per successor/arm of each conditional br, each multi-successor switch and each i1 select. Indices follow module and instruction order.
  • Declarations, available_externally functions and __orc_pgo* internals are skipped.
  • Computing the layout on two copies of the same IR gives identical indices. That's what will let the runtime map the AOT counters back onto the embedded bitcode snapshot without hashes or name tables.
  • The layout is computed on the pre-instrumentation IR, before any optimization, so later pass reordering doesn't affect it.

Coverage gate (scripts/cpp_coverage.sh)

  • Follows the eudsl-llvmpy model: build with -fprofile-instr-generate -fcoverage-mapping, run ctest, merge the profiles, then require 100% line and 100% function coverage over include/, lib/ and tools/ using the repo-root scripts/check_coverage.py.
  • llvm-cov/llvm-profdata must match the clang that produced the profiles, because raw .profraw formats are version-locked. On macOS that means Apple clang with the xcrun tools; on Linux, setup_base's llvm-18.

CI

  • build_test_orc_pgo.yml runs the coverage gate on ubuntu-22.04, ubuntu-22.04-arm and macos-14, using setup_base like the other workflows.
  • It gets the distro the same way build_test_release_eudsl.yml does: pip downloads the released mlir-wheel normally, or downloads the wheel artifact from the caller's run under workflow_call.
  • build_llvm.yml gets a call-build-orc-pgo job, so distro changes are tested against orc-pgo with the wheel built in that run.

Also adds /.claude/ to .gitignore.

Add projects/orc-pgo: a static OrcPgo library linked against the static
LLVM component libraries (never libLLVM), GoogleTest unit tests, and
scripts/cpp_coverage.sh, which enforces 100% line/function coverage via
the repo-root scripts/check_coverage.py.

CounterLayout assigns positional counters (one per function entry plus
one per successor/arm of conditional branches, multi-successor switches
and i1 selects), so computing it on two copies of the same IR yields
identical indices.

CI runs the coverage gate on macOS arm64 and Linux x86_64/aarch64, using
either the released mlir-wheel or, when called from build_llvm.yml, the
wheel built in that run.
Comment thread projects/orc-pgo/lib/CounterLayout.cpp

@makslevental makslevental left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review (code, tests, comments, silent-failure passes). Inline comments below; summary:

Important

  1. The 100% gate can pass vacuously for lib/ files no test links (static archive members that aren't pulled in emit no coverage mapping, and check_coverage.py never checks that every file under --sources appears in the report).
  2. CounterLayout::compute mis-numbers a lazily loaded module (unmaterialized bodies count as definitions with zero sites).
  3. SameIRSameLayout doesn't test the property the design depends on (no bitcode round-trip, sites never compared).

Suggestions: Linux tool defaults in cpp_coverage.sh, push path filter, fixture shape (multi-site functions, skipped functions between eligible ones), comments describing not-yet-existing pieces, unused TestSupport helpers.

Outside this diff (scripts/check_coverage.py, shared, moved in #724):

  • An unterminated LCOV_EXCL_START silently excludes the rest of the file (should fail at EOF).
  • llvm-cov export stderr is discarded on success, so "N functions have mismatched data" warnings (which drop those functions from the denominator) go unseen.
  • Output is still labelled "eudsl-llvmpy C++ coverage" when run for orc-pgo; a --label arg would fix it.

Verified fine: artifact names vs build_llvm.yml uploads, call-build-orc-pgo permissions, LLVM_DEFINITIONS handling, static (not dylib) LLVM linking, coverage-tool/compiler version matching in CI, bash 3.2 behavior of the SOURCES loop.

Comment thread projects/orc-pgo/CMakeLists.txt
Comment thread projects/orc-pgo/lib/CounterLayout.cpp
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp Outdated
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Comment thread projects/orc-pgo/scripts/cpp_coverage.sh Outdated
Comment thread projects/orc-pgo/include/orc-pgo/CounterLayout.h Outdated
Comment thread projects/orc-pgo/include/orc-pgo/CounterLayout.h
Comment thread projects/orc-pgo/unittests/TestSupport.h Outdated
Comment thread .github/workflows/build_test_orc_pgo.yml Outdated
Comment thread .github/workflows/build_test_orc_pgo.yml
Comment thread projects/orc-pgo/include/orc-pgo/CounterLayout.h Outdated
@makslevental

Copy link
Copy Markdown
Contributor Author

Outside this diff (scripts/check_coverage.py, shared, moved in #724):

  • An unterminated LCOV_EXCL_START silently excludes the rest of the file (should fail at EOF).
  • llvm-cov export stderr is discarded on success, so "N functions have mismatched data" warnings (which drop those functions from the denominator) go unseen.
  • Output is still labelled "eudsl-llvmpy C++ coverage" when run for orc-pgo; a --label arg would fix it.

yes fix all of these

… and docs

check_coverage.py (shared with eudsl-llvmpy):
- fail when a C/C++ source under --sources has no coverage record (e.g. a
  static library member no test links in)
- fail on an LCOV_EXCL_START without a matching LCOV_EXCL_STOP
- surface llvm-cov export warnings; fail on mismatched profile data
- add --label instead of hardcoding eudsl-llvmpy

cpp_coverage.sh: derive llvm-cov/llvm-profdata from $CC on Linux, fail
loudly on missing tools, zero tests, no .profraw or no coverage objects;
drop references to directories that don't exist yet.

CounterLayout: assert the module is materialized; document the site and
skip rules, the per-arm switch counting, why invoke/indirectbr/callbr are
not profiled, and the identical-IR precondition. Tests now cover a bitcode
round-trip with full site comparison, multiple sites in layout order,
skipped functions between eligible ones, unprofiled terminators,
duplicate-destination switches and cross-module lookup.

Also pin the googletest download hash, drop TestSupport helpers that are
not used yet, and add the workflow file to the push path filter.
@makslevental

Copy link
Copy Markdown
Contributor Author

Outside this diff (scripts/check_coverage.py)

All three are fixed in 000468d:

  • Unterminated LCOV_EXCL_START: get_excluded_lines raises at EOF, and the checker prints FAILED: <file>:<line>: LCOV_EXCL_START without a matching LCOV_EXCL_STOP and exits 1. Verified with a synthetic file.
  • llvm-cov stderr: llvm-cov export stderr is now always printed. Output containing mismatched data fails the check, since llvm-cov drops those functions from the report.
  • Label: new --label argument (default C++). orc-pgo passes --label orc-pgo and eudsl-llvmpy passes --label eudsl-llvmpy. The docstring and argparse description are now generic.

The checker also fails when a source file under --sources has no coverage record at all. See the CMakeLists thread.

Sites are counted from the condition that picks the successor, before the
instruction executes; these terminators have no such condition, so the
rationale is not about edge splitting.

@makslevental makslevental left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test-focused review using mutation testing: 41 single-change mutants of lib/CounterLayout.cpp, each rebuilt and run in an isolated worktree.

No cheating. Expected indices are hand-written constants, not recomputed by the code under test. Each counting rule has a test that fails only when that rule is broken. The fixtures really contain what the tests claim:

  • the invoke, indirectbr and callbr in UnprofiledTerminators;
  • layout order that differs from RPO in SitesAreContiguousInLayoutOrder;
  • a scalar i1 condition in vecops.

No asserting loop can run zero times unnoticed. The bitcode round-trip test catches a mutant that the same-text test can't: sites ordered by memory address.

Results: 35 mutants killed, 1 equivalent (isDeclarationForLinker), 5 survived. The survivors are fixture gaps, covered by the first three inline comments:

  • switch guard > 2;
  • isInternalName as contains / "__orc" / "__";
  • linkonce functions skipped.

The other inline comments are about robustness and naming.

Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp Outdated
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp Outdated
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp Outdated
Comment thread projects/orc-pgo/unittests/CounterLayoutTest.cpp
Kill the five surviving mutants: add a two-successor switch, near-miss
internal names, and linkonce_odr/internal/weak/__-prefixed definitions
among the skipped functions. Also guard lookup() and getFunction()
before dereferencing, assert the unprofiled fixture's terminator kinds,
wrap expectSameLayout in ASSERT_NO_FATAL_FAILURE, and rename
AssignsDeterministicPositions to AssignsExpectedPositions.
@makslevental
makslevental added this pull request to stack #729 October 10, 2026 19:09
@makslevental

Copy link
Copy Markdown
Contributor Author

Superseded by the LLVM-PGO-reuse redesign (tracking: #726).

@makslevental
makslevental deleted the orc-pgo/02-skeleton-counter-layout branch October 11, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant