Skip to content

test(probes): un-shadow leakreplay probe structure test#1914

Open
anxkhn wants to merge 1 commit into
NVIDIA:mainfrom
anxkhn:patch-15
Open

test(probes): un-shadow leakreplay probe structure test#1914
anxkhn wants to merge 1 commit into
NVIDIA:mainfrom
anxkhn:patch-15

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Two test functions in tests/probes/test_probes_leakreplay.py were both named
test_leakreplay_probe_structure. Python binds the later def over the earlier
one in the module namespace, and pytest collects test functions from that
namespace by name, so the first definition (the broad structure test
parametrized over LEAKREPLAY_PROBES, all 16 leakreplay probes) was silently
overwritten and never collected. Only the second definition ran, and it is
parametrized over CLOZE_PROBES (the 8 Cloze/ClozeFull probes). As a
result the 8 Complete/CompleteFull leakreplay probes had no structure, tag,
or %-escape coverage at all.

This renames the second, Cloze-specific function to
test_leakreplay_cloze_probe_structure (its @pytest.mark.parametrize already
targets CLOZE_PROBES) so both tests are now collected and run.

Un-shadowing the broad test surfaced a latent bug in that test: it looked the
probe class up with the fully qualified plugin name, e.g.
getattr(garak.probes.leakreplay, "probes.leakreplay.PotterComplete"), which
raises AttributeError. Because the test had never actually executed, the
defect had gone unnoticed. It is fixed by resolving the bare class name with
klassname.split(".")[-1], the same idiom already used elsewhere in the test
suite (e.g. tests/plugins/test_plugins.py, tests/detectors/test_detectors.py,
tests/generators/test_generators.py).

Net effect: the Complete/CompleteFull probes regain full structure/tag/
%-escape validation, and the two tests no longer collide.

Verification

  • Run the tests and ensure they pass python -m pytest tests/
    • python -m pytest tests/probes/test_probes_leakreplay.py -> 28 passed
      (was 20 before: the broad 16-case test was not being collected).
  • Verify collection is restored:
    python -m pytest tests/probes/test_probes_leakreplay.py --collect-only
    now shows 16 test_leakreplay_probe_structure[...] cases (all
    LEAKREPLAY_PROBES) plus 8 test_leakreplay_cloze_probe_structure[...]
    cases (was 8 total).
  • Verify the reinstated cases do real work: PotterComplete (a
    Complete probe) carries the expected tags and the %-escape handling the
    broad test asserts, so the newly collected cases exercise real assertions,
    not vacuous passes.
  • Formatting unchanged: black --check tests/probes/test_probes_leakreplay.py.

No production code changes; test-only, one file, +2/-2.

Not a duplicate

No open PR touches tests/probes/test_probes_leakreplay.py. The nearby
test-coverage PRs add different files (#1853 adds detector/probe tests for other
modules, including tests/detectors/test_detectors_leakreplay.py; #1818 adds a
new probe and its own test file), and no open issue tracks this shadowed
duplicate-name test.

AI assistance

AI assistance was used to prepare this change. I have reviewed every changed
line, reproduced the missing-coverage behaviour and the fix locally, and ran the
test suite for the affected file.


Notes for maintainers scope

  • Follows test(probes): conventional-commit style used across the repo.
  • DCO sign-off and Co-authored-by: Claude trailer are in the commit.
  • No new dependencies, no base-class edits, no SPDX header (nothing imported
    under another license).

Two test functions in tests/probes/test_probes_leakreplay.py shared the
name test_leakreplay_probe_structure. Python binds the later definition
over the earlier one, so the broad, parametrized structure test over all
16 leakreplay probes (LEAKREPLAY_PROBES) was never collected; only the 8
Cloze/ClozeFull probes were exercised, via the second definition. The 8
Complete/CompleteFull probes had no structure, tag, or %-escape coverage.

Rename the second, Cloze-specific function to
test_leakreplay_cloze_probe_structure (it is already parametrized over
CLOZE_PROBES) so both tests are collected.

Once un-shadowed, the broad test failed for every probe because it looked
the class up with the fully qualified plugin name
(getattr(garak.probes.leakreplay, "probes.leakreplay.PotterComplete")).
Resolve the bare class name with klassname.split(".")[-1], matching the
idiom used elsewhere in the test suite.

Collection now yields 16 test_leakreplay_probe_structure cases plus 8
test_leakreplay_cloze_probe_structure cases (was 8 total), and the file
passes: uv run pytest tests/probes/test_probes_leakreplay.py -> 28 passed.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant