test(probes): un-shadow leakreplay probe structure test#1914
Open
anxkhn wants to merge 1 commit into
Open
Conversation
Two test functions in tests/probes/test_probes_leakreplay.py shared the
name test_leakreplay_probe_structure. Python binds the later definition
over the earlier one, so the broad, parametrized structure test over all
16 leakreplay probes (LEAKREPLAY_PROBES) was never collected; only the 8
Cloze/ClozeFull probes were exercised, via the second definition. The 8
Complete/CompleteFull probes had no structure, tag, or %-escape coverage.
Rename the second, Cloze-specific function to
test_leakreplay_cloze_probe_structure (it is already parametrized over
CLOZE_PROBES) so both tests are collected.
Once un-shadowed, the broad test failed for every probe because it looked
the class up with the fully qualified plugin name
(getattr(garak.probes.leakreplay, "probes.leakreplay.PotterComplete")).
Resolve the bare class name with klassname.split(".")[-1], matching the
idiom used elsewhere in the test suite.
Collection now yields 16 test_leakreplay_probe_structure cases plus 8
test_leakreplay_cloze_probe_structure cases (was 8 total), and the file
passes: uv run pytest tests/probes/test_probes_leakreplay.py -> 28 passed.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two test functions in
tests/probes/test_probes_leakreplay.pywere both namedtest_leakreplay_probe_structure. Python binds the laterdefover the earlierone in the module namespace, and pytest collects test functions from that
namespace by name, so the first definition (the broad structure test
parametrized over
LEAKREPLAY_PROBES, all 16 leakreplay probes) was silentlyoverwritten and never collected. Only the second definition ran, and it is
parametrized over
CLOZE_PROBES(the 8Cloze/ClozeFullprobes). As aresult the 8
Complete/CompleteFullleakreplay probes had no structure, tag,or
%-escape coverage at all.This renames the second, Cloze-specific function to
test_leakreplay_cloze_probe_structure(its@pytest.mark.parametrizealreadytargets
CLOZE_PROBES) so both tests are now collected and run.Un-shadowing the broad test surfaced a latent bug in that test: it looked the
probe class up with the fully qualified plugin name, e.g.
getattr(garak.probes.leakreplay, "probes.leakreplay.PotterComplete"), whichraises
AttributeError. Because the test had never actually executed, thedefect had gone unnoticed. It is fixed by resolving the bare class name with
klassname.split(".")[-1], the same idiom already used elsewhere in the testsuite (e.g.
tests/plugins/test_plugins.py,tests/detectors/test_detectors.py,tests/generators/test_generators.py).Net effect: the
Complete/CompleteFullprobes regain full structure/tag/%-escape validation, and the two tests no longer collide.Verification
python -m pytest tests/python -m pytest tests/probes/test_probes_leakreplay.py-> 28 passed(was 20 before: the broad 16-case test was not being collected).
python -m pytest tests/probes/test_probes_leakreplay.py --collect-onlynow shows 16
test_leakreplay_probe_structure[...]cases (allLEAKREPLAY_PROBES) plus 8test_leakreplay_cloze_probe_structure[...]cases (was 8 total).
PotterComplete(aCompleteprobe) carries the expected tags and the%-escape handling thebroad test asserts, so the newly collected cases exercise real assertions,
not vacuous passes.
black --check tests/probes/test_probes_leakreplay.py.No production code changes; test-only, one file,
+2/-2.Not a duplicate
No open PR touches
tests/probes/test_probes_leakreplay.py. The nearbytest-coverage PRs add different files (#1853 adds detector/probe tests for other
modules, including
tests/detectors/test_detectors_leakreplay.py; #1818 adds anew probe and its own test file), and no open issue tracks this shadowed
duplicate-name test.
AI assistance
AI assistance was used to prepare this change. I have reviewed every changed
line, reproduced the missing-coverage behaviour and the fix locally, and ran the
test suite for the affected file.
Notes for maintainers scope
test(probes):conventional-commit style used across the repo.Co-authored-by: Claudetrailer are in the commit.under another license).