Skip to content

Refuse done when tests/ holds no test that unittest would run (#173) - #367

Open
the-ashish2005 wants to merge 1 commit into
YauhenBichel:mainfrom
the-ashish2005:squasher/fix-173-677d671f56
Open

the-ashish2005 wants to merge 1 commit into
YauhenBichel:mainfrom
the-ashish2005:squasher/fix-173-677d671f56

Conversation

@the-ashish2005

Copy link
Copy Markdown

Defect

py-harness run "write tests for apply_discount" finished with Action: done and a success summary after writing test files that python -m unittest discover collects nothing from. Reproduced with the exact artifact from the report: discovery exits 0 with Ran 0 tests, while the harness's finish gate returned no refusal at all.

tests/test_module.py held a test method with no class around it:

def test_apply_discount(self) -> None:
        from pricing import apply_discount
        got = apply_discount(100.0, 25.0)
        self.assertEqual(got, 75.0)

It compiles, so nothing errors. Discovery imports the file, finds no TestCase subclass, collects nothing and exits 0, and the run reported success.

Root cause

_a_test_uses in src/harness/skillkit/refuse_finish.py accepted any def test_... at any scope. The orphan method names apply_discount, so the no test calls apply_discount refusal never fired and done was reachable.

The run-path half of #173 (_suite_passed, NO_TESTS_RAN) already sends the model back when a suite it ran collected nothing; what was missing is that a model answering done anyway still had to get past this oracle, and the oracle counted a method no runner calls.

Change

  • tests_call/_a_test_uses now count a test only when it is a method of a class unittest discover would collect: a TestCase subclass, directly or through another class declared in the same file. Module-level def test_... functions, imports and classes with no TestCase base are no longer coverage.
  • Test files are judged one at a time instead of concatenated, so one bad file cannot decide the verdict for the rest, and a file that cannot be parsed is not coverage (discovery cannot import it either).
  • tests/agent/test_zero_tests_is_not_green.py gains TheFinishGateIsToldTest: the reported file is refused by the finish gate, a real TestCase still finishes, and the refusal is checked against what discovery actually collects for five file shapes (orphan method, import only, plain class, real TestCase, subclass of a local TestCase).
  • tests/scan/test_names.py had a fixture that wrote a module-level test method and asserted the oracle accepted it as coverage. It now writes a TestCase, which is what the rule always meant.

Verification

A reproducer rebuilds the project from the report in a temporary directory, runs discovery, then asks the finish gate for a verdict. Before the change it printed Ran 0 tests together with an empty refuse_done, and exited 1 with BUG REPRODUCED on 3 of 3 runs. After the change, on 3 of 3 runs it prints the same Ran 0 tests and the gate answers no test calls apply_discount. Action: patch Path: tests/test_module.py Append: one AAA test that calls apply_discount., exiting 0 with FIXED.

The three new regression tests fail on the base revision (FAILED (failures=3)) and pass after it. The full suite is green on both: python -m unittest discover -s tests -q reports OK (skipped=1) with 1132 tests after the change (1129 on the base revision).


Automated contribution. This pull request was prepared by Squasher, an automated agent that reproduces a reported defect in a sandbox before proposing a fix. A human reviewed and approved this patch before it was opened.

No maintainer requested this change. Please close it without hesitation if it is unwanted, out of scope, or does not meet the project's standards.

@the-ashish2005
the-ashish2005 marked this pull request as ready for review September 29, 2026 05:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant