Separate test layers and close invocation and scope coverage gaps

[Kerr1st]

The 912-case suite could stay green with Explorer/Thinker invocation disabled or guidance applicability bypassed. This change separates deterministic behavior checks, real database integration, and evidence-based AI outcome review, and closes those demonstrated coverage gaps.

• Introduces selectable behavior/integration layers and CI jobs; one guarded database setup replaces three local overrides. Python network calls are blocked in deterministic tests.
• Exercises actual Explorer/Thinker invocation, scope rejection, and Codex capture through context delivery and a recorded outcome.
• Removes six confirmed low-value cases and replaces three copied-payload checks. All 16 consensus permutations remain in a compact table.
• Evaluation commands now distinguish completed measurements, failures (exit 1), and insufficient evidence (exit 2). Aggregate reports can be written without reconnecting to a failed database.
• Documents the separate AI outcome protocol, six-criterion scorecard, holdout discipline, and historical Job Search limitations. Private source/output evidence remains outside Git. This PR does not claim a fresh AI usefulness pass.

Validation: 934 passed in 21.49s with coverage; 735 behavior and 199 integration cases also pass independently. The new checks detect all seven disabled-invocation scenarios and six applicability-bypass scenarios. Evaluation failures/empty tiers failed before their fix and pass afterward. The wrapper rejects non-disposable database names. Existing SQLite ResourceWarnings and the Starlette deprecation warning remain documented.

Application capture, retrieval, model prompts/configuration, and steering-approval behavior are unchanged. The executable runtime change is confined to evaluation reporting and exit status.

Clean GitHub validation: both behavior and integration jobs passed at commit bb1573b (run). The compact consensus table also detects the two-vote threshold fault (7 expected failures).

[amazon-q-developer]

This PR successfully separates test layers and closes invocation/scope coverage gaps. The implementation properly distinguishes behavior tests (fast, no database) from integration tests (real PostgreSQL), with appropriate network isolation and CI jobs for each layer.

Key improvements validated:

• Test classification via pytest hooks correctly identifies database dependencies and prevents conflicting markers
• Network isolation fixture blocks external calls in behavior tests while preserving PostgreSQL access for integration tests
• Evaluation runner now returns distinct exit codes: 0 (success), 1 (failure), 2 (insufficient evidence)
• Database safety checks prevent accidental use of non-disposable database names
• CI workflow properly configures separate jobs with appropriate timeouts and service dependencies

The changes maintain backward compatibility while enabling independent execution of test layers. All 21 files modified follow consistent patterns and proper error handling.

---

You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.