Skip to content

fix(rules): surface rule failures during export - #761

Open
tangcent wants to merge 1 commit into
masterfrom
fix/757-export-time-rule-failures
Open

fix(rules): surface rule failures during export#761
tangcent wants to merge 1 commit into
masterfrom
fix/757-export-time-rule-failures

Conversation

@tangcent

Copy link
Copy Markdown
Owner

Summary

Fixes #757 — AI-generated custom.* rules that throw at evaluation time no
longer silently skip every endpoint.

Problem

A throwing custom.method.is.api rule made every method evaluate to
"not an API" (runCatching{...}.getOrDefault(false) in the exporter), so
exports came back silently empty with only WARN lines in idea.log.
Three defects in one generated rule file went unnoticed because
RuleProposalValidator only did static checks and never executed the
proposed scripts:

  1. it.static threw MissingPropertyException on method contexts
    (isStatic() was missing, unlike ClassContext/FieldContext).
  2. it.args (property access) threw — args() is a plain method.
  3. it.canonicalText() on a param context returned the element path
    (com.example.Foo#bar.userId), not the parameter's type.

Changes

  • RuleFailureMonitor (new, project service) aggregates rule-evaluation
    failures per run; ExportOrchestrator brackets each run so one warning
    balloon surfaces at the end instead of a silent empty export.
  • RuleEngine now logs and records throwing filters and values per
    occurrence (previously dropped silently by aggregation).
  • RuleDryRunValidator (new) executes every groovy: value/filter in a
    proposal once against representative PSI contexts before staging —
    compile errors or API misses on every context kind block the proposal,
    others become reviewer notes. Wired into ProposeRuleContentTool and
    OrchestratorProposeRuleContentTool (which now use
    reviewWithDryRun).
  • isStatic() added to method contexts (API symmetry with class/field
    contexts) so it.static resolves.
  • Soft warning for it.canonicalText() on parameter-context keys
    (suggests type().name()).

Testing

  • RuleFailureMonitorTest (6 new) — run windowing, aggregation, balloon
    content, buffer clearing.
  • RuleDryRunValidatorTest (15 new) — script collection, context-kind
    selection, compile-error/API-miss classification, extra-binding-key
    skipping.
  • ScriptPsiContextsTest.testMethodContext_IsStatic + RuleProposalValidatorTest
    param-canonicalText warning tests.
  • Full RuleEngineTest, AntiPatternGateTest, and
    RuleProposalValidatorTest suites pass.
  • ./gradlew compileTestKotlin clean.

Risks / rollback

  • Dry-run validation scans up to 100 project classes and executes scripts
    at proposal time; infrastructure failures are logged and swallowed — a
    proposal is never blocked by a dry-run infra error. Keys whose
    evaluation stage needs extra bindings (api, request/response,
    collection, …) are skipped.
  • Well-behaved rules behave exactly as before: no failure, no balloon.
  • Failures outside a run window (e.g. background dashboard scans) are
    logged per occurrence but never ballooned.
  • Revert by reverting this commit; the branch is feature-isolated.

Rules that throw while evaluating endpoints previously skipped those
endpoints silently: aggregation drops RuleResult failures, so a throwing
custom.method.is.api made every method evaluate to "not an API" and the
export came back empty with no explanation. AI-authored proposals calling
context API that does not exist (e.g. it.static as a property on a method
context, it.args) passed the static-only review and then broke every
export.

Root cause: RuleEngine aggregation silently drops failed evaluations, and
RuleProposalValidator never executes the proposed groovy scripts, so
API-miss scripts slipped through to production rule files.

Solution:
- RuleFailureMonitor aggregates rule failures per run; ExportOrchestrator
  brackets runs so one warning balloon surfaces at the end instead of a
  silent empty export. RuleEngine records throwing filters and values and
  logs each occurrence.
- RuleDryRunValidator executes every groovy value/filter once against
  representative PSI contexts before a proposal is staged; compile errors
  or API misses on every context kind block the proposal, others become
  reviewer notes.
- Add missing isStatic() to the method context (API symmetry with class
  and field contexts) so it.static resolves.
- Soft-warn canonicalText() on parameter contexts (returns the element
  path, not the parameter type).

Impact: well-behaved rules behave exactly as before (no failure, no
balloon); only failing rules become visible. Dashboard scans outside a
run window are logged per occurrence but never ballooned.

Fixes: #757
@github-actions github-actions Bot added the type: bug Something isn't working label Aug 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Plugin has been packaged for this PR. You can download easy-api-3.1.9.252.0.zip from the GitHub Actions workflow run by clicking on the "Artifacts" dropdown.

@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.68750% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.927%. Comparing base (f6f7d03) to head (9186c48).

Files with missing lines Patch % Lines
...itangcent/easyapi/core/rule/RuleDryRunValidator.kt 80.159% 4 Missing and 21 partials ⚠️
...yapi/core/rule/context/RuleScriptContextCatalog.kt 25.000% 0 Missing and 6 partials ⚠️
...m/itangcent/easyapi/core/rule/engine/RuleEngine.kt 40.000% 2 Missing and 1 partial ⚠️
...angcent/easyapi/core/rule/RuleProposalValidator.kt 85.714% 0 Missing and 2 partials ⚠️
...ore/ai/tools/OrchestratorProposeRuleContentTool.kt 0.000% 1 Missing ⚠️
...tangcent/easyapi/core/export/ExportOrchestrator.kt 91.667% 1 Missing ⚠️
...ent/easyapi/core/rule/engine/RuleFailureMonitor.kt 95.833% 0 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##            master      #761       +/-   ##
=============================================
+ Coverage   66.827%   66.927%   +0.100%     
=============================================
  Files          443       445        +2     
  Lines        25952     26136      +184     
  Branches      6195      6251       +56     
=============================================
+ Hits         17343     17492      +149     
- Misses        6289      6294        +5     
- Partials      2320      2350       +30     
Flag Coverage Δ
unittests 66.927% <79.688%> (+0.100%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...nt/easyapi/core/ai/tools/ProposeRuleContentTool.kt 94.286% <100.000%> (ø)
...ent/easyapi/core/rule/context/ScriptPsiContexts.kt 49.716% <100.000%> (+0.143%) ⬆️
...ore/ai/tools/OrchestratorProposeRuleContentTool.kt 54.717% <0.000%> (ø)
...tangcent/easyapi/core/export/ExportOrchestrator.kt 66.250% <91.667%> (+3.750%) ⬆️
...ent/easyapi/core/rule/engine/RuleFailureMonitor.kt 95.833% <95.833%> (ø)
...angcent/easyapi/core/rule/RuleProposalValidator.kt 91.597% <85.714%> (-0.856%) ⬇️
...m/itangcent/easyapi/core/rule/engine/RuleEngine.kt 72.807% <40.000%> (+0.080%) ⬆️
...yapi/core/rule/context/RuleScriptContextCatalog.kt 86.059% <25.000%> (-1.338%) ⬇️
...itangcent/easyapi/core/rule/RuleDryRunValidator.kt 80.159% <80.159%> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6f7d03...9186c48. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] AI-generated custom rules can throw at export time — all endpoints silently skipped

1 participant