fix(rules): surface rule failures during export - #761
Open
tangcent wants to merge 1 commit into
Open
Conversation
Rules that throw while evaluating endpoints previously skipped those endpoints silently: aggregation drops RuleResult failures, so a throwing custom.method.is.api made every method evaluate to "not an API" and the export came back empty with no explanation. AI-authored proposals calling context API that does not exist (e.g. it.static as a property on a method context, it.args) passed the static-only review and then broke every export. Root cause: RuleEngine aggregation silently drops failed evaluations, and RuleProposalValidator never executes the proposed groovy scripts, so API-miss scripts slipped through to production rule files. Solution: - RuleFailureMonitor aggregates rule failures per run; ExportOrchestrator brackets runs so one warning balloon surfaces at the end instead of a silent empty export. RuleEngine records throwing filters and values and logs each occurrence. - RuleDryRunValidator executes every groovy value/filter once against representative PSI contexts before a proposal is staged; compile errors or API misses on every context kind block the proposal, others become reviewer notes. - Add missing isStatic() to the method context (API symmetry with class and field contexts) so it.static resolves. - Soft-warn canonicalText() on parameter contexts (returns the element path, not the parameter type). Impact: well-behaved rules behave exactly as before (no failure, no balloon); only failing rules become visible. Dashboard scans outside a run window are logged per occurrence but never ballooned. Fixes: #757
Contributor
|
📦 Plugin has been packaged for this PR. You can download |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #761 +/- ##
=============================================
+ Coverage 66.827% 66.927% +0.100%
=============================================
Files 443 445 +2
Lines 25952 26136 +184
Branches 6195 6251 +56
=============================================
+ Hits 17343 17492 +149
- Misses 6289 6294 +5
- Partials 2320 2350 +30
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #757 — AI-generated
custom.*rules that throw at evaluation time nolonger silently skip every endpoint.
Problem
A throwing
custom.method.is.apirule made every method evaluate to"not an API" (
runCatching{...}.getOrDefault(false)in the exporter), soexports came back silently empty with only
WARNlines inidea.log.Three defects in one generated rule file went unnoticed because
RuleProposalValidatoronly did static checks and never executed theproposed scripts:
it.staticthrewMissingPropertyExceptionon method contexts(
isStatic()was missing, unlikeClassContext/FieldContext).it.args(property access) threw —args()is a plain method.it.canonicalText()on a param context returned the element path(
com.example.Foo#bar.userId), not the parameter's type.Changes
RuleFailureMonitor(new, project service) aggregates rule-evaluationfailures per run;
ExportOrchestratorbrackets each run so one warningballoon surfaces at the end instead of a silent empty export.
RuleEnginenow logs and records throwing filters and values peroccurrence (previously dropped silently by aggregation).
RuleDryRunValidator(new) executes everygroovy:value/filter in aproposal once against representative PSI contexts before staging —
compile errors or API misses on every context kind block the proposal,
others become reviewer notes. Wired into
ProposeRuleContentToolandOrchestratorProposeRuleContentTool(which now usereviewWithDryRun).isStatic()added to method contexts (API symmetry with class/fieldcontexts) so
it.staticresolves.it.canonicalText()on parameter-context keys(suggests
type().name()).Testing
RuleFailureMonitorTest(6 new) — run windowing, aggregation, ballooncontent, buffer clearing.
RuleDryRunValidatorTest(15 new) — script collection, context-kindselection, compile-error/API-miss classification, extra-binding-key
skipping.
ScriptPsiContextsTest.testMethodContext_IsStatic+RuleProposalValidatorTestparam-
canonicalTextwarning tests.RuleEngineTest,AntiPatternGateTest, andRuleProposalValidatorTestsuites pass../gradlew compileTestKotlinclean.Risks / rollback
at proposal time; infrastructure failures are logged and swallowed — a
proposal is never blocked by a dry-run infra error. Keys whose
evaluation stage needs extra bindings (
api,request/response,collection, …) are skipped.logged per occurrence but never ballooned.