fix(analyzers): rely on runner for example filtering in E5 and TM4 (close executable bypass) - #237
Conversation
b1c3657 to
308da53
Compare
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Approved. Removing analyzer-level is_code_example() suppression from E5/TM4 correctly restores the shared runner’s file-type-aware policy: non-executable examples remain filtered, while executable examples are only downweighted and can no longer evade detection. Focused regressions cover both rules. The current CI format failure is in unchanged mcp_least_privilege.py, not this diff.
|
Fix linting please @CharmingGroot |
308da53 to
691791f
Compare
|
@CharmingGroot - Please address review comments and resolve merge conflicts. |
E5 (NVIDIA#218) and TM4 (NVIDIA#220) called is_code_example() with an unconditional continue, letting a nearby example marker (e.g. "# for example") suppress findings in executable files. The shared runner already filters examples in non-executable docs and only downweights executables, so the analyzer-level call was redundant and created an attacker-controlled bypass — the same issue fixed for SC7 in NVIDIA#224. Remove it from both analyzers; replace TM4's analyze-level doc-exclusion test with an executable-evasion test and add the equivalent E5 regression. Signed-off-by: CharmingGroot <ohyes9711@gmail.com>
691791f to
70f81d0
Compare
|
Rebased onto current main — conflict in The |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Approved. E5/TM4 no longer apply an analyzer-local example hard-drop; the shared runner remains the single policy point, preserving documentation filtering while closing executable-file bypasses. Required CI is green.
Summary
The SC7 review (#224) established that calling
is_code_example()inside an analyzer with an unconditionalcontinuelets a nearby example marker suppress findings in executable files — the shared runner already filters examples in non-executable docs and only downweights executables, so the analyzer-level call is redundant and opens an attacker-controlled bypass. E5 (#218) and TM4 (#220), both merged, use the same pattern. This applies the same fix.Changes
static_patterns_data_exfiltration.py(E5): drop theis_code_examplecall and import; rely on the runner's file-type-aware handling.static_patterns_tool_misuse.py(TM4): same.# for examplenext toprivileged: truestill fires); an equivalent E5 regression is added.Why this is safe
Detection is unchanged — the runner still suppresses non-executable docs and downweights executables. Only the bypass is closed: a
# for examplenears3.put_object(...)orprivileged: truein a.sh/.pyno longer evades E5/TM4.anti_refusalalready uses a confidence penalty (not a skip), so it is intentionally left as-is.Testing
make formatandmake lintpass; the full suite reports 1256 passed, 0 failed. A re-scan confirms detection is preserved (E5 ×2, TM4 ×6 still fire on the same fixtures).Follows up on the review in #224.