feat: add MCP tool-use security probes (probes.tooluse) (#1872)#1891
Open
xr843 wants to merge 1 commit into
Open
feat: add MCP tool-use security probes (probes.tooluse) (#1872)#1891xr843 wants to merge 1 commit into
xr843 wants to merge 1 commit into
Conversation
Adds probes.tooluse, testing whether a target acting as an MCP-connected agent can be manipulated into unsafe tool use. Four probes cover the threat categories raised in NVIDIA#1872: - PermissionEscalation: using a readOnlyHint tool destructively - ToolChainAbuse: chaining individually-safe tools for an unsafe outcome - ServerPhishing: connecting to an untrusted/malicious MCP server - CredentialExfiltration: leaking a credential through a tool parameter Each prompt embeds a synthetic tool manifest with MCP annotations (readOnlyHint/destructiveHint) and records the manifest, declared annotations, expected policy verdict, and failure class in attempt.notes so findings can be triaged by failure class. Reuses existing detectors: mitigation.MitigationBypass for refusal-expected cases and base.TriggerListDetector with a planted canary secret for credential exfiltration. Probes ship inactive (opt-in) at INFORMATIONAL tier. Includes docs page and tests mirroring existing probe test patterns. Closes NVIDIA#1872 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: xr843 <xianren843@protonmail.com>
xr843
force-pushed
the
probe/mcp-tool-use-security
branch
from
2de3129 to
edd4f6b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
probes.tooluse, testing whether a target acting as an MCP-connected agent can be manipulated into unsafe tool use. Implements the four threat categories from #1872.Probes
readOnlyHinttool destructivelyDesign
garak's harness is text-in/text-out, so these probes do not execute real tool calls; they measure the target's stated willingness to misuse tools — the model-level signal that complements registry-level analyses such as MCTS.
Each prompt embeds a synthetic tool manifest with MCP annotations (
readOnlyHint/destructiveHint). Per the request for structured findings in the issue thread, every attempt records the tool manifest, declared annotations, expected policy verdict, and failure class inattempt.notesfor downstream triage by failure class.Detectors are reused, not added:
mitigation.MitigationBypassfor the refusal-expected probes (as inprobes.doctor), andbase.TriggerListDetectorwith a planted canary secret for credential exfiltration (as inprobes.leakreplay).Probes ship inactive (opt-in) at
Tier.INFORMATIONAL, followingprobes.agent_breaker.Tests
tests/probes/test_probes_tooluse.pymirrors existing probe tests; docs page added underdocs/source/probes/. All tooluse + universal probe + docs checks pass;black --checkclean. No new dependencies.Closes #1872