Skip to content

feat(probes): signal when an IntentProbe is skipped for inactive intents#1903

Open
WatchTree-19 wants to merge 1 commit into
NVIDIA:feature/technique_intentfrom
WatchTree-19:intent-skip-notice-1889
Open

feat(probes): signal when an IntentProbe is skipped for inactive intents#1903
WatchTree-19 wants to merge 1 commit into
NVIDIA:feature/technique_intentfrom
WatchTree-19:intent-skip-notice-1889

Conversation

@WatchTree-19

Copy link
Copy Markdown

Description

An IntentProbe whose intents are all outside the active set (e.g. filtered out by an intent: run.spec selector) produces no prompts and returns early from probe(). Before this change that path emitted only a logging.debug, so the skip was silent: the terminal showed no notice and the report carried no context for the missing probe (#1889).

This adds _notify_skipped_no_active_intents(), called on that early return, which:

  • emits a clear terminal notice (⚠️ probe ... skipped: none of its intents are in the active set ...), matching the existing warn_unconsumed_intents convention, and
  • writes a probe_skipped report entry (reason: no_active_intents, plus the intents considered) so the skip surfaces in the report/digest.

Targets feature/technique_intent since the intent run-spec selector (from #1866) lives on that branch, not main.

Fixes #1889.

Type of change

  • Bug fix / small feature (non-breaking)

Verification

Added tests/cas/test_intent_skip_notice.py, which verifies the skipped probe (a) writes exactly one probe_skipped report entry with reason: no_active_intents, and (b) prints the terminal notice. Existing tests/cas/test_cas_intentprobe.py and tests/cas/test_intent_run_spec.py still pass (20 tests). ruff/black clean on the changed lines.

pytest tests/cas/test_intent_skip_notice.py tests/cas/test_cas_intentprobe.py tests/cas/test_intent_run_spec.py
  • DCO sign-off included
  • Unit tests added

@leondz leondz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine catch, this situation should be handled appropriately

there is a more general case that could probably be dealt with here. for any probe that fails to produce prompts, the probe behaviour should lead to the harness skipping (via "SKIPPED") that probe. could be mediated via exception or via probe return.

suggestion:

  • add a hook method in probes.base.Probe for checking probe consistency (bool)
  • update harness to call this method between probe instantiation and calling of probe()
  • in the probe's method for handling the error, add a check for consistency and return value, e.g.
if not self.prompts: 
  return False
return True
  • add a test for a probe having no prompts that ends up being skipped by harness
  • be careful for probes that intentionally don't know all their prompts before probe() is called
  • add a test that checks probes for consistency after loading them
  • check that tests pass

alternatively another hook location might work, or passing by exception might work, let's see

Comment thread garak/probes/base.py Outdated
logging.debug("%s has no active intents; no prompts to send", self.probename)
# yields no prompts; no-op so the rest of the run proceeds (3A). Surface
# the skip so it is not silent on the terminal or in the report (#1889).
self._notify_skipped_no_active_intents()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why have this method here?

  • seems like there should be some logging etc whenever self.prompts is empy
  • a guard also belongs around L385 in Probe
  • the behavior should be to SKIP at harness level
  • value of this method is limited with such tightly scoped functionality
  • probes don't write to stdout
  • concrete probes don't write to reports

Comment thread garak/probes/base.py Outdated
f"no prompts to send (check your intent: run.spec selector)"
)
logging.info(msg)
print(f"⚠️ {msg}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no stdout from probes thanks

Comment thread garak/probes/base.py Outdated
)
logging.info(msg)
print(f"⚠️ {msg}")
reportfile = getattr(_config.transient, "reportfile", None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_config shouldn't be externally accessed from probes during runtime

Comment thread garak/probes/base.py Outdated
print(f"⚠️ {msg}")
reportfile = getattr(_config.transient, "reportfile", None)
if reportfile is not None:
reportfile.write(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probe's should not write to report about things other than attempts

Address review on NVIDIA#1889: instead of a bespoke IntentProbe method that
printed and wrote to the report, handle the general case.

- base Probe.probe: guard empty self.prompts (logging.info + return []),
  which also fixes a latent prompts[0] IndexError.
- remove the IntentProbe.probe override; base now covers it.
- harness: skip and log when a probe returns no attempts.
- no stdout from probes, no _config access, no report writes.

Update tests to assert the logging / no-stdout / no-report behaviour.

Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
@WatchTree-19 WatchTree-19 force-pushed the intent-skip-notice-1889 branch from 87efd07 to 972f460 Compare July 3, 2026 13:01
@WatchTree-19

Copy link
Copy Markdown
Author

thanks, that all makes sense. reworked:

  • dropped the IntentProbe method and moved the empty-prompts handling into base Probe.probe (~L385), so any probe left with no prompts is covered, not just IntentProbe. also guards the prompts[0] access that would otherwise IndexError.
  • logging only: logging.info on the empty case, no stdout from the probe.
  • removed the report write and the _config access from the probe; only attempts go to the report now.
  • skip is at harness level: the harness logs and continues when a probe returns no attempts.

one judgment call: i keyed the harness skip off an empty result rather than if not probe.prompts, so it doesn't wrongly skip probes that build attempts dynamically (e.g. atkgen). happy to switch to a prompts check if you'd prefer the harness decides before calling probe().

tests updated to cover the logging / no-stdout / no-report behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants