pass@k for ASR

[stefanoamorelli]

Each probe sends a prompt to the model a few times (probes/base.py#L299-L304), and the attack success rate represents how many of those replies resulted in a breach.

The problem that I see with this approach is that an attacker doesn't need the model to fail most of the time. They need it to fail once, and they can just try again. So a jailbreak that only lands on 1 try in 5 is effectievly a full break of that prompt, but it shows up as only 20% ASR, missing the point of a critical vulnerability.

an approach I would suggest is pass@k, the metric introduced in OpenAI's Codex paper (Chen et al., 2021). instead of pooling, you ask, per prompt: "given k tries, does at least one get through? "

it gives a more meaningful ASR reading as it reflects how the model actually gets attacked, because, fundamentally, an attacker can keep trying.

[jmartin-tech]

Linking to discussion #1684 that is open relating to this issue. Meaningful metrics vary by user intent, but the point here on how this tool defines ASR is acknowledged to need improvement.

[chuenchen309]

Thanks @stefanoamorelli — the "an attacker only needs to land once" framing is the right lens, and the pooled rate genuinely understates single-prompt breaks. Having just read through the evaluation path, a few notes on where this would slot in and one design fork worth deciding early:

Where the pooling happens. Evaluator.evaluate() (evaluators/base.py) flattens across both loops — outer over attempts (prompts), inner over each generation's detector_results — into a single passes/fails tally, so failrate = (evals - passes) / evals is a per-generation mean. pass@k is a per-prompt reduction applied before that flattening: for each attempt, "did ≥1 of its k generations breach?", then average over attempts. The data already supports this cleanly — each attempt.detector_results[detector] is the per-generation vector for one prompt, aligned to attempt.outputs.

Additive, not a replacement. Since @jmartin-tech noted meaningful metrics vary by user intent, I'd suggest reporting attack-success@k alongside the current pooled ASR in the eval_record rather than changing the existing number — they answer different questions (expected per-response breach rate vs. per-prompt breakability under retries), and both are useful.

One naming caveat for the codebase: garak's evaluator vocabulary uses pass = model resisted (safe), so this metric is really "attack-success within k tries." Worth naming it to match garak's convention rather than Codex's pass (which meant success), to avoid an inverted-sense trap for readers.

The design fork — empirical vs. estimator. k maps naturally to _config.run.generations. If generations == k, attack-success@k is just the empirical "any breach in the k". But if a user runs generations = n > k, the Codex paper's unbiased estimator (1 − C(n−c, k)/C(n, k) over c breaches in n samples) gives a lower-variance estimate of the same quantity. Do we want v1 to be the simple empirical any-of-k (report at k = generations), and treat the n>k estimator as a follow-up — or bake the estimator in from the start? I'd lean empirical-first for interpretability.

Also worth deciding: how to treat None scores (unevaluated generations) in the per-prompt denominator — I'd count a prompt only if it has ≥1 non-None result.

Happy to help implement or review whichever direction you and the maintainers land on, @stefanoamorelli — didn't want to get ahead of you on your issue.