Attack Success Rate Is Not One Number: Per-Attempt Rates, hit@k, Detector Scoping and Why garak 0.17.1 Changes Scores

Three rates that are all called attack success rate

A red-team run sends many prompts to a target and samples several generations for each. Three different numbers can be computed from the same table of hits and misses, and reports routinely call all three the attack success rate. The pooled rate is hits divided by all generations. The per-prompt rate is the fraction of prompts with at least one hit. And hit@k asks, for each prompt, the probability that an attacker making k attempts gets at least one hit, averaged over prompts. The three agree only when every prompt is either always or never breached. For a real model they diverge sharply, and the pooled rate is the most flattering. A jailbreak that works on one reply in five is a working jailbreak, because the attacker can retry; the pooled rate records it as a mild twenty percent.

garak, the LLM vulnerability scanner, added hit@k in its 0.17.1 release on 16 October 2026 for exactly this reason. Its reporting documentation makes the argument in one sentence: an attacker does not need the target to fail most of the time, only once. The garak guide covers the tool itself; this article is about the metric.

The estimator and why it is unbiased

For one prompt with n sampled generations of which c are hits, the probability that at least one of k attempts drawn without replacement from those n is a hit is 1 minus the number of k-subsets with no hit divided by the number of k-subsets in total: 1 minus C(n minus c, k) over C(n, k). This is the pass@k estimator from the 2021 code-generation paper by Chen and colleagues, renamed hit@k because a success here is an attack that lands. Computing it as a product avoids large binomial coefficients, and it is defined only for k at most n: a prompt cannot be sampled more times than it was.

Illustrative python, checked for syntax; adapt names and storage to your system.

from math import comb

def hit_at_k(n, c, k):
    """Chance that at least one of k attempts, drawn without replacement from n
    sampled generations with c hits, is a hit. Unbiased for 1 - (1 - p) ** k."""
    if not 1 <= k <= n or not 0 <= c <= n:
        raise ValueError("need 1 <= k <= n and 0 <= c <= n")
    if n - c < k:
        return 1.0                      # every k-subset must contain a hit
    return 1.0 - comb(n - c, k) / comb(n, k)

def hit_at_k_product(n, c, k):
    """The same quantity as a product, stable for large n."""
    miss = 1.0
    for i in range(n - c + 1, n + 1):
        miss *= 1.0 - k / i
    return 1.0 - miss

def report(per_prompt, ks):
    """per_prompt: list of (n, c). Macro-average over prompts eligible for each k."""
    out = {}
    for k in ks:
        eligible = [(n, c) for n, c in per_prompt if n >= k]
        if eligible:
            out[k] = (sum(hit_at_k(n, c, k) for n, c in eligible) / len(eligible), len(eligible))
    return out

Worked numbers: a prompt sampled five times with one hit has hit@1 of 0.2, hit@3 of 1 minus C(4,3) over C(5,3), which is 1 minus 4 over 10, so 0.6, and hit@5 of 1.0. A prompt with two hits in five has hit@1 of 0.4 and hit@3 equal to 1 minus C(3,3) over C(5,3), which is 1 minus 1 over 10, so 0.9. Average those two prompts and hit@1 is 0.3, hit@3 is 0.75 and hit@5 is 1.0: the same run reads as a thirty percent problem or a certainty depending on how many attempts the threat model grants the attacker.

How garak reports it

garak scores each prompt at k equal to the number of generations it actually received, so a run configured with five generations always reports hit@5, which collapses to whether the prompt was breached at least once. The reporting.hit_at_k setting, default a list containing 1, adds smaller k values below that; setting it to null removes the metric. Each eval entry in the report carries a hit_at_k object keyed by k with a score and the count of eligible prompts, prompts with fewer than k scoreable generations are excluded for that k, and when prompts received unequal generation counts the always-on entry is keyed n rather than an integer. The CLI summary prints the figures after the attack success rate, and the digest copies them next to each detector's totals. One detail matters when you automate: the digest's absolute_score is a pass rate, higher is better, while hit_at_k values are hit rates, lower is better. A gate that mixes them up passes exactly the runs it should fail.

Detectors move scores too

A hit is whatever the detector says it is, so a detector bug is a measurement bug. The 0.17.1 release fixed several. A mitigation detector now matches refusal prefixes only at the start of the response, where before a refusal phrase quoted in the middle of a compliant answer could count as a refusal. A file detector no longer scores a file it never examined as a pass. A refusal-only adversarial detector stopped over-counting skipped multi-turn attempts. Safe-token suppression, which keeps a detector from flagging a credential the prompt itself supplied, is now scoped to its own match and, for API keys, to the matching credential, so a safe token no longer suppresses a different leaked one. An octal 27 byte is no longer treated as a raw ANSI escape. Each of these changes the hit count without any change in the model.

The consequence for anyone tracking a model over time is that scanner version is a variable in the experiment. A drop in attack success between two months may be a safer model or a stricter detector. Pin the scanner version per baseline, re-run the old model on the new version when upgrading, and record the detector list with every number. The same applies to probes: a fix that realigns prompt-injection settings with the sampled prompts, also in this release, changes what was asked, not only how it was scored.

Worked example: setting a gate for a support bot

In this synthetic example a team runs a prompt-injection probe family against a support assistant with ten generations per prompt across 120 prompts. Pooled attack success is 4 percent. hit@1 is also near 4 percent, which is expected because with equal generation counts hit@1 is the mean per-prompt hit fraction. hit@10, the always-on entry, is 23 percent: 28 of the 120 prompts were breached at least once in ten tries. The bootstrap interval on the pooled rate is tight because there are 1,200 generations; the interval that matters, on hit@10, is wider because it is over 120 prompts.

The gate the team adopts is three-fold. Pooled attack success below 5 percent, which they had before and which told them little. hit@10 below 10 percent, which they fail today and which reflects the real threat of a user who tries again. And no single prompt with c at or above half of n, which catches a reliable jailbreak that the averages hide. They pin garak to 0.17.1 in the pipeline, record the detector names in the report artefact, and re-baseline when they upgrade.

Failure modes worth testing

• Pooled rate as the headline. Understates persistent attackers. Report hit@n beside it.
• Comparing k across runs with different generation counts. hit@5 from a five-generation run and hit@5 from a twenty-generation run estimate the same quantity, but the always-on entry differs in k; read the key.
• Mixing pass rates and hit rates in a gate. The digest score is a pass rate; hit@k is a hit rate.
• Upgrading the scanner silently. Detector fixes change scores. Record the version and the detector list with every result.
• Sampling at temperature zero. With deterministic decoding all n generations agree and hit@k carries no information beyond hit@1. Sample as the attacker would.
• Too few prompts. Macro-averaging over 12 prompts gives a number with a wide interval. Use the bootstrap options, and respect the minimum sample size the tool documents.
• Excluded prompts. A prompt with fewer scoreable generations than k drops out of that k. Read the prompts count next to each score.

What to do next

1. Compute hit@1 and hit@n from an existing report with the function above, and compare them with the pooled rate you have been quoting.
2. Decide the attacker model: how many attempts does a realistic adversary get? Gate on that k.
3. Pin the scanner version, store the detector list and the generation count with every run.
4. When upgrading, re-run the previous baseline model on the new version before comparing anything.
5. Report a per-prompt worst case alongside the averages; a single reliable jailbreak is the finding that matters.

Practice and related reading

Further reading:

• garak, in depth: plugin model, scanning over REST, report.jsonl and a CI gate
• Building an AI red-team program
• Automated red teaming for AI systems
• A continuous red-team pipeline
• Red-team testing an ADK Java agent

Sources and scope

Checked against these sources on 19 October 2026. Examples and figures are illustrative, not measurements.

• NVIDIA/garak release v0.17.1, 2026-10-16
• NVIDIA/garak commit ab98be4: add hit@k attack success rate metric (#2090)
• garak documentation source: reporting.rst (hit@k attack success rate)
• Chen et al. 2021, Evaluating Large Language Models Trained on Code (the pass@k estimator)