Microsoft PyRIT's October 25 commit 27152dd810a7702ef0bf9bdc112d1c18f04c1a12 expands CredentialLeakScorer to recognize Azure SAS tokens, Azure Service Bus or Event Hub shared keys, and Databricks API tokens. This October 26 review is based on the pinned implementation and unit-test diff. It is a source change, not a claim about a newly published PyRIT release or a measured reduction in real incidents.
What changed in the detector
The patch adds the formats to both the default and optional expanded pattern sets. Its tests update the expected pattern counts from 15 to 18 and from 59 to 62. They cover positive and negative examples, token boundaries, synthetic exclusions and SAS query-parameter order. Those counts describe recognition patterns, not a benchmark score or a count of distinct attack techniques.
The engineering lesson is narrower and more useful than saying a scanner now supports more services. A detector must define what constitutes a match and which matched text an exclusion applies to. Otherwise the same synthetic fixture can become a false alert in one context and an accidental allowlist in another.
Query order should not decide whether a fixture matches
A signed storage URL puts credential-bearing information in query parameters. An application or URL builder may arrange those parameters differently. If a detector recognizes only one ordering, an ordinary serialization change can alter its apparent leak rate without changing the underlying exposure.
The patch tests permutations of four SAS parameters. Four distinct items have 24 possible orders; running those orders against two pattern configurations produces 48 parameterized configurations before considering the other assertions. This is arithmetic from the test definition, not a locally executed test result. The purpose is to avoid treating one hand-written URL as adequate coverage of a structured format.
Exclusions need an exact fixture identity
Red-team evaluations often need deliberate fake credentials. Suppose an original test fixture contains a synthetic signed URL and the harness is told to exclude that exact value. The same bytes appearing in the response should not count as evidence that the agent discovered a new secret. But a changed permission parameter should remain a distinct candidate match.
The pinned tests exercise this distinction by changing a permission value and checking exclusion behavior. An exclusion defined only by a friendly resource name or a broad prefix could hide more than the intended fixture. Store an explicit fixture identifier and its permitted matched representation. Review changes to exclusions as changes to the evaluation, since they alter what the score means.
Recognizing a shape does not establish a live credential
A regular expression finds text shaped like a token. It does not establish that the credential is valid, that its permissions are dangerous or that the model retrieved it from a protected source. A score can also miss transformed text, split values or a format outside its vocabulary. Treat these as limits of the detection method, not as proof that a passing response contains no sensitive information.
Keep the evaluation entirely synthetic. Do not send real secrets to a model to test whether they come back, and do not call a provider's authentication endpoint just to check whether a matched candidate works. Instead, record the detector version, fixture provenance, selected pattern set and exclusion set alongside each result. Compare runs only when those inputs are known.
Use a matrix rather than one happy-path example
An original acceptance plan has four axes: positive versus negative shape, plain versus surrounding text, ordinary versus reordered parameters, and excluded versus changed fixture. Add length boundaries and non-ASCII surrounding text where your users need them. A helpful negative fixture resembles a token enough to exercise a boundary but is explicitly invalid by the detector's intended format.
Review detection and response separately. The scorer can label a candidate leak; your application still has to decide what to redact, whether to retain the evidence and who may inspect it. Avoid placing matched token text in general logs. Keep restricted evidence references so an analyst can investigate without copying the sensitive value into every monitoring system.
Interpret score changes before declaring a regression
After an upgrade, a higher leak count may indicate broader recognition rather than a less safe agent. Re-score a fixed synthetic response corpus with the old and new detector revisions, then inspect which cases changed. Separate newly recognized formats from changes in generated behavior. No upstream suite or provider endpoint was run for this article; the result here is a reviewed source-level change and a concrete plan for making future evaluation comparisons meaningful.
Sources and evidence
Version detectors and exclusions with the evaluation corpus. Parameter-order and boundary tests reduce format blind spots, while a regex match remains evidence of token-shaped text rather than proof of a valid credential.