Repository navigation
ApiKey detector safe token masks a separate credential match #2172
Description
ApiKey detector safe token masks a separate credential match
Steps to reproduce
Run the following against main at commit 3f50ea5ff9cd7050099940647c15c39b07a93392:
from garak.attempt import Attempt, Message
from garak.detectors.apikey import ApiKey
attempt = Attempt(prompt=Message(text="audit prompt"))
attempt.outputs = [
"The AWS access key is AKIAIOSFODNN7EXAMPLE. "
"A safe MongoDB example is "
"mongodb+srv://myuser:mypassword@mycluster.mongodb.net/mydatabase."
]
print(ApiKey().detect(attempt))This prints:
[0.0]
The AWS key sample by itself produces [1.0], and the MongoDB example by itself produces [0.0] as expected.
Expected behavior
The safe placeholder should suppress only the credential-pattern match that contains mypassword. A separate API key match in the same output should still produce [1.0].
Current behavior
After any regex match sets key_found = True, the detector searches the entire output for each safe token and resets key_found = False. Consequently, a safe placeholder anywhere in the output masks every other credential match.
garak version
0.17.1.pre1, source commit 3f50ea5ff9cd7050099940647c15c39b07a93392.
Additional Information
- Operating system: Windows
- Python version: 3.12.5
- Install method: editable source checkout
- No target, model, API, GPU, or network access is required
- A focused regression test reproduces the issue deterministically
I found no open issue or pull request describing this mixed-output behavior. Open PR #1907 covers tests for the API key probe and does not modify this detector or its tests.
Activity
fix(detectors): scope safe-token suppression to its own match
test(detectors): cover the safe-token scope matrix reported in NVIDIA…
MRX-72 commentedon Oct 4, 2026
I've opened #2270 with a fix for this.
Root cause: the
SAFE_TOKENSpass runs against the whole output and resets the singlekey_foundflag, so one safe placeholder masks every other match. The fix scopes the safe-token check to each individual match (match.group(0)), so a safe example only neutralizes its own match. Added a regression test (real AWS key + themypasswordmongodb example now correctly scores[1.0]); the 15 existing detector tests still pass andblackis clean.It's currently a draft — happy to adjust the approach if you'd prefer the safe-token handling live elsewhere, or if this is out of scope. Could a maintainer confirm it's a good fit?
Scope ApiKey safe-token suppression to the matching credential