Skip to content

Read garak reports and line them up with RAGProbe by OWASP risk - #66

Merged
khushi-git-7 merged 2 commits into
mainfrom
feat/garak-report
Nov 12, 2026
Merged

khushi-git-7 merged 2 commits into
mainfrom
feat/garak-report

Conversation

@khushi-git-7

Copy link
Copy Markdown
Owner

Adds ragprobe security garak REPORT.jsonl: it reads a garak run's report and lines garak's results up with RAGProbe's, one row per OWASP risk.

garak tests the model; RAGProbe tests the app around it (documents, tools, defences). With --with reports/security.json, the two sit side by side, so the model layer and the app layer can be read in one table.

How garak's numbers are read

  • Only garak's own counts: a detector's fails are hits. Nothing is re-scored; nones stay out of the rate, as in garak.
  • A probe counts once, by its detector with the highest hit rate. garak runs several detectors over the same outputs, so adding them up would count each output more than once.
  • garak tags probes with OWASP v1.1 (2023); RAGProbe uses 2025, which renumbered the list (v1.1 LLM02 is 2025 LLM05, v1.1 LLM06 is 2025 LLM02, and so on). Each tag maps to the 2025 risk it became; v1.1's Insecure Plugin Design and Model Theft have no 2025 entry and keep their v1.1 names.
  • "not probed" is shown where a tool sent nothing for a risk, never as a pass.

Checked

  • On real garak 0.17.2 reports from a full probe sweep (the test.Repeat generator): web_injection gives 349/626 for LLM02 and LLM05, the same as summing each probe's worst detector by hand from the report's eval lines.
  • tests/test_garak_report.py: the v1.1 to 2025 mapping, a probe with two detectors counted once, an older report with only passed/total, a malformed line, a file with no results, the side-by-side table, and the command end to end. security garak joins the missing-file test in tests/test_cli_errors.py.
  • docs/garak.md explains the mapping and what the two columns can and can't be compared on; the README and CHANGELOG link it.

ragprobe security garak REPORT.jsonl reads a garak run's report, maps
its OWASP v1.1 tags to the 2025 risks, counts each probe once by its
worst detector, and with --with puts garak's model-level results beside
RAGProbe's app-level scorecard.
@khushi-git-7
khushi-git-7 merged commit 7ff7a9a into main Nov 12, 2026
6 checks passed
@khushi-git-7
khushi-git-7 deleted the feat/garak-report branch November 12, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant