AI evaluation integrity.
We break the benchmarks, judges and scoring pipelines AI is measured with, and attest the ones that hold.

Authensor is a security research firm in San Francisco, founded by John Daniel Kearney. It audits the instruments the AI industry measures itself with: evaluation harnesses, benchmarks, LLM judges, leaderboards and scoring pipelines.

Every defect we report is proven with a running reproduction against the genuine code at a pinned commit, and disclosed to the maintainer first. Fixed-scope audits end in an attestation with a scope, a standard version and an expiry date.

• The Evaluator Trust Boundary (ETB-01 to ETB-10). An evaluator has an ETB failure when its score depends on an artifact the evaluated system can influence. The page lists dated cases with upstream fixes in SWE-bench, Promptfoo, NVIDIA garak and the UK AI Security Institute's Inspect and ControlArena. Preprint: 10.5281/zenodo.21633620.
• etb-scan, an offline scanner that measures ETB-01 (injected verdicts) against your own judge. Standard library only, no API keys, MIT.

pip install etb-scan
etbscan --judge mypkg:judge --trials 10

• The ledger: every defect filed, every fix that landed upstream, and every target read and found sound. Includes the MLflow CVE-2026-64849 credit.

Evaluation-integrity audits, and a free claim check: one published number checked against the instrument that produced it, within 48 hours.

authensor.com · john@authensor.com

This repository first held Authensor's open-source agent-safety toolkit: the policy engine, the Aegis content scanner, the Sentinel monitor, SafeClaw and the @authensor npm packages. That product line is retired and no longer maintained. Its code and documentation stay here for reference:

Quickstart · Features · Red teaming · API reference · Deployment · Development

MIT.