A statistical release gate for ML models — CI for model quality.
Real artifacts rendered by the harness in CI on the latest commit: a 🟢 passing gate report, a 🔴 failing gate report (deliberately sabotaged run — the gate blocks it with exit 1), the 📈 trend chart with gate-colored markers, and the machine-readable gate JSON. Rebuilt on every push by the pages workflow.
vision-eval-harness audits machine-learning model quality the way CI audits code. It runs
evaluation suites against local vision classifiers and LLM endpoints, computes accuracy /
consistency / drift metrics, ingests large inference logs through a parallel ETL pipeline, and
enforces a statistically rigorous release gate that blocks merges when a candidate model is
genuinely worse than the promoted baseline — not just noisily different.
No server, no dashboard, no cloud dependency. The product is a CLI + a migration-versioned SQLite database + report artifacts. Everything runs offline, including the entire test suite.
Comparing two accuracy numbers and eyeballing "0.93 vs 0.95, ship it" is how silent regressions reach production. This harness replaces that with:
- Paired exact McNemar tests — per-sample pairing removes dataset variance, giving real statistical power at n≈60 where naive proportion comparisons see nothing.
- Dual criteria — a regression must be statistically significant and practically meaningful to fail the gate. Large-n trivia and small-n noise both stay out of your way.
- Benjamini–Hochberg correction — gate ten suites at α=0.05 naively and you false-alarm ~40% of the time; BH controls the false-discovery rate across the family.
- Tamper-evident baselines — the reference model's results live in a checksummed golden file committed to the repo, so stateless CI runners gate against a reviewed, verifiable truth.
- Exit-code discipline —
0pass,1gate failure (merge blocker),2harness error. CI branches on it; humans read the check table.
flowchart TD
CLI["CLI (Typer) — harness\ninit-db · run · runs · init-logs · pipeline · drift\nbaseline · gate · report · events"]
CLI --> RUN["Suite Runners\nvision (sync) · llm (asyncio)"]
CLI --> ETL["Transcript ETL\ningest · bench · stats · quarantine"]
CLI --> GATE["Gate Policy Engine\nCHECKS registry + BH correction"]
CLI --> REP["Reporting\nSVG charts · Jinja2 HTML"]
RUN --> VA["VISION_ADAPTERS registry\ndummy_color · onnx"]
RUN --> LLM["Async LLM stack\nAsyncLLMClient · TokenBucket\nSQLite response cache"]
RUN --> GR["GRADERS registry\nexact · contains · regex\nnumeric · json_subset"]
ETL --> WORK["ProcessPoolExecutor\npure parse workers"]
GATE --> STATS["gate.stats\nMcNemar · z-test · Mann-Whitney\nbootstrap CI · Benjamini-Hochberg"]
GATE --> BASE["Baseline lifecycle\npromote · export · import\nrollback · sha256 checksums"]
VA --> DB[("SQLite harness.db\n9 numbered migrations\nruns · results · metrics · transcripts\nquarantine · llm_cache · baselines\ngate_decisions · run_events")]
LLM --> DB
GR --> DB
WORK --> DB
STATS --> DB
BASE --> DB
REP --> DB
DB --> ART["./artifacts\ngate-*.json · report-*.html · trend-*.html"]
Dependency rule: cli → {runners, pipeline, gate, report} → {models, grading, metrics, datasets, obs} → {registry, db, config, errors} — dependencies point strictly downward, never
sideways or upward. One generic Registry[T] powers three extension points (vision adapters,
graders, gate checks): adding any of them is one registered function, zero caller changes.
flowchart LR
A["candidate run\n(latest completed)"] --> C{Gate}
B["promoted baseline\n(from checksummed\ngolden file)"] --> C
C --> D["policy\nexplicit gate: YAML block\nor synthesized from thresholds"]
D --> E["7 checks via CHECKS registry\nMcNemar regression · accuracy floor\npractical drop · score PSI · consistency\nMann-Whitney latency · bootstrap CI"]
E --> F{"multi-suite?"}
F -- yes --> G["Benjamini-Hochberg\nover regression p-values"]
F -- no --> H["verdict"]
G --> H
H --> I["gate_decisions row\n+ artifacts/gate-*.json"]
H --> J(["exit 0 = pass\nexit 1 = BLOCK MERGE\nexit 2 = error"])
sequenceDiagram
participant Dev as Developer
participant DB as harness.db
participant File as baselines/vision_smoke.json
participant CI as CI runner (stateless)
Dev->>DB: harness run --suite vision_smoke
Dev->>DB: harness baseline promote latest
Dev->>File: harness baseline export (sha256 checksum embedded)
Dev->>File: git commit — reviewed golden file
CI->>File: harness baseline import (checksum verified)
CI->>DB: harness run (candidate) → harness gate
Note over CI: hand-edited or corrupted baseline → import refuses, exit 2
git clone git@github.com:Jenil133/Automated-Vision-Model-Validation-Harness.git
cd Automated-Vision-Model-Validation-Harness
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
harness db migrate # apply schema migrations 1-9
harness init-data --seed 42 # deterministic synthetic image dataset (offline)
harness run --suite vision_smoke # evaluate → runs/results/metrics in harness.db
harness baseline promote latest --suite vision_smoke
harness baseline export --suite vision_smoke -o baselines/vision_smoke.json
harness run --suite vision_smoke # a second candidate run
harness gate --suite vision_smoke # → GATE PASSED, exit 0
harness report run latest --suite vision_smoke # HTML report in ./artifacts
harness report trend --suite vision_smoke # accuracy + p95 trend chartsManufacture a regression and watch the gate catch it:
harness init-data --seed 42 --brightness 3.0 # saturates channels → model breaks
harness run --suite vision_smoke
harness gate --suite vision_smoke; echo "exit=$?" # FAIL rows, GATE FAILED, exit=1| Command | What it does | Exit |
|---|---|---|
harness db migrate / db status |
Apply / inspect numbered schema migrations | 0 / 2 |
harness init-data |
Seeded synthetic image dataset (--brightness, --mislabel-rate) |
0 / 2 |
harness init-logs |
Seeded synthetic transcript logs (--drift-file for the drift demo) |
0 / 2 |
harness run --suite S |
Run a vision or LLM suite (--limit, --git-sha, --no-cache) |
0 / 2 |
harness runs list / runs show ID |
Inspect past runs (12-char id prefixes accepted) | 0 / 2 |
harness pipeline ingest [PATHS…] |
Incremental, quarantining, parallel log ingest | 0 / 2 |
harness pipeline bench |
Serial vs parallel parse benchmark (warmup + medians) | 0 / 2 |
harness pipeline stats / quarantine |
Per-model aggregates / dead-letter viewer | 0 / 2 |
harness drift runs / drift transcripts |
PSI + KS + χ² between runs / across a time split | 0 / 2 |
harness baseline promote|show|rollback|export|import|verify |
Audited baseline lifecycle | 0 / 2 |
harness gate --suite S…|--all |
The statistical release gate | 0 / 1 / 2 |
harness report run|trend |
Self-contained HTML reports with SVG charts | 0 / 2 |
harness events |
Query the structured run_events audit trail |
0 / 2 |
global --verbose / --log-json |
Debug logging / one-JSON-per-line logs | — |
Suites are declarative YAML in configs/suites/. Two kinds:
Vision (vision_smoke.yaml) — local classifiers behind the VISION_ADAPTERS registry.
Ships with a deterministic dummy_color adapter (zero downloads, CI-safe) and a config-driven
ONNX adapter (
LLM (llm_smoke.yaml) — any OpenAI-compatible /chat/completions endpoint, driven by an
asyncio stack: bounded concurrency (Semaphore), client-side token-bucket rate limiting,
retries with jittered backoff, and a persistent SQLite response cache — re-running an
unchanged suite costs zero API calls and reports cache_hit_rate = 1.0.
name: llm_smoke
kind: llm
model:
adapter: openai_compat
model_id: gpt-4o-mini
base_url: ${LLM_BASE_URL:-https://api.openai.com/v1} # env interpolation
temperature: 0.0
dataset:
prompts_jsonl: data/prompts/llm_smoke.jsonl # graded by the GRADERS registry
repeats: 3 # repeats power the consistency metrics
concurrency: 8
rate_limit_rps: 4
cache: true
thresholds:
min_accuracy: 0.90
max_accuracy_drop: 0.02
alpha: 0.05
max_psi: 0.25
min_consistency: 0.80Environment: LLM_API_KEY (never in YAML; warns and sends a placeholder if unset — fine for
local servers like Ollama) and LLM_BASE_URL via ${VAR:-default} interpolation. Live
endpoints are only needed to run llm_smoke — every test is offline via httpx.MockTransport.
Policies come from an explicit gate: block in the suite YAML, or are synthesized from
thresholds so every suite gates without extra configuration. All comparative checks
auto-skip (informational) when there's no baseline or too little data — the gate never
fabricates significance it doesn't have.
| Check | Method | Fails when |
|---|---|---|
min_accuracy |
absolute floor | candidate accuracy < floor |
accuracy_regression |
paired exact McNemar (unpaired z fallback) | drop > 0 and p < α |
max_accuracy_drop |
practical limit | drop > limit (significant or not) |
score_psi |
Population Stability Index, baseline-referenced bins | PSI > max_psi |
min_consistency |
floor on consistency_rate |
consistency < floor |
latency_regression |
Mann-Whitney U + p95 delta | significant and p95 worse by > X% |
bootstrap_ci |
seeded percentile bootstrap on the accuracy delta | never — informational |
With multiple suites (--all or repeated --suite), the regression p-values form one family
under Benjamini–Hochberg; an improving suite can never fail on BH (the drop > 0 conjunct is
load-bearing). Every decision persists to gate_decisions and artifacts/gate-<id>.json — a
stable machine-readable contract.
Multi-GB inference logs → queryable SQLite, built for repeated runs:
- Incremental — a sha256 manifest skips unchanged completed files; changed or forced files are reprocessed atomically (stale rows deleted in the same transaction).
- Quarantining — malformed lines are never dropped: each becomes a dead-letter row with a
reason code (
invalid_json,missing_field:<f>,bad_type:<f>) so a fixed file re-ingests clean. - Parallel — CPU-bound JSON parsing fans out over a
ProcessPoolExecutorof pure workers; only the parent touches SQLite.pipeline benchmeasures the speedup honestly (warmup pass, median of repetitions) — typically 30–70% wall-time reduction on 4+ cores. - Crash-safe — a failure mid-file marks the manifest
failed; the next ingest reprocesses it.
Drift analytics run on top: PSI + KS on scores/latencies between runs, χ² on predicted-label
mixes, and a temporal split over transcript latencies (drift transcripts --split <ISO>).
Runners, ingest, and the gate emit structured events (run_started, run_finished,
file_ingested, gate_decided) into a queryable run_events table — harness events reads
it, --log-json mirrors logs as JSON lines for machine ingestion.
report run renders a self-contained HTML page: metrics, gate verdict with the full check
table, a latency histogram, and the worst samples. report trend charts accuracy and p95
latency across runs with gate-colored markers. Charts are hand-rolled SVG — zero chart
dependencies, deterministic output, and every DB-sourced string renders through Jinja2
autoescape (model output is untrusted; reports must not be an XSS vector).
.github/workflows/eval-gate.yml runs on every PR:
- checks (Python 3.11 + 3.12 matrix):
ruff→mypy→pytest(190 tests, offline) - gate: fresh DB →
baseline import(checksum-verified golden file) → seeded candidate run →harness gate— exit 1 fails the job and blocks the merge - Gate JSON + HTML report upload as artifacts even when the gate fails (
if: always())
No secrets required — the gated suite is fully offline. To make it binding: repo Settings →
Branches → protect main → require the gate job as a status check.
A PR's gate went red.
- Download the
gate-artifactsfrom the run (or readartifacts/gate-*.json) — which check failed, with what value vs threshold? harness report run latest --suite <S>— the worst-samples table usually names the damage.- Real regression → fix the model/code. Intentional change → refresh the baseline in the
same PR:
run → baseline promote latest → baseline export → commit the JSON diff. The reviewer sees exactly what the new reference behavior is.
A bad baseline got promoted. harness baseline rollback --suite <S> — reactivates the
previous baseline from history; baseline show --history shows the audit trail.
Someone edited the golden file by hand. Import refuses (checksum mismatch, exit 2).
Regenerate it from a real run; never hand-edit — that's the point.
run_events grows forever. By design. Prune with
DELETE FROM run_events WHERE ts < '<cutoff>' on your own schedule.
src/harness/
├── cli.py # Typer CLI — arg parsing + exit-code mapping only
├── config.py # env-interpolated YAML → validated Pydantic models
├── registry.py # generic Registry[T] — adapters, graders, gate checks
├── db/ # the ONLY SQL speaker: DAL + 9 numbered migrations + facade
├── datasets.py # image + prompt loaders, fail-fast validation
├── models/ # vision adapters + async LLM stack (client/limiter/cache)
├── grading/ # 5 pluggable graders behind grade()
├── runners/ # vision (sync) + llm (asyncio) suite orchestration
├── pipeline/ # synthetic logs, parallel ingest, bench, aggregates
├── metrics/ # accuracy, consistency, drift (PSI/KS/χ²)
├── gate/ # stats, checks, policy, engine, baseline_io
├── report/ # SVG primitives + Jinja2 HTML reports
└── obs/ # structured events + logging setup
Testing: 190 tests, 96% line coverage, entirely offline — unit, CLI-contract (exit codes through the real entry point), failure-injection (corrupt inputs, transport faults, mid-pipeline crashes), and property-based tests (hypothesis) proving the statistical invariants.
- SQLite over Postgres — single-node local-first is the product shape; the
db/facade makes a swap a one-package change. Single-writer discipline is documented, not hidden. - Registry pattern over if/elif chains — one mechanism, three extension points; adding an adapter/grader/check never touches callers.
- Client-side rate limiting over "let the server 429 us" — polite, retry-storm-free, and keeps latency metrics honest.
- Cache freezes stochasticity — a fully cached rerun replays the first run's answers, so
consistency metrics reflect that run's diversity;
--no-cachere-measures fresh (andcache_hit_ratemakes it visible). - Forward-only migrations — no
down(); nine versions and counting, adopt-in-place from any earlier phase.
Concurrent harness processes on one DB will hit SQLite locks. BH corrects only the
accuracy-regression family (mixing heterogeneous tests would muddy FDR semantics). Bootstrap
CIs are percentile, not BCa — fine for reporting, upgrade before making them decision inputs.
Future: more adapters (real ONNX zoo models, more LLM providers), a live dashboard, PyPI
publication.
1.0.0 — see CHANGELOG.md. Built in three phases: foundation (config, schema, vertical slice) → engine (registries, async LLM stack, ETL, drift) → teeth (statistical gate, baseline lifecycle, observability, reports, CI).