Implements #7980. Depends on #7982 (shared calibration module).
Problem
computeGateEval/auto-tune.ts's circuit breakers score precision AGGREGATED PER PROJECT — one bad deterministic rule (like the 2026-07-21/22 regex bug) can sit at effectively 0% precision while hiding inside an otherwise-healthy project-wide close-precision number, because it's diluted by every OTHER correct close reason. The breaker can never isolate and react to a single broken rule this way, even in principle, even without the CONCRETE_EVIDENCE_BLOCKER_CODES exemption (#7986) in the picture.
Area
Built on the shared module (#7982) and the ORB adapter's ground-truth data (pr_outcome + gate_decision blocker codes, already both recorded — this is a new aggregation dimension on EXISTING data, not a new collection pipeline).
Proposal
- Extend the gate-eval confusion matrix (
GateEvalRow) with a per-(project, blocker/rule code) breakdown, alongside the existing per-project rows — reusing gate_outcomes.blocker_codes_json / review_audit's existing gate_decision rows, which already carry the codes.
- Surface this breakdown wherever the aggregate is already surfaced (
computeTuningRecommendations, any operator dashboard) so a maintainer can see "rule X: 0/4 correct" even while the project aggregate looks fine.
- Available to AMS's own policy-code equivalents via the shared module.
Deliverables
- Per-rule
GateEvalRow-equivalent computation + tests, validated against the incident data (secret-or-credential/surface_lane_reject on metagraphed should show as a clear, isolated bad-precision rule).
- Surfaced in at least one existing operator-facing read path.
Resources
src/review/parity.ts (computeGateEval — the aggregate this extends)gate_outcomes.blocker_codes_json, review_audit gate_decision rows (existing data to re-aggregate)
Boundaries
Read/reporting only in this issue — #7986 is what actually changes breaker behavior using this new data.
maintainer-only.
Implements #7980. Depends on #7982 (shared calibration module).
Problem
computeGateEval/auto-tune.ts's circuit breakers score precision AGGREGATED PER PROJECT — one bad deterministic rule (like the 2026-07-21/22 regex bug) can sit at effectively 0% precision while hiding inside an otherwise-healthy project-wide close-precision number, because it's diluted by every OTHER correct close reason. The breaker can never isolate and react to a single broken rule this way, even in principle, even without theCONCRETE_EVIDENCE_BLOCKER_CODESexemption (#7986) in the picture.Area
Built on the shared module (#7982) and the ORB adapter's ground-truth data (
pr_outcome+gate_decisionblocker codes, already both recorded — this is a new aggregation dimension on EXISTING data, not a new collection pipeline).Proposal
GateEvalRow) with a per-(project, blocker/rule code) breakdown, alongside the existing per-project rows — reusinggate_outcomes.blocker_codes_json/review_audit's existinggate_decisionrows, which already carry the codes.computeTuningRecommendations, any operator dashboard) so a maintainer can see "rule X: 0/4 correct" even while the project aggregate looks fine.Deliverables
GateEvalRow-equivalent computation + tests, validated against the incident data (secret-or-credential/surface_lane_rejecton metagraphed should show as a clear, isolated bad-precision rule).Resources
src/review/parity.ts(computeGateEval— the aggregate this extends)gate_outcomes.blocker_codes_json,review_auditgate_decisionrows (existing data to re-aggregate)Boundaries
Read/reporting only in this issue — #7986 is what actually changes breaker behavior using this new data.
maintainer-only.