Skip to content

Replace the blanket CONCRETE_EVIDENCE_BLOCKER_CODES breaker exemption with a per-rule track record #7986

Description

@JSONbored

Implements #7980. Depends on #7984 (per-rule precision tracking).

Problem

downgradeCloseToHold (src/settings/agent-actions.ts:561-563) treats any close backed by a code in CONCRETE_EVIDENCE_BLOCKER_CODES (including surface_lane_reject/secret-or-credential) as categorically immune to the close-precision circuit breaker, regardless of that specific rule's own track record. This is the deepest reason the self-correction system was structurally incapable of stopping the 2026-07-21/22 incident even in the best case (perfect reversal detection, full precision-over-time data) — "deterministic" was treated as synonymous with "trustworthy."

Area

src/settings/agent-actions.ts (hasConcreteCloseEvidence, downgradeCloseToHold). Depends on #7984's per-rule precision data existing first.

Proposal

  • Once per-rule precision (Per-rule (not just per-project) precision tracking in the gate-eval pipeline #7984) is available, gate the concrete-evidence exemption on that rule's OWN measured precision clearing a bar (e.g. the same AUTOTUNE_CLOSE_PRECISION_FLOOR), not on blanket code-list membership alone.
  • A rule with insufficient sample size defaults to keeping its exemption (fail toward today's behavior, not toward blocking everything on day one) — only a rule with enough decided samples AND bad precision loses its immunity and starts respecting an engaged closehold breaker.
  • Preserve the existing AI-judgment-vs-concrete-evidence distinction as the STARTING classification; this issue only adds a track-record override on top, it doesn't remove the distinction.

Deliverables

  • The gating change + tests, including a synthetic scenario mirroring the incident (a concrete-evidence code with 0% precision over a real sample) and asserting the breaker now engages for it specifically.
  • Confirmation no existing, healthy concrete-evidence rule (e.g. a well-functioning malformed-JSON check) gets spuriously held.

Resources

Boundaries

Tightening-only, matching auto-tune.ts's existing design principle — this can only make the breaker MORE willing to hold a bad rule back, never loosen anything. No change to the AI-judgment breaker's own existing logic.

maintainer-only.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions