Skip to content

fix(agent-actions): replace blanket concrete-evidence breaker exemption with per-rule track record - #8124

Merged
JSONbored merged 1 commit into
mainfrom
feat/per-rule-track-record-breaker-exemption
Jul 22, 2026
Merged

fix(agent-actions): replace blanket concrete-evidence breaker exemption with per-rule track record#8124
JSONbored merged 1 commit into
mainfrom
feat/per-rule-track-record-breaker-exemption

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • downgradeCloseToHold treated any close backed by a code in CONCRETE_EVIDENCE_BLOCKER_CODES as categorically immune to the close-precision circuit breaker, regardless of that specific rule's own measured accuracy — the deepest reason the self-correction system was structurally incapable of stopping the 2026-07-21/22 incident even with perfect reversal detection.
  • PlannedAgentAction now carries closeConcreteEvidenceCodes (the specific blocker code(s) that justified the concrete-evidence exemption), and downgradeCloseToHold checks those codes against a live, cron-refreshed per-rule close-precision track record (computeBlendedRuleGateEval / rulesBelowClosePrecisionFloor, Per-rule (not just per-project) precision tracking in the gate-eval pipeline #7984) cached via a new system_flags key (readUntrustworthyRuleCodes/writeUntrustworthyRuleCodes in runSelfTuneBreaker).
  • The per-rule check is independent of the project-level closeHoldOnly flag — a single systematically wrong rule can sit at 0% precision while diluted into an otherwise-healthy project aggregate, so it must not wait for the project flag to engage. A close's exemption is only lost when EVERY justifying code is below its own floor; a mix of a trustworthy and untrustworthy code keeps the exemption. Insufficient sample size still defaults to keeping the exemption (tightening-only, matching auto-tune.ts's design).
  • Wired through both live paths that apply the breaker: the webhook path (applyPrecisionBreakers in processors.ts) and the approval-queue accept path (decidePendingAgentAction in agent-approval-queue.ts).

Closes #7986.

Test plan

  • npx tsc --noEmit
  • New unit tests in test/unit/agent-actions.test.ts (closeConcreteEvidenceCodes population + the full downgradeCloseToHold per-rule matrix, including an incident-replay scenario and mixed-code exemption behavior)
  • New unit tests in test/unit/outcomes-wire.test.ts (readUntrustworthyRuleCodes fail-safe behavior + runSelfTuneBreaker's cache-write round-trip, including a write-failure swallow test)
  • npm run test:coverage (unsharded) — 100% line/branch coverage on every changed line, confirmed via targeted coverage-report inspection
  • npm run engine-parity:drift-check — clean, no twin-pair files touched

…on with per-rule track record

CONCRETE_EVIDENCE_BLOCKER_CODES membership alone made a heuristic close
categorically immune to the close-precision circuit breaker, regardless
of that specific rule's own measured accuracy. A single systematically
wrong rule can sit at 0% precision while diluted into an otherwise-
healthy project aggregate, so downgradeCloseToHold now also checks a
close's justifying code(s) against a live, cron-refreshed per-rule
track record (computeBlendedRuleGateEval) and drops the exemption only
when every justifying code is below its own close-precision floor.
Insufficient sample size still defaults to keeping the exemption.

Closes #7986.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 22, 2026
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.40%. Comparing base (01d2b5a) to head (a6796f1).
⚠️ Report is 9 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8124      +/-   ##
==========================================
- Coverage   92.01%   90.40%   -1.62%     
==========================================
  Files         758      101     -657     
  Lines       77304    26288   -51016     
  Branches    23372     5231   -18141     
==========================================
- Hits        71135    23765   -47370     
+ Misses       5061     2245    -2816     
+ Partials     1108      278     -830     
Flag Coverage Δ
shard-1 76.39% <61.29%> (+18.94%) ⬆️
shard-2 32.62% <77.41%> (-18.52%) ⬇️
shard-3 41.74% <61.29%> (-12.71%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/processors.ts 95.76% <100.00%> (+<0.01%) ⬆️
src/review/outcomes-wire.ts 90.35% <100.00%> (+0.68%) ⬆️
src/services/agent-approval-queue.ts 99.17% <100.00%> (ø)
src/settings/agent-actions.ts 98.06% <100.00%> (+0.06%) ⬆️

... and 657 files with indirect coverage changes

@JSONbored
JSONbored merged commit c02a277 into main Jul 22, 2026
11 of 12 checks passed
@JSONbored
JSONbored deleted the feat/per-rule-track-record-breaker-exemption branch July 22, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the blanket CONCRETE_EVIDENCE_BLOCKER_CODES breaker exemption with a per-rule track record

1 participant