fix(agent-actions): replace blanket concrete-evidence breaker exemption with per-rule track record - #8124
Merged
Conversation
…on with per-rule track record CONCRETE_EVIDENCE_BLOCKER_CODES membership alone made a heuristic close categorically immune to the close-precision circuit breaker, regardless of that specific rule's own measured accuracy. A single systematically wrong rule can sit at 0% precision while diluted into an otherwise- healthy project aggregate, so downgradeCloseToHold now also checks a close's justifying code(s) against a live, cron-refreshed per-rule track record (computeBlendedRuleGateEval) and drops the exemption only when every justifying code is below its own close-precision floor. Insufficient sample size still defaults to keeping the exemption. Closes #7986.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8124 +/- ##
==========================================
- Coverage 92.01% 90.40% -1.62%
==========================================
Files 758 101 -657
Lines 77304 26288 -51016
Branches 23372 5231 -18141
==========================================
- Hits 71135 23765 -47370
+ Misses 5061 2245 -2816
+ Partials 1108 278 -830
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
downgradeCloseToHoldtreated any close backed by a code inCONCRETE_EVIDENCE_BLOCKER_CODESas categorically immune to the close-precision circuit breaker, regardless of that specific rule's own measured accuracy — the deepest reason the self-correction system was structurally incapable of stopping the 2026-07-21/22 incident even with perfect reversal detection.PlannedAgentActionnow carriescloseConcreteEvidenceCodes(the specific blocker code(s) that justified the concrete-evidence exemption), anddowngradeCloseToHoldchecks those codes against a live, cron-refreshed per-rule close-precision track record (computeBlendedRuleGateEval/rulesBelowClosePrecisionFloor, Per-rule (not just per-project) precision tracking in the gate-eval pipeline #7984) cached via a newsystem_flagskey (readUntrustworthyRuleCodes/writeUntrustworthyRuleCodesinrunSelfTuneBreaker).closeHoldOnlyflag — a single systematically wrong rule can sit at 0% precision while diluted into an otherwise-healthy project aggregate, so it must not wait for the project flag to engage. A close's exemption is only lost when EVERY justifying code is below its own floor; a mix of a trustworthy and untrustworthy code keeps the exemption. Insufficient sample size still defaults to keeping the exemption (tightening-only, matchingauto-tune.ts's design).applyPrecisionBreakersinprocessors.ts) and the approval-queue accept path (decidePendingAgentActioninagent-approval-queue.ts).Closes #7986.
Test plan
npx tsc --noEmittest/unit/agent-actions.test.ts(closeConcreteEvidenceCodes population + the fulldowngradeCloseToHoldper-rule matrix, including an incident-replay scenario and mixed-code exemption behavior)test/unit/outcomes-wire.test.ts(readUntrustworthyRuleCodesfail-safe behavior +runSelfTuneBreaker's cache-write round-trip, including a write-failure swallow test)npm run test:coverage(unsharded) — 100% line/branch coverage on every changed line, confirmed via targeted coverage-report inspectionnpm run engine-parity:drift-check— clean, no twin-pair files touched