You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104
src/rules/advisory.ts's isConfiguredGateBlocker(finding, policy) is the single function that decides whether ANY finding code — linked_issue_scope_mismatch, ai_consensus_defect, ai_review_split (together AI_JUDGMENT_BLOCKER_CODES), missing_linked_issue, duplicate_pr_risk, secret_leak, manifest_missing_tests, pre_merge_check_required, cla_check_unresolved, and the review-thread blocker code — actually blocks the gate. Its single caller, src/rules/advisory.ts around line 614 (const configuredBlockers = advisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective));), is the one place every one of those codes becomes a real, gate-authority-bearing decision. None of them currently record anything into the shared calibration module (#7982) — confirmed by grep, zero calls to createSignalStore(env).recordRuleFired anywhere in this codebase.
⚠️ Scope: every isConfiguredGateBlocker code EXCEPT linked_issue_scope_mismatch. #8101 already wires that specific code (fired + reversed) at its own upstream push site in src/queue/processors.ts, and has a PR in flight against it. This issue covers every OTHER code the function recognizes, wired generically at the shared configuredBlockers filter site instead of one bespoke PR per code. Do not also wire linked_issue_scope_mismatch here — recording it twice (once via #8101's site-specific hook, once via this issue's generic loop) would double-count its fired/reversed history. Explicitly skip any finding whose code === "linked_issue_scope_mismatch" in the loop described below.
Requirements
At the configuredBlockers computation in src/rules/advisory.ts (the advisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective)) line), for every finding in the resulting configuredBlockers array whose code is not "linked_issue_scope_mismatch", call createSignalStore(env).recordRuleFired(...):
The function containing this line will need env, repoFullName, and prNumber in scope. Read the function's current signature first (evaluateGateCheck/evaluateGateCheckCore or whichever function actually contains this line as of the current checkout — verify against live code, do not assume the name from this issue text) — if any of these three aren't already available at that exact point, add them as new parameters to that function rather than reaching for a module-level/global env, and thread them through from its existing callers.
Import createSignalStore from ../review/signal-tracking-wire; nowIso from ../utils/json.
Best-effort: wrap each call in .catch(() => undefined) — same fail-open discipline as Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS #7982's own SignalStore doc comment ("every write is best-effort... a failure to record a signal must never fail the review pass that produced it"). Recording failures must never affect configuredBlockers's own return value or any downstream gate decision.
In src/review/outcomes-wire.ts's recordReversalSignals, generalize the reversal hook: when a reversal is about to be recorded for a targetId (both the immediate contributor-reopen reversal_reopened path and the owner-reopen-then-merge path), check every currently-recognized blocker code EXCEPT linked_issue_scope_mismatch (already handled by calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own hook in the same function) for a prior fired event on that target, and record a "reversed"HumanOverrideEvent for each one that has one. Build the candidate code list from isConfiguredGateBlocker's own recognized codes minus that one exclusion (e.g. Array.from(AI_JUDGMENT_BLOCKER_CODES) plus the other literal codes that function checks, read directly from its current body so the list can never silently drift from what it actually recognizes). For each candidate code, call createSignalStore(env).queryRuleHistory(code, sinceMs) (a 30-day lookback in milliseconds is sufficient) and check whether any entry's targetKey matches targetId; if so, call recordHumanOverride({ ruleId: code, targetKey: targetId, verdict: "reversed", occurredAt: nowIso() }). Same best-effort .catch(() => undefined) discipline — must never affect whether the underlying reversal itself is recorded. If calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own reversal hook has already landed by the time this is implemented, structure this as an addition alongside it (e.g. one shared loop over the full candidate list minus nothing, refactoring calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's single-code check into this generalized one) rather than two separate, redundant lookups against the same targetId — read the current state of recordReversalSignals before starting, since calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 may already have changed it.
No "confirmed" override path in this issue. There is currently no existing signal anywhere in this codebase for "a maintainer explicitly confirmed an automated close/block was correct," for any rule. Tracked as its own separate, maintainer-only design issue under this epic (Design a positive-confirmation ('this automated decision was correct') signal #8106) — not folded into this mechanical wiring change.
Deliverables
recordRuleFired wired generically at the configuredBlockers computation (or its nearest caller with env/repo/PR context, per Requirements above) in src/rules/advisory.ts, excluding linked_issue_scope_mismatch.
recordHumanOverride (verdict "reversed") generalized in recordReversalSignals to check every currently-recognized blocker code except linked_issue_scope_mismatch for a prior fired event.
Tests covering: a fired event is recorded for every OTHER finding that ends up in configuredBlockers, verified independently for at least ai_consensus_defect, ai_review_split, and one deterministic code (e.g. secret_leak or missing_linked_issue); explicitly, NO fired event is recorded for linked_issue_scope_mismatch even when it's present in configuredBlockers (the exclusion is a required, separately-asserted test case, not an assumption); no fired event for a finding isConfiguredGateBlocker returns false for; a reversal is recorded against whichever non-excluded code(s) actually had a fired event for a reopened-then-merged PR, for more than one code in the same test run; no reversal recorded for a target with no prior fired event; both new write paths degrade silently (no thrown error, no effect on the surrounding function's normal return value) when the SignalStore call rejects.
Test Coverage Requirements
99%+ patch coverage (branch-counted), this repo's standard gate. Cover both new call sites, multiple distinct blocker codes (not just one), the linked_issue_scope_mismatch exclusion branch explicitly, and the fail-open .catch branches.
Expected Outcome
Every isConfiguredGateBlocker finding code except linked_issue_scope_mismatch (already covered by #8101) gets fired + reversed history in the shared calibration module automatically, with no per-code wiring PR required going forward for anything new added to that function. #7983/#7984's own eventual per-rule detection and this epic's backtest primitives (#8083-#8088) get coverage across the full set between this issue and #8101, with zero overlap.
src/rules/advisory.ts (isConfiguredGateBlocker, AI_JUDGMENT_BLOCKER_CODES, the configuredBlockers computation — read the full function before starting)
src/review/signal-tracking-wire.ts (createSignalStore, the adapter to call — already built, do not create a second one)
Context
src/rules/advisory.ts'sisConfiguredGateBlocker(finding, policy)is the single function that decides whether ANY finding code —linked_issue_scope_mismatch,ai_consensus_defect,ai_review_split(togetherAI_JUDGMENT_BLOCKER_CODES),missing_linked_issue,duplicate_pr_risk,secret_leak,manifest_missing_tests,pre_merge_check_required,cla_check_unresolved, and the review-thread blocker code — actually blocks the gate. Its single caller,src/rules/advisory.tsaround line 614 (const configuredBlockers = advisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective));), is the one place every one of those codes becomes a real, gate-authority-bearing decision. None of them currently record anything into the shared calibration module (#7982) — confirmed by grep, zero calls tocreateSignalStore(env).recordRuleFiredanywhere in this codebase.Requirements
configuredBlockerscomputation insrc/rules/advisory.ts(theadvisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective))line), for everyfindingin the resultingconfiguredBlockersarray whosecodeis not"linked_issue_scope_mismatch", callcreateSignalStore(env).recordRuleFired(...):env,repoFullName, andprNumberin scope. Read the function's current signature first (evaluateGateCheck/evaluateGateCheckCoreor whichever function actually contains this line as of the current checkout — verify against live code, do not assume the name from this issue text) — if any of these three aren't already available at that exact point, add them as new parameters to that function rather than reaching for a module-level/globalenv, and thread them through from its existing callers.createSignalStorefrom../review/signal-tracking-wire;nowIsofrom../utils/json..catch(() => undefined)— same fail-open discipline as Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS #7982's ownSignalStoredoc comment ("every write is best-effort... a failure to record a signal must never fail the review pass that produced it"). Recording failures must never affectconfiguredBlockers's own return value or any downstream gate decision.src/review/outcomes-wire.ts'srecordReversalSignals, generalize the reversal hook: when a reversal is about to be recorded for atargetId(both the immediate contributor-reopenreversal_reopenedpath and the owner-reopen-then-merge path), check every currently-recognized blocker code EXCEPTlinked_issue_scope_mismatch(already handled by calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own hook in the same function) for a prior fired event on that target, and record a"reversed"HumanOverrideEventfor each one that has one. Build the candidate code list fromisConfiguredGateBlocker's own recognized codes minus that one exclusion (e.g.Array.from(AI_JUDGMENT_BLOCKER_CODES)plus the other literal codes that function checks, read directly from its current body so the list can never silently drift from what it actually recognizes). For each candidate code, callcreateSignalStore(env).queryRuleHistory(code, sinceMs)(a 30-day lookback in milliseconds is sufficient) and check whether any entry'stargetKeymatchestargetId; if so, callrecordHumanOverride({ ruleId: code, targetKey: targetId, verdict: "reversed", occurredAt: nowIso() }). Same best-effort.catch(() => undefined)discipline — must never affect whether the underlying reversal itself is recorded. If calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own reversal hook has already landed by the time this is implemented, structure this as an addition alongside it (e.g. one shared loop over the full candidate list minus nothing, refactoring calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's single-code check into this generalized one) rather than two separate, redundant lookups against the sametargetId— read the current state ofrecordReversalSignalsbefore starting, since calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 may already have changed it.Deliverables
recordRuleFiredwired generically at theconfiguredBlockerscomputation (or its nearest caller withenv/repo/PR context, per Requirements above) insrc/rules/advisory.ts, excludinglinked_issue_scope_mismatch.recordHumanOverride(verdict"reversed") generalized inrecordReversalSignalsto check every currently-recognized blocker code exceptlinked_issue_scope_mismatchfor a prior fired event.configuredBlockers, verified independently for at leastai_consensus_defect,ai_review_split, and one deterministic code (e.g.secret_leakormissing_linked_issue); explicitly, NO fired event is recorded forlinked_issue_scope_mismatcheven when it's present inconfiguredBlockers(the exclusion is a required, separately-asserted test case, not an assumption); no fired event for a findingisConfiguredGateBlockerreturnsfalsefor; a reversal is recorded against whichever non-excluded code(s) actually had a fired event for a reopened-then-merged PR, for more than one code in the same test run; no reversal recorded for a target with no prior fired event; both new write paths degrade silently (no thrown error, no effect on the surrounding function's normal return value) when theSignalStorecall rejects.Test Coverage Requirements
99%+ patch coverage (branch-counted), this repo's standard gate. Cover both new call sites, multiple distinct blocker codes (not just one), the
linked_issue_scope_mismatchexclusion branch explicitly, and the fail-open.catchbranches.Expected Outcome
Every
isConfiguredGateBlockerfinding code exceptlinked_issue_scope_mismatch(already covered by #8101) gets fired + reversed history in the shared calibration module automatically, with no per-code wiring PR required going forward for anything new added to that function. #7983/#7984's own eventual per-rule detection and this epic's backtest primitives (#8083-#8088) get coverage across the full set between this issue and #8101, with zero overlap.Links & Resources
linked_issue_scope_mismatchspecifically)src/rules/advisory.ts(isConfiguredGateBlocker,AI_JUDGMENT_BLOCKER_CODES, theconfiguredBlockerscomputation — read the full function before starting)src/review/signal-tracking-wire.ts(createSignalStore, the adapter to call — already built, do not create a second one)src/review/outcomes-wire.ts(recordReversalSignals, and Count an owner reopen-then-merge as a real reversal signal, not administrative noise #7985's own owner-reopen-then-merge design to mirror the target-matching logic against — check its current state, calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 may have already touched it)