Skip to content

calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104

Description

@JSONbored

Context

src/rules/advisory.ts's isConfiguredGateBlocker(finding, policy) is the single function that decides whether ANY finding code — linked_issue_scope_mismatch, ai_consensus_defect, ai_review_split (together AI_JUDGMENT_BLOCKER_CODES), missing_linked_issue, duplicate_pr_risk, secret_leak, manifest_missing_tests, pre_merge_check_required, cla_check_unresolved, and the review-thread blocker code — actually blocks the gate. Its single caller, src/rules/advisory.ts around line 614 (const configuredBlockers = advisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective));), is the one place every one of those codes becomes a real, gate-authority-bearing decision. None of them currently record anything into the shared calibration module (#7982) — confirmed by grep, zero calls to createSignalStore(env).recordRuleFired anywhere in this codebase.

⚠️ Scope: every isConfiguredGateBlocker code EXCEPT linked_issue_scope_mismatch. #8101 already wires that specific code (fired + reversed) at its own upstream push site in src/queue/processors.ts, and has a PR in flight against it. This issue covers every OTHER code the function recognizes, wired generically at the shared configuredBlockers filter site instead of one bespoke PR per code. Do not also wire linked_issue_scope_mismatch here — recording it twice (once via #8101's site-specific hook, once via this issue's generic loop) would double-count its fired/reversed history. Explicitly skip any finding whose code === "linked_issue_scope_mismatch" in the loop described below.

Requirements

  • At the configuredBlockers computation in src/rules/advisory.ts (the advisoryResult.findings.filter((finding) => isConfiguredGateBlocker(finding, effective)) line), for every finding in the resulting configuredBlockers array whose code is not "linked_issue_scope_mismatch", call createSignalStore(env).recordRuleFired(...):
    {
      ruleId: finding.code,
      targetKey: `${repoFullName}#${prNumber}`,
      outcome: finding.severity ?? "blocker",
      occurredAt: nowIso(),
      ...(finding.confidence !== undefined ? { metadata: { confidence: finding.confidence } } : {}),
    }
    • The function containing this line will need env, repoFullName, and prNumber in scope. Read the function's current signature first (evaluateGateCheck/evaluateGateCheckCore or whichever function actually contains this line as of the current checkout — verify against live code, do not assume the name from this issue text) — if any of these three aren't already available at that exact point, add them as new parameters to that function rather than reaching for a module-level/global env, and thread them through from its existing callers.
    • Import createSignalStore from ../review/signal-tracking-wire; nowIso from ../utils/json.
    • Best-effort: wrap each call in .catch(() => undefined) — same fail-open discipline as Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS #7982's own SignalStore doc comment ("every write is best-effort... a failure to record a signal must never fail the review pass that produced it"). Recording failures must never affect configuredBlockers's own return value or any downstream gate decision.
  • In src/review/outcomes-wire.ts's recordReversalSignals, generalize the reversal hook: when a reversal is about to be recorded for a targetId (both the immediate contributor-reopen reversal_reopened path and the owner-reopen-then-merge path), check every currently-recognized blocker code EXCEPT linked_issue_scope_mismatch (already handled by calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own hook in the same function) for a prior fired event on that target, and record a "reversed" HumanOverrideEvent for each one that has one. Build the candidate code list from isConfiguredGateBlocker's own recognized codes minus that one exclusion (e.g. Array.from(AI_JUDGMENT_BLOCKER_CODES) plus the other literal codes that function checks, read directly from its current body so the list can never silently drift from what it actually recognizes). For each candidate code, call createSignalStore(env).queryRuleHistory(code, sinceMs) (a 30-day lookback in milliseconds is sufficient) and check whether any entry's targetKey matches targetId; if so, call recordHumanOverride({ ruleId: code, targetKey: targetId, verdict: "reversed", occurredAt: nowIso() }). Same best-effort .catch(() => undefined) discipline — must never affect whether the underlying reversal itself is recorded. If calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's own reversal hook has already landed by the time this is implemented, structure this as an addition alongside it (e.g. one shared loop over the full candidate list minus nothing, refactoring calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101's single-code check into this generalized one) rather than two separate, redundant lookups against the same targetId — read the current state of recordReversalSignals before starting, since calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 may already have changed it.
  • No "confirmed" override path in this issue. There is currently no existing signal anywhere in this codebase for "a maintainer explicitly confirmed an automated close/block was correct," for any rule. Tracked as its own separate, maintainer-only design issue under this epic (Design a positive-confirmation ('this automated decision was correct') signal #8106) — not folded into this mechanical wiring change.

Deliverables

  • recordRuleFired wired generically at the configuredBlockers computation (or its nearest caller with env/repo/PR context, per Requirements above) in src/rules/advisory.ts, excluding linked_issue_scope_mismatch.
  • recordHumanOverride (verdict "reversed") generalized in recordReversalSignals to check every currently-recognized blocker code except linked_issue_scope_mismatch for a prior fired event.
  • Tests covering: a fired event is recorded for every OTHER finding that ends up in configuredBlockers, verified independently for at least ai_consensus_defect, ai_review_split, and one deterministic code (e.g. secret_leak or missing_linked_issue); explicitly, NO fired event is recorded for linked_issue_scope_mismatch even when it's present in configuredBlockers (the exclusion is a required, separately-asserted test case, not an assumption); no fired event for a finding isConfiguredGateBlocker returns false for; a reversal is recorded against whichever non-excluded code(s) actually had a fired event for a reopened-then-merged PR, for more than one code in the same test run; no reversal recorded for a target with no prior fired event; both new write paths degrade silently (no thrown error, no effect on the surrounding function's normal return value) when the SignalStore call rejects.

Test Coverage Requirements

99%+ patch coverage (branch-counted), this repo's standard gate. Cover both new call sites, multiple distinct blocker codes (not just one), the linked_issue_scope_mismatch exclusion branch explicitly, and the fail-open .catch branches.

Expected Outcome

Every isConfiguredGateBlocker finding code except linked_issue_scope_mismatch (already covered by #8101) gets fired + reversed history in the shared calibration module automatically, with no per-code wiring PR required going forward for anything new added to that function. #7983/#7984's own eventual per-rule detection and this epic's backtest primitives (#8083-#8088) get coverage across the full set between this issue and #8101, with zero overlap.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions