Skip to content

calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101

Description

@JSONbored

Context

src/services/linked-issue-satisfaction.ts (the pure AI-backed analysis core) plus its orchestration in src/queue/processors.ts produce a real, live, AI-confidence-driven finding — code: "linked_issue_scope_mismatch" — that hard-blocks a PR when args.settings.linkedIssueSatisfactionGateMode === "block" and the assessed result.result.status === "unaddressed" (see the args.advisory.findings.push({ code: "linked_issue_scope_mismatch", ... }) call in processors.ts, guarded by exactly that condition; the comment immediately above it confirms advisory mode never pushes this finding at all).

Unlike every deterministic rule this epic's other sub-issues (#8083-#8088) were built to backtest, this finding has never been wired into the shared calibration module (#7982, packages/loopover-engine/src/calibration/signal-tracking.ts). Confirmed by grep: there are zero calls to createSignalStore(env).recordRuleFired anywhere in src/review/ or src/queue/ today — the ORB-side adapter (src/review/signal-tracking-wire.ts) exists but has no live caller yet. That means this specific, gate-authority-bearing AI judgment is currently invisible to both the reactive self-correction system (#7980/#7983/#7984/#7986) and this epic's own backtest primitives.

⚠️ Scope. This issue wires exactly one finding code (linked_issue_scope_mismatch) and exactly one override direction (reversal). It does NOT implement a "confirmed" override path — see Boundaries below for why.

Requirements

  • In src/queue/processors.ts, inside the existing if (args.settings.linkedIssueSatisfactionGateMode === "block" && result.result.status === "unaddressed") block (the one that pushes the linked_issue_scope_mismatch finding), also call createSignalStore(env).recordRuleFired(...) — import createSignalStore from ../review/signal-tracking-wire. Pass:
    {
      ruleId: "linked_issue_scope_mismatch",
      targetKey: `${args.repoFullName}#${args.pr.number}`,
      outcome: result.result.status,
      occurredAt: nowIso(),
      metadata: { confidence: result.result.confidence },
    }
    (nowIso from ../utils/json, matching signal-tracking-wire.ts's own usage.)
    • Wrap the call in .catch(() => undefined) — best-effort, matching the fail-open discipline the immediately-adjacent cache-write-error handling in this same function already uses, and SignalStore's own doc comment ("every write is best-effort... a failure to record a signal must never fail the review pass that produced it"). A recordRuleFired failure must never throw out of this code path.
    • Do not record anything outside this if block — no finding fires in advisory mode or for any status other than "unaddressed", so nothing should be recorded in those cases either.
  • In src/review/outcomes-wire.ts's recordReversalSignals, wire a reversal override: when this function is about to record a reversal for a targetId (both the immediate contributor-reopen reversal_reopened path, and the owner reopen-then-merge path from Count an owner reopen-then-merge as a real reversal signal, not administrative noise #7985), first check whether a linked_issue_scope_mismatch RuleFiredEvent exists for that same target — call createSignalStore(env).queryRuleHistory("linked_issue_scope_mismatch", sinceMs) (a lookback window of 30 days in milliseconds is sufficient; a stricter/looser window is an acceptable implementation choice as long as it is a fixed constant, not configurable in this PR) and check whether any entry's targetKey matches targetId. If it does, additionally call recordHumanOverride({ ruleId: "linked_issue_scope_mismatch", targetKey: targetId, verdict: "reversed", occurredAt: nowIso() }) on the same createSignalStore(env) instance. Same best-effort .catch(() => undefined) discipline as above — this must never affect whether the underlying reversal itself is recorded.

Deliverables

  • recordRuleFired call wired at the finding-push site in src/queue/processors.ts.
  • recordHumanOverride (verdict "reversed") wired into recordReversalSignals in src/review/outcomes-wire.ts, gated on a prior linked_issue_scope_mismatch fired event existing for the same target.
  • Tests extending the existing test file(s) for the touched functions, covering: a fired event is recorded exactly when the finding is pushed (mode "block" + status "unaddressed"); no fired event when mode is "advisory"/"off" or status is "addressed"/"partial"; a "reversed" override is recorded when a PR with a prior fired event is reopened-then-merged (both the immediate-contributor-reopen and owner-reopen-then-merge paths); no override recorded when a PR is reopened with no prior fired event for that target; both new write paths degrade silently (no thrown error, no effect on the surrounding function's normal return value) when the SignalStore call rejects.

Test Coverage Requirements

99%+ patch coverage (branch-counted), this repo's standard gate. Cover both new call sites and their fail-open .catch branches explicitly — a test that never exercises the SignalStore failure path does not satisfy this requirement.

Expected Outcome

linked_issue_scope_mismatch's fired + reversed history becomes real, queryable data in the shared calibration module — usable immediately by #7983/#7984 (once built) for reactive per-rule detection, and by this epic's own backtest primitives (#8083-#8086) for prospective validation of a proposed change to this specific AI judgment (e.g. a different confidence floor or prompt), with zero new scoring code required.

Links & Resources

Boundaries

No "confirmed" override path in this issue. There is currently no existing signal anywhere in this codebase for "a maintainer explicitly confirmed an automated close/block was correct," for this or any other rule (verified: recordReversalSignals only ever detects disagreement/reversal, never confirmation — there is no positive-confirmation precedent to mirror). Inventing one (a maintainer command, an explicit approval action, or any other new UX) is a product-design decision that belongs in a separate, maintainer-scoped issue, not folded into this mechanical wiring change. No change to linkedIssueSatisfactionGateMode's own blocking behavior — this issue only records data about decisions already being made, it does not change what gets blocked.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions