Design a positive-confirmation ('this automated decision was correct') signal

[JSONbored]

Problem

Every backtest/precision system built or planned in this epic (and in #7980's own reactive self-correction family) can only ever see REVERSALS — a maintainer disagreeing with an automated call. Confirmed via grep: src/review/outcomes-wire.ts's recordReversalSignals has no positive-confirmation counterpart anywhere in this codebase. RulePrecisionReport.precision (packages/loopover-engine/src/calibration/signal-tracking.ts) and this epic's own BacktestScoreReport.precision/recall both, by design, treat "no override" as "no signal either way" — never coerced into a confirmed count. That's the right behavior given no real confirmation signal exists, but it means every precision number in this system is a floor, not a true value: it can only go up as reversals are found, never as correct calls get positively confirmed. At real scale this systematically undercounts precision for every rule, everywhere in the system.

Area

Cross-cutting — affects the ground-truth quality feeding #7982's calibration module, #7980's reactive detection, and this epic's own backtest primitives, wherever a "confirmed" count matters.

Proposal

Decide what a real confirmation signal should look like. Candidates to weigh (this issue does not decide between them):

• An explicit maintainer action — a bot-recognized comment or command on a PR that was automatically blocked/closed — highest-confidence, but requires new maintainer-facing UX and only captures cases the maintainer bothers to explicitly confirm.
• A time-boxed "stayed closed with no reopen" heuristic — cheap, no new UX, but conflicts with RulePrecisionReport's own stated discipline against absence-of-signal-as-signal: a maintainer might simply not have gotten around to reviewing a stale closed PR, not have confirmed it was correctly closed.
• Something derived from an existing maintainer action already captured elsewhere in the system (worth auditing what's already recorded — e.g. an explicit PR review approval/request-changes event — before inventing a new mechanism).
• A hybrid: the lightest-weight explicit action that still counts as a real, deliberate signal rather than an inferred one.

Deliverables

A written decision on which approach (or combination) to build, with reasoning — followed by a real implementation issue once decided.

Resources

• Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082 (parent epic); calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104 (the generic fired/reversed wiring this would complement)
• src/review/outcomes-wire.ts (recordReversalSignals — confirmed to have no confirmation counterpart)
• packages/loopover-engine/src/calibration/signal-tracking.ts (RulePrecisionReport.precision's own doc comment on the "absence of override is not a signal" discipline any solution here must not violate)

Boundaries

This issue is decision-only — it does not implement anything. Whatever gets decided becomes its own follow-up implementation issue.

maintainer-only — a real product-design decision affecting every rule's precision measurement, not a build task for contributors.

[linear-code]

JSO-5594

[JSONbored]

Decision: reuse the existing "held for manual review" mechanism as the confirmation signal

AgentActionPlanInput (src/settings/agent-actions.ts) already has three hold fields sharing the exact same shape — migrationCollisionHold, unlinkedIssueMatchHold, aiReviewLowConfidenceHold (each { reason: string; comment: string } | undefined) — every one routing a would-be-close PR into a manual-review hold instead, applying the manualReview label. resolveAiReviewLowConfidenceHold's own doc comment (src/rules/advisory.ts) confirms the intended semantics directly: "held for a maintainer to confirm instead of closing automatically... ask a maintainer to override." This is an existing, already-shipped "pending human confirmation" state — nothing currently reads or records what the maintainer actually does with it.

The decision: when a PR is held via one of these mechanisms (attributable to a specific ruleId's finding), and the repository OWNER's subsequent, deliberate action on that PR is to CLOSE it — reusing the exact owner-vs-contributor actor check recordReversalSignals already applies (ownerLogin === senderLogin, src/review/outcomes-wire.ts) — record a HumanOverrideEvent(verdict: "confirmed") for that ruleId.

This requires no new maintainer-facing UX (no bot command, no comment convention) — it's a smarter read of an event sequence that already happens today, reusing an actor-identity check already proven correct for the reversal side.

Explicitly not captured (an accepted, documented limitation, not something to fix here):

• A held PR simply left untouched generates no signal — consistent with this system's existing "absence of override is not itself a signal" discipline (see RulePrecisionReport.precision's own doc comment).
• An owner closing a held PR for an unrelated reason while it happens to also be in a held state is a real, accepted source of imprecision — a heuristic proxy, not perfect ground truth. Same class of imprecision the existing reversal signal already accepts.

Implementation filed as a separate follow-up issue, linked below.