calibration: evidence-weighted reviewer routing (flag-gated, report-first)

[JSONbored]

Problem

Epic #8211 track F capstone. With measured per-provider precision available, reviewer selection can prefer the provider that is measurably better for a given repo — but routing touches spend and review quality simultaneously, so it enters report-first.

Requirements

⚠️ Required pattern. Two stages inside one issue, second gated on the first's recorded evidence: (1) REPORT-ONLY — the orchestration path computes what routing WOULD have chosen per review and records it (audit metadata + a recap line), no behavior change; (2) behind a default-off flag, actual weighting with hard floors (never route away from consensus entirely; sample minimums per (provider, repo) before any preference applies; instant flag-off restore).

• Track-record read is fail-safe: missing/否-density data ⇒ existing behavior byte-identical.
• Spend note: routing must never increase per-review call count.

Deliverables

• Stage 1 shipped + 2 weeks of would-have-routed evidence posted here
• Stage 2 behind flag with 100% branch coverage and the restore path tested

Links & Resources

#8211 (epic), the track-records sub-issue (blocked-by), src/queue/ai-review-orchestration.ts

Boundaries

Default-off; consensus reviews keep both providers; no per-review spend increase.

maintainer-only — review-quality + spend authority.

[linear-code]

JSO-5713

[JSONbored]

Scoping finding before stage 1 ships: per-provider votes are not persisted anywhere today. #8228's pure module consumes ProviderReviewSignal[], but the orchestration's dual-review result is SLOT-based (ModelReview[]), with a deliberate reviewer-SWAP path (ai-review.ts ~1809) — so provider attribution must be threaded from the review plan through the combiner to the persistence point, or the track records will be built on misattributed votes (the same poison class as wrong corpus labels, and strictly worse than no data).

Concrete stage-0 prerequisite, now part of this issue's plan:

1. Vote capture writer at dual-review completion: one best-effort audit event per reviewer — reviewer_vote, actor = the PROVIDER ID resolved from the plan's slot mapping (swap-aware), targetKey, metadata {vote: fail|non_fail, repoFullName} — mirroring the calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101/calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104 capture discipline (never affects the verdict; both slots recorded even on consensus).
2. Swap-correctness invariant test: the same physical reviewer in normal slot 0 and swapped slot 1 must attribute to the same provider id.
3. Only then stage 1 as filed: a trailing-window reader assembling ProviderReviewSignal[] from those events + the labeled corpus → computeProviderTrackRecords → the would-have-routed audit metadata + recap line, zero behavior change.

Votes need to accumulate before would-have-routed means anything, so the capture writer should land ASAP even though the routing report follows later — will pick this up as the next maintainer slot after the in-flight gates (#8213/#8227) land.

[JSONbored]

Stage 1 seed: replay-derived provider track records (first same-seed multi-provider table)

Per #8278's acceleration path: the #8221 harness replayed two real providers over the identical seeded production sample (40 fixtures, seed counterfactual-replay-v1:production-first-comparison, minimal judge prompt, temperature 0), and the cached verdicts were fed through computeProviderTrackRecords (#8228) — the exact function live reviewer_vote rows will feed.

Provider	Signals	Decided	Precision (fail⇒confirmed)	Agreement	Consensus
qwen3-vl:8b-instruct	40	40	n/a	0.450	0.700
qwen3:8b	40	40	0.583	0.500	0.700

Reading: the two providers agree on 70% of shared targets, but behave differently where it matters — qwen3:8b flags defects with 0.583 precision against human labels, while qwen3-vl never cast a fail vote on this sample (precision n/a — the null-below-floor discipline, not zero), which also drags its agreement rate below coin-flip on a reversal-heavy sample. That's exactly the evidence-weighted-routing signal this issue exists to capture: identical prompt, identical fixtures, materially different judge behavior.

Caveats, per #8278's segregation rule: replay-derived, offline, never persisted as live votes — smoke-path 8B models, 40-fixture bounded sample. The live reviewer_vote capture (stage 0, deployed) accumulates the authoritative version; production BYOK models join the campaign behind explicit flags. Tooling: scripts/provider-replay-track-record.ts (landing with the #8277 PR).