feat(calibration): shared ORB+AMS signal-tracking module, with AMS wired live

packages/loopover-engine/src/calibration/signal-tracking.ts

@@ -0,0 +1,113 @@
+// Shared deterministic-rule signal tracking (#7982) -- the deployment-agnostic primitive both ORB's gate
+// blockers and AMS's eligibility/policy heuristics record through, so a systematically-wrong rule can be
+// detected the same way in both subsystems instead of ORB alone having a self-correction story.
+//
+// SELF-CONTAINED, STORAGE-AGNOSTIC: every type + function here is pure -- no DB, no env, no host-specific
+// event vocabulary. Mirrors src/review/auto-tune.ts's own FlagStore-injection precedent (see that file's
+// header comment): the pure calibration math lives here, and each host (ORB, AMS) supplies its own
+// `SignalStore` implementation wired to whatever it already uses for durable storage (ORB: audit_events over
+// D1/Postgres; AMS: the local append-only event ledger). This module does NOT replace either of those --
+// see outcomes-wire.ts (ORB) and event-ledger.ts (AMS), which this wraps.
+//
+// DEFERRED (out of scope here -- foundation only, no behavior change for either consumer until #7983/#7984/
+// #7986 actually consume it):
+//   • the live SignalStore implementations (the ORB and AMS adapters, wired at the host layer).
+//   • any circuit-breaker / alerting action taken FROM a RulePrecisionReport or repeat count -- this module
+//     only computes the numbers, exactly like auto-tune.ts's GateEvalReport is computed elsewhere and only
+//     READ by the breaker logic.
+
+/** A single instance of a deterministic rule firing against a target -- the shared "the system made a call"
+ *  primitive. `ruleId` is host-defined (an ORB gate-blocker code like `missing_linked_issue`, or an AMS
+ *  eligibility-exclusion reason like `missing_eligibility_label`); `targetKey` is host-defined too (ORB:
+ *  `owner/repo#123`; AMS: `owner/repo#issue-456`) -- this module never parses or interprets either string. */
+export type RuleFiredEvent = {
+  ruleId: string;
+  targetKey: string;
+  outcome: string;
+  occurredAt: string;
+  metadata?: Record<string, unknown>;
+};
+
+/** A human's later, explicit judgment on a specific prior rule firing: `"reversed"` means the target should
+ *  NOT have been blocked/excluded (the rule was wrong this time); `"confirmed"` means it should have been
+ *  (the rule was right). Absence of an override is NOT itself a signal either way -- most fired rules never
+ *  get an explicit human judgment, and {@link computeRulePrecision} only scores the ones that do (mirrors
+ *  auto-tune.ts's GateEvalRow: `decided` is always <= `fired`/`wouldMerge`, never assumed equal to it). */
+export type HumanOverrideEvent = {
+  ruleId: string;
+  targetKey: string;
+  verdict: "reversed" | "confirmed";
+  occurredAt: string;
+  metadata?: Record<string, unknown>;
+};
+
+/** The minimal storage seam a host implements. Every method is async so a real implementation can hit a DB;
+ *  a pure in-memory test double satisfies this trivially. Mirrors FlagStore's shape (auto-tune.ts): a small,
+ *  named set of operations, not a generic read/write-anything interface. */
+export interface SignalStore {
+  recordRuleFired(event: RuleFiredEvent): Promise<void>;
+  recordHumanOverride(event: HumanOverrideEvent): Promise<void>;
+  /** Every fired + override event for `ruleId` at or after `sinceMs` (epoch millis), oldest first. A host MAY
+   *  scope this further (e.g. to one repo) internally; the interface itself is unscoped beyond `ruleId`. */
+  queryRuleHistory(ruleId: string, sinceMs: number): Promise<{ fired: RuleFiredEvent[]; overrides: HumanOverrideEvent[] }>;
+}
+
+/** Per-rule confusion-style report over a window: how many times it fired, how many of those got an explicit
+ *  human verdict, and the resulting precision. Mirrors auto-tune.ts's GateEvalRow shape (fired ~ wouldMerge/
+ *  wouldClose, reversed ~ mergeFalse/closeFalse) at a per-RULE grain instead of per-project -- the same
+ *  "confirmed / decided, decided <= fired" relationship, just keyed differently. */
+export type RulePrecisionReport = {
+  ruleId: string;
+  fired: number;
+  reversed: number;
+  confirmed: number;
+  decided: number;
+  /** confirmed / decided, or null when decided === 0 (no human verdict yet -- never coerced to 0 or 1, same
+   *  "unknown stays unknown" discipline as GateEvalRow's null precision fields). */
+  precision: number | null;
+};
+
+/** True for a override event that targets the same rule as `ruleId` -- the shared filter both
+ *  {@link computeRulePrecision} and any future per-target lookup would need. */
+function overrideMatchesRule(event: HumanOverrideEvent, ruleId: string): boolean {
+  return event.ruleId === ruleId;
+}
+
+/**
+ * Compute a {@link RulePrecisionReport} for `ruleId` from its fired + override events. Only overrides whose
+ * `ruleId` matches are counted (a caller MAY pass a mixed-rule event list without filtering first); a
+ * `targetKey` that never fired but has an override is impossible by construction upstream and is simply
+ * counted as a decided verdict with no matching fire (does not affect `fired`, only `reversed`/`confirmed`/
+ * `decided`) -- this function does not attempt to cross-validate the two lists against each other, mirroring
+ * computeGateEval's own "trust the caller's already-joined rows" posture.
+ */
+export function computeRulePrecision(ruleId: string, fired: readonly RuleFiredEvent[], overrides: readonly HumanOverrideEvent[]): RulePrecisionReport {
+  const firedCount = fired.reduce((count, event) => (event.ruleId === ruleId ? count + 1 : count), 0);
+  let reversed = 0;
+  let confirmed = 0;
+  for (const event of overrides) {
+    if (!overrideMatchesRule(event, ruleId)) continue;
+    if (event.verdict === "reversed") reversed += 1;
+    else confirmed += 1;
+  }
+  const decided = reversed + confirmed;
+  return {
+    ruleId,
+    fired: firedCount,
+    reversed,
+    confirmed,
+    decided,
+    precision: decided > 0 ? confirmed / decided : null,
+  };
+}
+
+/**
+ * Count how many times `ruleId` fired against the exact same `targetKey` within `fired` -- the #7983
+ * "same-rule repeat alarm" primitive (a rule re-firing against a target it already fired against once is a
+ * stronger signal than a bare one-off fire, independent of whether either fire has been overridden yet).
+ * Pure counting, no time-windowing here -- a caller windows `fired` itself before calling this (e.g. via
+ * `queryRuleHistory`'s own `sinceMs`), matching how this whole module leaves all storage/scoping to the host.
+ */
+export function computeRuleRepeatCount(ruleId: string, targetKey: string, fired: readonly RuleFiredEvent[]): number {
+  return fired.reduce((count, event) => (event.ruleId === ruleId && event.targetKey === targetKey ? count + 1 : count), 0);
+}

packages/loopover-engine/src/index.ts

@@ -162,6 +162,7 @@ export * from "./governor/run-halt.js";
 export * from "./governor/kill-switch.js";
 export * from "./governor/action-mode.js";
 export * from "./governor/chokepoint.js";
+export * from "./calibration/signal-tracking.js";
 export {
   GOVERNOR_LEDGER_EVENT_TYPES,
   normalizeGovernorLedgerEvent,

packages/loopover-engine/test/signal-tracking.test.ts

@@ -0,0 +1,92 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+
+import { computeRulePrecision, computeRuleRepeatCount, type HumanOverrideEvent, type RuleFiredEvent } from "../dist/index.js";
+
+function fired(ruleId: string, targetKey: string, overrides: Partial<RuleFiredEvent> = {}): RuleFiredEvent {
+  return { ruleId, targetKey, outcome: "block", occurredAt: "2026-07-22T00:00:00.000Z", ...overrides };
+}
+
+function override(
+  ruleId: string,
+  targetKey: string,
+  verdict: HumanOverrideEvent["verdict"],
+  overrides: Partial<HumanOverrideEvent> = {},
+): HumanOverrideEvent {
+  return { ruleId, targetKey, verdict, occurredAt: "2026-07-22T00:00:00.000Z", ...overrides };
+}
+
+test("barrel: the public entrypoint re-exports the signal-tracking primitives (#7982)", () => {
+  assert.equal(typeof computeRulePrecision, "function");
+  assert.equal(typeof computeRuleRepeatCount, "function");
+});
+
+test("computeRulePrecision: no overrides -> decided is 0 and precision is null (unknown stays unknown, never coerced)", () => {
+  const report = computeRulePrecision("missing_linked_issue", [fired("missing_linked_issue", "a#1"), fired("missing_linked_issue", "a#2")], []);
+  assert.deepEqual(report, {
+    ruleId: "missing_linked_issue",
+    fired: 2,
+    reversed: 0,
+    confirmed: 0,
+    decided: 0,
+    precision: null,
+  });
+});
+
+test("computeRulePrecision: mixes confirmed and reversed verdicts into a real precision", () => {
+  const report = computeRulePrecision(
+    "missing_linked_issue",
+    [fired("missing_linked_issue", "a#1"), fired("missing_linked_issue", "a#2"), fired("missing_linked_issue", "a#3")],
+    [
+      override("missing_linked_issue", "a#1", "confirmed"),
+      override("missing_linked_issue", "a#2", "confirmed"),
+      override("missing_linked_issue", "a#3", "reversed"),
+    ],
+  );
+  assert.equal(report.fired, 3);
+  assert.equal(report.confirmed, 2);
+  assert.equal(report.reversed, 1);
+  assert.equal(report.decided, 3);
+  assert.equal(report.precision, 2 / 3);
+});
+
+test("computeRulePrecision: 100% reversed yields precision 0, not null (a real, scored bad outcome, not an unknown one)", () => {
+  const report = computeRulePrecision("bad_rule", [fired("bad_rule", "a#1")], [override("bad_rule", "a#1", "reversed")]);
+  assert.equal(report.decided, 1);
+  assert.equal(report.precision, 0);
+});
+
+test("computeRulePrecision: ignores fired/override events for a DIFFERENT ruleId entirely", () => {
+  const report = computeRulePrecision(
+    "rule_a",
+    [fired("rule_a", "a#1"), fired("rule_b", "a#2")],
+    [override("rule_a", "a#1", "confirmed"), override("rule_b", "a#2", "reversed")],
+  );
+  assert.equal(report.fired, 1);
+  assert.equal(report.confirmed, 1);
+  assert.equal(report.reversed, 0);
+});
+
+test("computeRulePrecision: an override with no matching fired event still counts toward decided (no cross-validation between the two lists)", () => {
+  const report = computeRulePrecision("rule_a", [], [override("rule_a", "a#1", "confirmed")]);
+  assert.equal(report.fired, 0);
+  assert.equal(report.decided, 1);
+  assert.equal(report.precision, 1);
+});
+
+test("computeRuleRepeatCount: counts only fires matching BOTH ruleId and targetKey", () => {
+  const events = [
+    fired("rule_a", "a#1"),
+    fired("rule_a", "a#1"),
+    fired("rule_a", "a#2"),
+    fired("rule_b", "a#1"),
+  ];
+  assert.equal(computeRuleRepeatCount("rule_a", "a#1", events), 2);
+  assert.equal(computeRuleRepeatCount("rule_a", "a#2", events), 1);
+  assert.equal(computeRuleRepeatCount("rule_b", "a#1", events), 1);
+  assert.equal(computeRuleRepeatCount("rule_a", "a#3", events), 0);
+});
+
+test("computeRuleRepeatCount: zero fired events yields 0, not an error", () => {
+  assert.equal(computeRuleRepeatCount("rule_a", "a#1", []), 0);
+});

packages/loopover-miner/docs/ams-signal-tracking-design.md

@@ -0,0 +1,81 @@
+# AMS signal-tracking design — the #7982 shared calibration module and what AMS records into it
+
+Design doc for **#7982**, the "extract a shared, deployment-agnostic calibration/signal-tracking module for
+ORB + AMS" foundation issue under the false-positive/self-correction roadmap (#7980). Covers: where the shared
+primitive lives, the exact shape of what AMS now writes into it, and — the explicit gap this doc exists to
+name — what AMS still does **not** write, and why.
+
+## The shared primitive
+
+`packages/loopover-engine/src/calibration/signal-tracking.ts` — pure, storage-agnostic, mirrors
+`src/review/auto-tune.ts`'s own `FlagStore`-injection precedent:
+
+- `RuleFiredEvent` — a deterministic rule firing against a target (`ruleId`, `targetKey`, `outcome`,
+  `occurredAt`, optional `metadata`). Host-defined strings throughout; the engine module never parses either.
+- `HumanOverrideEvent` — a human's later, explicit judgment on a specific prior firing (`"reversed"` — the
+  rule was wrong — or `"confirmed"` — it was right).
+- `SignalStore` — the injected storage seam (`recordRuleFired`, `recordHumanOverride`, `queryRuleHistory`).
+- `computeRulePrecision` / `computeRuleRepeatCount` — pure functions over already-fetched event lists; the
+  primitives #7983 (same-rule repeat alarm) and #7984 (per-rule precision tracking) build on directly.
+
+Two adapters implement `SignalStore`, each wrapping existing storage rather than inventing new tables:
+
+- **ORB**: `src/review/signal-tracking-wire.ts`, wrapping `audit_events` (via `recordAuditEvent` /
+  `listAuditEventsByType`, `src/db/repositories.ts`). `ruleId` is folded into `event_type` as
+  `signal.rule_fired:<ruleId>` / `signal.human_override:<ruleId>`, keeping a per-rule history query an
+  efficient index range scan (`audit_events_type_created_idx`) instead of a metadata scan.
+- **AMS**: `packages/loopover-miner/lib/signal-tracking-store.ts`, wrapping the miner's local append-only
+  `event-ledger.ts` under two new event types (`signal_rule_fired`, `signal_human_override`). No indexed
+  per-rule query exists on this store, so `queryRuleHistory` scans the whole local ledger and filters
+  client-side — the same pattern `calibration-cli.ts`'s `toOutcomeRecords` already uses for that same ledger.
+  Fine at AMS's bounded, single-operator local volume; not a hosted-scale query shape.
+
+## What AMS now records (live, not deferred)
+
+`packages/miner-lib/discover-cli.ts`'s real (non-`--dry-run`) run wires the eligibility filter
+(`contribution-profile-filter.ts`'s `filterCandidatesByProfiles`) to `recordRuleFired`: every candidate the
+filter excludes writes one event, `ruleId` = the exclusion reason (`exclusion_label`,
+`missing_eligibility_label`, `conflicting_signals`, `excluded_assignee` — see
+`ELIGIBILITY_EXCLUSION_REASONS`), `targetKey` = `<owner>/<repo>#issue-<N>`, `outcome` = `"exclude"`.
+
+Deliberately **not** wired on `--dry-run`: a dry run previews what a real run would do (it already uses a
+no-op portfolio-queue store for the same reason) and must not itself contribute real data to a future
+precision report. Deliberately best-effort: a store-open failure or a single event's write failure never
+aborts discovery, matching every other optional store in this file (policy caches, ranked-candidates
+snapshot).
+
+This closes the exact gap #7982's own audit found: `contribution-profile-filter.ts` was previously a 100%
+pure function with zero persistence — when a rule excluded a candidate, nothing recorded that decision at
+all, so there was no way to later ask "how often was this exclusion actually right?"
+
+## What AMS still does not record — the human-override gap
+
+`recordHumanOverride` exists on the interface and the AMS adapter implements it correctly, but **nothing in
+AMS calls it yet.** This is a real, known gap, not an oversight in scope:
+
+ORB's human-override signal (see `src/review/outcomes-wire.ts`'s `recordReversalSignals`) has a natural
+trigger: a human directly acts on the exact artifact the bot produced (reopens a bot-closed PR, reverts a
+bot-merged one) — the same PR, a GitHub-native action, unambiguous provenance.
+
+AMS's eligibility exclusion has no equivalent natural trigger today. Excluding a candidate means AMS never
+even attempts the issue — there is no AMS-authored PR, no AMS-facing artifact a human could act on to signal
+"you were wrong to skip this." Discovering that an exclusion was wrong currently requires an operator to
+notice, out-of-band, that AMS skipped a genuinely-eligible issue (e.g. by reading `discover --json`'s
+`excluded` field themselves) — and nothing today captures that observation back into the ledger.
+
+**This is explicitly out of scope for #7982** (foundation only) but is the concrete, actionable follow-up a
+future sub-issue should own. Two candidate designs, neither implemented here:
+
+1. **Operator-driven**: a `loopover-miner discover mark-eligible <repo>#<issue>` command that looks up the
+   most recent `signal_rule_fired` event for that target and writes a matching `recordHumanOverride("reversed")`
+   — cheap, but requires an operator to actually run it.
+2. **Signal-driven**: if a repo's `ContributionProfile` is later re-extracted (profiles are re-resolved
+   periodically) and an issue that was previously excluded would now be *kept* under the fresh profile, treat
+   that transition as an implicit `"reversed"` signal for the original exclusion. Requires diffing two
+   `filterCandidatesByProfiles` runs over time, which discover-cli.ts does not currently retain.
+
+Either design is a real, scoped follow-up issue, not a blocking dependency of #7983/#7984/#7986 — both of
+those can compute meaningful repeat-count/precision reports from `recordRuleFired` data alone; precision
+reports will simply show `decided: 0, precision: null` for every AMS rule until an override path exists,
+which `computeRulePrecision`'s own contract already represents correctly (unknown stays unknown, never
+coerced to "always right").