Skip to content

feat(calibration): shared ORB+AMS signal-tracking module, with AMS wired live - #8079

Merged
JSONbored merged 1 commit into
mainfrom
feat/signal-tracking-calibration-module
Jul 22, 2026
Merged

feat(calibration): shared ORB+AMS signal-tracking module, with AMS wired live#8079
JSONbored merged 1 commit into
mainfrom
feat/signal-tracking-calibration-module

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes #7982.

Summary

  • New packages/loopover-engine/src/calibration/signal-tracking.ts: a storage-agnostic RuleFiredEvent/HumanOverrideEvent + SignalStore interface, mirroring src/review/auto-tune.ts's own FlagStore-injection precedent, plus computeRulePrecision/computeRuleRepeatCount — the primitives Same-rule repeat alarm: fast-path detection when one deterministic rule rejects N distinct targets in a short window #7983 (same-rule repeat alarm) and Per-rule (not just per-project) precision tracking in the gate-eval pipeline #7984 (per-rule precision tracking) will build on directly. No behavior change to either consumer's existing decision logic — purely additive.
  • Two adapters, each wrapping existing storage rather than a new table: ORB's src/review/signal-tracking-wire.ts over audit_events (new db/repositories.ts helper listAuditEventsByType, using the existing (event_type, created_at) index), and AMS's packages/loopover-miner/lib/signal-tracking-store.ts over the local event-ledger.ts.
  • AMS is wired live, not deferred: discover-cli.ts's real (non---dry-run) run now records a rule-fired signal for every eligibility exclusion filterCandidatesByProfiles produces. This closes the exact gap the issue's own audit found — that filter was previously a 100% pure function with zero persistence, so there was no way to later ask how often an exclusion was actually right. Best-effort throughout (a store-open or write failure never aborts discovery); never recorded on a dry run (matches the dry-run path's own no-op portfolio-queue precedent).
  • packages/loopover-miner/docs/ams-signal-tracking-design.md documents what's live vs. the one known, explicitly out-of-scope gap: AMS has no natural human-override trigger yet (excluding a candidate produces no AMS-authored artifact a human can act on, unlike reopening/reverting a PR for ORB) — flagged as a concrete follow-up issue, not a blocker for Same-rule repeat alarm: fast-path detection when one deterministic rule rejects N distinct targets in a short window #7983/Per-rule (not just per-project) precision tracking in the gate-eval pipeline #7984/Replace the blanket CONCRETE_EVIDENCE_BLOCKER_CODES breaker exemption with a per-rule track record #7986 (they can compute real reports off fired-only data; precision just reads null/undecided until an override path exists — computeRulePrecision's own contract already represents that correctly).
  • Bonus fix: discovered a real test-isolation leak while wiring this — test/unit/miner-discover-cli.test.ts had no default-store redirect for the event ledger (unlike its existing portfolio-queue/policy-cache redirects), so its eligibility-filtering tests were writing real rows into the developer's own ~/.config/loopover-miner/event-ledger.sqlite3 on every run. Fixed with the same LOOPOVER_MINER_CONFIG_DIR redirect + closeDefaultEventLedger() reset pattern test/unit/miner-event-ledger.test.ts already established.

Test plan

  • npm run typecheck, npm run build --workspace @loopover/engine, npm run build:mcp, npm run build:miner
  • packages/loopover-engine's own npm test (node:test against dist/): 606/606 passing, including 9 new signal-tracking.test.ts cases
  • New tests: test/unit/signal-tracking-wire.test.ts (ORB adapter, 16 cases, 100% line/branch on the new file), test/unit/miner-signal-tracking-store.test.ts (AMS adapter, 12 cases), test/unit/miner-discover-cli.test.ts additions (5 new cases covering the wiring: records on exclusion, no-op when nothing excluded, never records on dry-run, degrades on store-open failure, per-event write failures don't stop the rest)
  • Verified (and fixed) the real-file leak: ls ~/.config/loopover-miner/event-ledger.sqlite3 confirmed absent after every full-suite run post-fix
  • Full unsharded npm run test:coverage: 1091/1091 files, 20362 tests, 0 failures
  • npm run engine-parity:drift-check: clean, no version bump needed (no gate-decision twin-pair file touched)

…red live

Closes #7982.

New packages/loopover-engine/src/calibration/signal-tracking.ts: a
storage-agnostic RuleFiredEvent/HumanOverrideEvent + SignalStore
interface, mirroring auto-tune.ts's own FlagStore-injection precedent,
plus computeRulePrecision/computeRuleRepeatCount -- the primitives
#7983 (same-rule repeat alarm) and #7984 (per-rule precision tracking)
build on. No behavior change to either consumer's existing decision
logic; this is a new, additive signal path.

Two adapters, each wrapping EXISTING storage rather than a new table:
- ORB: src/review/signal-tracking-wire.ts over audit_events (new
  db/repositories.ts helper: listAuditEventsByType).
- AMS: packages/loopover-miner/lib/signal-tracking-store.ts over the
  local event-ledger.

AMS is wired live (not deferred): discover-cli.ts's real (non-dry-run)
run now records a rule-fired signal for every eligibility exclusion
filterCandidatesByProfiles produces, closing the exact gap #7982's own
audit found -- that filter was previously a pure function with zero
persistence, so there was no way to later ask how often an exclusion
was actually right. Best-effort throughout (a write/open failure never
aborts discovery); never recorded on --dry-run.

packages/loopover-miner/docs/ams-signal-tracking-design.md documents
what's live vs. the one known, explicitly out-of-scope gap: AMS has no
natural human-override trigger yet (excluding a candidate produces no
AMS-authored artifact a human can act on the way reopening/reverting a
PR does for ORB) -- flagged as a concrete follow-up, not blocking
#7983/#7984/#7986 (they can compute real reports off fired-only data;
precision just reads null/undecided until an override path exists).

Also fixes a real test-isolation leak discovered while wiring this up:
test/unit/miner-discover-cli.test.ts had no default-store redirect for
the event ledger (unlike its existing portfolio-queue/policy-cache
redirects), so running its eligibility-filtering tests wrote real rows
into the developer's own ~/.config/loopover-miner/event-ledger.sqlite3.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui f242dd1 Commit Preview URL

Branch Preview URL
Jul 22 2026, 08:39 PM

@JSONbored JSONbored self-assigned this Jul 22, 2026
@JSONbored
JSONbored merged commit 873ecf2 into main Jul 22, 2026
13 checks passed
@JSONbored
JSONbored deleted the feat/signal-tracking-calibration-module branch July 22, 2026 20:51
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.60870% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.97%. Comparing base (1026fb3) to head (f242dd1).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...loopover-engine/src/calibration/signal-tracking.ts 0.00% 12 Missing ⚠️
...ckages/loopover-miner/lib/signal-tracking-store.ts 90.00% 0 Missing and 3 partials ⚠️
packages/loopover-miner/lib/discover-cli.ts 91.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8079      +/-   ##
==========================================
- Coverage   91.98%   91.97%   -0.02%     
==========================================
  Files         748      751       +3     
  Lines       76617    76709      +92     
  Branches    23239    23268      +29     
==========================================
+ Hits        70474    70550      +76     
- Misses       5041     5053      +12     
- Partials     1102     1106       +4     
Flag Coverage Δ
shard-1 54.95% <70.65%> (+1.47%) ⬆️
shard-2 54.16% <0.00%> (-1.42%) ⬇️
shard-3 53.98% <20.65%> (-0.55%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/db/repositories.ts 96.73% <100.00%> (+0.01%) ⬆️
src/review/signal-tracking-wire.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/discover-cli.ts 99.51% <91.66%> (-0.49%) ⬇️
...ckages/loopover-miner/lib/signal-tracking-store.ts 90.00% <90.00%> (ø)
...loopover-engine/src/calibration/signal-tracking.ts 0.00% <0.00%> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS

1 participant