You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
The human-override verdict vocabulary is closed: HumanOverrideEvent.verdict is "reversed" | "confirmed" (packages/loopover-engine/src/calibration/signal-tracking.ts:36-42), written into metadata.verdict by recordHumanOverride (src/review/signal-tracking-wire.ts:86-94).
Two readers fold those rows into precision, and both treat "anything that is not the string 'reversed'" as a
confirmation:
src/review/public-rule-precision.ts:56-69 — the PUBLIC surface:
SELECT ... COUNT(*) AS decided,
SUM(CASE WHEN json_extract(metadata_json, '$.verdict') ='reversed' THEN 1 ELSE 0 END) AS reversed
then const confirmed = decided - reversed;.
src/services/rule-calibration-trend.ts:167 — the operator trend:
SUM(CASE WHEN json_extract(metadata_json, '$.verdict') ='reversed' THEN 0 ELSE 1 END) AS confirmed
A row whose metadata_json is malformed, whose verdict key is absent, or whose value is any third string
(json_extract yields SQL NULL in the first two cases, and NULL = 'reversed' is never true) is therefore
counted as a confirmation — it raises precision toward 100% and simultaneously raises decided toward
the PUBLIC_PRECISION_MIN_DECIDED sample floor that gates publication at all.
The direction matters: this is the one surface whose entire purpose is to state, publicly, how often the gate's
rules are right, and its degradation mode is to overstate that. The module header explicitly claims "the public
claim and the internal number can never diverge because they are one number" and "Sparse rules report null
precision, never a misreadable 0%" — the second discipline is applied, the unknown-verdict case is not.
The asymmetry is visible inside rule-calibration-trend.ts itself. Its sibling loader loadBacktestRunDayRows (lines 182-197) handles exactly this hazard, with an explicit third bucket and a doc
comment stating the rule: "A row whose verdict is missing/unrecognized counts as unchanged — a malformed run
must not vanish from runs entirely." The override loader thirty lines above has no such bucket and folds the
unknown case into the favourable side.
Requirements
Both queries must classify override rows three ways, not two: reversed, confirmed, and unrecognized (verdict absent, SQL NULL, or any value that is neither literal 'reversed' nor literal 'confirmed').
decided must be reversed + confirmed only. An unrecognized row must NOT enter decided, must NOT enter confirmed, and must NOT count toward PUBLIC_PRECISION_MIN_DECIDED or MIN_CALIBRATION_TREND_SAMPLE.
The unrecognized count must be surfaced, not silently dropped: add a required unrecognized: number field to PublicRulePrecisionRow (src/review/public-rule-precision.ts:26) and to CalibrationRuleTrendWeek
(src/services/rule-calibration-trend.ts:29). A malformed row must be visible as a data-quality signal, per loadBacktestRunDayRows's stated rule.
src/openapi/schemas.ts must be updated for the new PublicRulePrecisionRow field and npm run ui:openapi re-run and committed.
⚠️ Required pattern: mirror loadBacktestRunDayRows in src/services/rule-calibration-trend.ts:182-197 —
an explicit NOT IN (...) OR ... IS NULL third bucket in the same SUM(CASE ...) style, in both queries.
What does NOT satisfy this issue: filtering unrecognized rows out in the WHERE clause so they disappear from
every count; fixing only the public reader and leaving the operator trend; classifying in TypeScript after the
read (both loaders aggregate in SQL and must stay that way for the same reason safeAll is used); or making unrecognized an optional field.
Deliverables
loadPublicRulePrecision in src/review/public-rule-precision.ts computes reversed, confirmed and unrecognized as three separate SQL sums, with decided = reversed + confirmed.
loadOverrideDayRows in src/services/rule-calibration-trend.ts does the same, and buildCalibrationTrend's decided excludes unrecognized rows.
PublicRulePrecisionRow.unrecognized and CalibrationRuleTrendWeek.unrecognized exist as required number fields; src/openapi/schemas.ts is updated and the regenerated ui:openapi artifact is committed.
A named regression test asserting that a rule with 9 confirmed, 1 reversed and 5 rows whose metadata_json lacks $.verdict reports decided: 10, confirmed: 9, unrecognized: 5 and precision: 0.9 — where today it reports decided: 15, confirmed: 14, precision: 0.933.
An equivalent test for the weekly trend, covering a week that falls BELOW MIN_CALIBRATION_TREND_SAMPLE once unrecognized rows are excluded and therefore reports null confirmed/reversed/precisionPct.
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding
the third SQL bucket without removing unrecognized rows from decided — does not resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside src/**, which
is inside coverage.include, so every changed line and branch is measured and gated. Both arms of every
changed conditional need a test, including the above-floor/below-floor arms of PUBLIC_PRECISION_MIN_DECIDED
and MIN_CALIBRATION_TREND_SAMPLE under the new decided. The regression test in Deliverable 4 is mandatory
and must fail against the current code.
Expected Outcome
A malformed or schema-drifted override row can no longer raise the publicly-published per-rule precision, and
its existence is visible as an explicit unrecognized count on both the public and the operator surface
instead of being laundered into the confirmation column.
Context
The human-override verdict vocabulary is closed:
HumanOverrideEvent.verdictis"reversed" | "confirmed"(packages/loopover-engine/src/calibration/signal-tracking.ts:36-42), written intometadata.verdictbyrecordHumanOverride(src/review/signal-tracking-wire.ts:86-94).Two readers fold those rows into precision, and both treat "anything that is not the string
'reversed'" as aconfirmation:
src/review/public-rule-precision.ts:56-69— the PUBLIC surface:then
const confirmed = decided - reversed;.src/services/rule-calibration-trend.ts:167— the operator trend:A row whose
metadata_jsonis malformed, whoseverdictkey is absent, or whose value is any third string(
json_extractyields SQL NULL in the first two cases, andNULL = 'reversed'is never true) is thereforecounted as a confirmation — it raises
precisiontoward 100% and simultaneously raisesdecidedtowardthe
PUBLIC_PRECISION_MIN_DECIDEDsample floor that gates publication at all.The direction matters: this is the one surface whose entire purpose is to state, publicly, how often the gate's
rules are right, and its degradation mode is to overstate that. The module header explicitly claims "the public
claim and the internal number can never diverge because they are one number" and "Sparse rules report null
precision, never a misreadable 0%" — the second discipline is applied, the unknown-verdict case is not.
The asymmetry is visible inside
rule-calibration-trend.tsitself. Its sibling loaderloadBacktestRunDayRows(lines 182-197) handles exactly this hazard, with an explicit third bucket and a doccomment stating the rule: "A row whose verdict is missing/unrecognized counts as
unchanged— a malformed runmust not vanish from
runsentirely." The override loader thirty lines above has no such bucket and folds theunknown case into the favourable side.
Requirements
reversed,confirmed, andunrecognized(verdict absent, SQL NULL, or any value that is neither literal'reversed'nor literal'confirmed').decidedmust bereversed + confirmedonly. An unrecognized row must NOT enterdecided, must NOT enterconfirmed, and must NOT count towardPUBLIC_PRECISION_MIN_DECIDEDorMIN_CALIBRATION_TREND_SAMPLE.unrecognized: numberfield toPublicRulePrecisionRow(src/review/public-rule-precision.ts:26) and toCalibrationRuleTrendWeek(
src/services/rule-calibration-trend.ts:29). A malformed row must be visible as a data-quality signal, perloadBacktestRunDayRows's stated rule.src/openapi/schemas.tsmust be updated for the newPublicRulePrecisionRowfield andnpm run ui:openapire-run and committed.buildEvalScoreRecordsFromRulePrecision(src/review/eval-score-records.ts:96-103) readsrow.decidedandrow.confirmed; it must keep compiling and must continue to emit exactly those two counts — do not addunrecognizedtoEvalScoreRecordScore, whose shape is fixed by Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215.Deliverables
loadPublicRulePrecisioninsrc/review/public-rule-precision.tscomputesreversed,confirmedandunrecognizedas three separate SQL sums, withdecided = reversed + confirmed.loadOverrideDayRowsinsrc/services/rule-calibration-trend.tsdoes the same, andbuildCalibrationTrend'sdecidedexcludes unrecognized rows.PublicRulePrecisionRow.unrecognizedandCalibrationRuleTrendWeek.unrecognizedexist as requirednumberfields;src/openapi/schemas.tsis updated and the regeneratedui:openapiartifact is committed.confirmed, 1reversedand 5 rows whosemetadata_jsonlacks$.verdictreportsdecided: 10,confirmed: 9,unrecognized: 5andprecision: 0.9— where today it reportsdecided: 15,confirmed: 14,precision: 0.933.MIN_CALIBRATION_TREND_SAMPLEonce unrecognized rows are excluded and therefore reports nullconfirmed/reversed/precisionPct.All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding
the third SQL bucket without removing unrecognized rows from
decided— does not resolve this issue.Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside
src/**, whichis inside
coverage.include, so every changed line and branch is measured and gated. Both arms of everychanged conditional need a test, including the above-floor/below-floor arms of
PUBLIC_PRECISION_MIN_DECIDEDand
MIN_CALIBRATION_TREND_SAMPLEunder the newdecided. The regression test in Deliverable 4 is mandatoryand must fail against the current code.
Expected Outcome
A malformed or schema-drifted override row can no longer raise the publicly-published per-rule precision, and
its existence is visible as an explicit
unrecognizedcount on both the public and the operator surfaceinstead of being laundered into the confirmation column.
Links & Resources
src/review/public-rule-precision.ts:51-77,src/services/rule-calibration-trend.ts:163-197,packages/loopover-engine/src/calibration/signal-tracking.ts:31-42,src/review/signal-tracking-wire.ts:85-95,src/review/eval-score-records.ts:96-103,src/openapi/schemas.ts:130.