Skip to content

orb(precision): a missing or unrecognized $.verdict is counted as CONFIRMED #9640

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

The human-override verdict vocabulary is closed: HumanOverrideEvent.verdict is
"reversed" | "confirmed" (packages/loopover-engine/src/calibration/signal-tracking.ts:36-42), written into
metadata.verdict by recordHumanOverride (src/review/signal-tracking-wire.ts:86-94).

Two readers fold those rows into precision, and both treat "anything that is not the string 'reversed'" as a
confirmation:

src/review/public-rule-precision.ts:56-69 — the PUBLIC surface:

SELECT ... COUNT(*) AS decided,
       SUM(CASE WHEN json_extract(metadata_json, '$.verdict') = 'reversed' THEN 1 ELSE 0 END) AS reversed

then const confirmed = decided - reversed;.

src/services/rule-calibration-trend.ts:167 — the operator trend:

SUM(CASE WHEN json_extract(metadata_json, '$.verdict') = 'reversed' THEN 0 ELSE 1 END) AS confirmed

A row whose metadata_json is malformed, whose verdict key is absent, or whose value is any third string
(json_extract yields SQL NULL in the first two cases, and NULL = 'reversed' is never true) is therefore
counted as a confirmation — it raises precision toward 100% and simultaneously raises decided toward
the PUBLIC_PRECISION_MIN_DECIDED sample floor that gates publication at all.

The direction matters: this is the one surface whose entire purpose is to state, publicly, how often the gate's
rules are right, and its degradation mode is to overstate that. The module header explicitly claims "the public
claim and the internal number can never diverge because they are one number" and "Sparse rules report null
precision, never a misreadable 0%" — the second discipline is applied, the unknown-verdict case is not.

The asymmetry is visible inside rule-calibration-trend.ts itself. Its sibling loader
loadBacktestRunDayRows (lines 182-197) handles exactly this hazard, with an explicit third bucket and a doc
comment stating the rule: "A row whose verdict is missing/unrecognized counts as unchanged — a malformed run
must not vanish from runs entirely."
The override loader thirty lines above has no such bucket and folds the
unknown case into the favourable side.

Requirements

  • Both queries must classify override rows three ways, not two: reversed, confirmed, and
    unrecognized (verdict absent, SQL NULL, or any value that is neither literal 'reversed' nor literal
    'confirmed').
  • decided must be reversed + confirmed only. An unrecognized row must NOT enter decided, must NOT enter
    confirmed, and must NOT count toward PUBLIC_PRECISION_MIN_DECIDED or MIN_CALIBRATION_TREND_SAMPLE.
  • The unrecognized count must be surfaced, not silently dropped: add a required unrecognized: number field to
    PublicRulePrecisionRow (src/review/public-rule-precision.ts:26) and to CalibrationRuleTrendWeek
    (src/services/rule-calibration-trend.ts:29). A malformed row must be visible as a data-quality signal, per
    loadBacktestRunDayRows's stated rule.
  • src/openapi/schemas.ts must be updated for the new PublicRulePrecisionRow field and
    npm run ui:openapi re-run and committed.
  • buildEvalScoreRecordsFromRulePrecision (src/review/eval-score-records.ts:96-103) reads row.decided and
    row.confirmed; it must keep compiling and must continue to emit exactly those two counts — do not add
    unrecognized to EvalScoreRecordScore, whose shape is fixed by Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215.

⚠️ Required pattern: mirror loadBacktestRunDayRows in src/services/rule-calibration-trend.ts:182-197
an explicit NOT IN (...) OR ... IS NULL third bucket in the same SUM(CASE ...) style, in both queries.
What does NOT satisfy this issue: filtering unrecognized rows out in the WHERE clause so they disappear from
every count; fixing only the public reader and leaving the operator trend; classifying in TypeScript after the
read (both loaders aggregate in SQL and must stay that way for the same reason safeAll is used); or making
unrecognized an optional field.

Deliverables

  • loadPublicRulePrecision in src/review/public-rule-precision.ts computes reversed, confirmed and
    unrecognized as three separate SQL sums, with decided = reversed + confirmed.
  • loadOverrideDayRows in src/services/rule-calibration-trend.ts does the same, and
    buildCalibrationTrend's decided excludes unrecognized rows.
  • PublicRulePrecisionRow.unrecognized and CalibrationRuleTrendWeek.unrecognized exist as required
    number fields; src/openapi/schemas.ts is updated and the regenerated ui:openapi artifact is committed.
  • A named regression test asserting that a rule with 9 confirmed, 1 reversed and 5 rows whose
    metadata_json lacks $.verdict reports decided: 10, confirmed: 9, unrecognized: 5 and
    precision: 0.9 — where today it reports decided: 15, confirmed: 14, precision: 0.933.
  • An equivalent test for the weekly trend, covering a week that falls BELOW
    MIN_CALIBRATION_TREND_SAMPLE once unrecognized rows are excluded and therefore reports null
    confirmed/reversed/precisionPct.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding
the third SQL bucket without removing unrecognized rows from decided — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside src/**, which
is inside coverage.include, so every changed line and branch is measured and gated. Both arms of every
changed conditional need a test, including the above-floor/below-floor arms of PUBLIC_PRECISION_MIN_DECIDED
and MIN_CALIBRATION_TREND_SAMPLE under the new decided. The regression test in Deliverable 4 is mandatory
and must fail against the current code.

Expected Outcome

A malformed or schema-drifted override row can no longer raise the publicly-published per-rule precision, and
its existence is visible as an explicit unrecognized count on both the public and the operator surface
instead of being laundered into the confirmation column.

Links & Resources

src/review/public-rule-precision.ts:51-77, src/services/rule-calibration-trend.ts:163-197,
packages/loopover-engine/src/calibration/signal-tracking.ts:31-42, src/review/signal-tracking-wire.ts:85-95,
src/review/eval-score-records.ts:96-103, src/openapi/schemas.ts:130.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions