Skip to content

calibration: reliability curves — empirical precision by claimed-confidence bucket, with derived threshold suggestion #8226

Description

@JSONbored

Problem

Epic #8211 track E. Knob evaluation steps down hand-picked candidate ladders. The labeled corpus supports something strictly better: per-rule reliability curves (claimed confidence bucket → empirical precision over decided cases) from which an optimal floor FALLS OUT instead of being guessed.

Requirements

⚠️ Required pattern. Engine-pure module beside the backtest primitives; root mirror suite; N/A-over-zero discipline everywhere (a bucket below the sample floor reports null precision, never 0 — the #8085 scorer's own rule).

  • computeReliabilityCurve(cases, buckets): fixed bucket edges (document the choice), per-bucket {cases, confirmed, reversed, precision|null}.
  • deriveThresholdSuggestion(curve, targetPrecision, hardMinimum): the loosest floor whose at-or-above buckets' pooled precision meets target, never below the hard minimum, null when no bucket qualifies or density is insufficient — deterministic and conservative.
  • Both engine-exported; no consumer changes in this issue.

Deliverables

  • Both functions, 100% line+branch mirror coverage (bucket boundaries, null-density arms, suggestion at/below/at-minimum, monotonicity invariant test)

Links & Resources

#8211 (epic), packages/loopover-engine/src/calibration/backtest-score.ts

Boundaries

Pure math only. No advisor/registry integration (maintainer follow-on).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions