Skip to content

fix(engine): pairwise-calibration.js's normalizePairwiseWeights silently restores the default 50/50 blend on explicit all-zero weights, unlike its three sibling calibration modules (#6170 precedent) #7443

Description

@JSONbored

Context

Issue #6170 (closed) established that packages/loopover-engine/src has a family of composite-scorer
weight-resolution modules that must all handle an explicit all-zero weight input identically: when a caller
passes every weight as 0, the correct behavior is to preserve the explicit zeros through normalization
(not silently substitute the module's own default blend), so that a downstream "usable weights" stage can
detect the all-zero signal and fall back to an objective-only composite score instead of silently blending
with stale defaults the caller never asked for.

reviewer-consensus-calibration.ts is the reference implementation for both halves of this pattern:

  1. normalizeCompositeWeights (lines 388-409) returns { objectiveAnchor: 0, pairwiseJudge: 0, structuredReviewerConsensus: 0 } on total <= 0, NOT the module's DEFAULT_COMPOSITE_WEIGHTS.
  2. buildRepoRewardRisk's composite-score assembly (lines 566-583) then takes those rawWeights, builds a
    second usableWeights/total pass (zeroing out any component whose OWN signal is unavailable, e.g.
    pairwiseJudgeScore === null), and falls back to { objectiveAnchor: 1, pairwiseJudge: 0, structuredReviewerConsensus: 0 } — an explicit objective-only score — only when that final usable total
    is <= 0.

#6170 backported half 1 (and a related malformed-repo-row fix) to gate-verdict-calibration.ts and
finding-severity-calibration.ts, explicitly naming "three sibling calibration modules" and stating
reviewer-consensus-calibration.ts is "already correct and is the reference implementation." Both named
siblings are now confirmed fixed (gate-verdict-calibration.ts:322, finding-severity-calibration.ts:369
both return explicit zeros; both also have the matching second-stage objective-only fallback at their own
composite-assembly call sites).

A FOURTH module with the identical composite-scorer weight-resolution shape exists in the same package and
was not part of #6170's audit scope at all: pairwise-calibration.ts. Its normalizePairwiseWeights (lines
58-72) has the exact same un-fixed bug #6170 found in the other two:

const total = raw.objectiveAnchor + raw.pairwiseJudge;
if (total <= 0) return DEFAULT_PAIRWISE_WEIGHTS;

Worse, pairwise-calibration.ts is also missing half 2 of the pattern entirely: computePairwiseCalibrationScore
(lines 123-154) uses normalizePairwiseWeights(input.weights)'s result directly as weights with no second
"usable weights" fallback stage at all. This means a mechanical copy of #6170's fix (just deleting the
DEFAULT_PAIRWISE_WEIGHTS fallback) would make things WORSE here, not better: with pairwiseJudgeScore
non-null and truly-zero weights passed through, compositeScore would become objectiveAnchorScore * 0 + pairwiseJudgeScore * 0 = 0 — a real, valid score silently collapsed to zero instead of falling back to the
objective-anchor score alone the way the reference implementation does.

Requirements

  • normalizePairwiseWeights must return the explicit zeroed weights ({ objectiveAnchor: 0, pairwiseJudge: 0 }) when total <= 0, matching all three already-fixed siblings — not DEFAULT_PAIRWISE_WEIGHTS.
  • computePairwiseCalibrationScore must gain the missing second-stage fallback: compute a usableWeights
    total from the (possibly explicit-zero) normalized weights and, when that usable total is <= 0, use an
    objective-anchor-only weight set ({ objectiveAnchor: 1, pairwiseJudge: 0 }) for the actual composite-score
    formula — mirroring reviewer-consensus-calibration.ts's buildRepoRewardRisk assembly (lines 566-583)
    exactly, adapted to this module's two-weight (not three-weight) shape.
  • Genuinely invalid input (NaN, negative numbers) must continue to recover to DEFAULT_PAIRWISE_WEIGHTS as
    today — this fix only changes the behavior for a caller's EXPLICIT, well-formed all-zero input, exactly as
    fix(engine): backport reviewer-consensus-calibration.ts's zero-weight and malformed-repo handling to its two older siblings #6170 scoped its own two fixes.
  • Do not change reviewer-consensus-calibration.ts, gate-verdict-calibration.ts, or
    finding-severity-calibration.ts — they are already correct.

Deliverables

  • normalizePairwiseWeights: explicit-zero passthrough fix (matching the three siblings).
  • computePairwiseCalibrationScore: add the missing objective-only fallback stage when the usable
    weights total is <= 0.
  • Regression test: weights: { objectiveAnchor: 0, pairwiseJudge: 0 } with a real (non-null)
    pairwiseJudgeScore present — asserts compositeScore === objectiveAnchorScore (the objective-only
    fallback), not 0.
  • Existing test at packages/loopover-engine/test/pairwise-calibration.test.ts:146 (NaN/-1 recovers to
    the 50/50 default) continues to pass unchanged.

Test Coverage Requirements

99%+ Codecov patch coverage (branch-counted) on both changed functions. The explicit-all-zero-weights
regression test above is the one that actually proves this issue is fixed — without it, both the
old-silent-default bug and a naive "return zeros with no second-stage fallback" incomplete fix would look
identical to the existing test suite.

Expected Outcome

All four sibling calibration modules (gate-verdict-calibration.ts, finding-severity-calibration.ts,
reviewer-consensus-calibration.ts, pairwise-calibration.ts) handle an explicit all-zero weight input
identically — a real, valid signal falls back to an objective-only score, never to a silently-substituted
default blend the caller never asked for, and never to a spuriously-zeroed composite score.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions