Skip to content

orb(calibration): two aggregators over one ledger publish contradictory positiveRate values #9701

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

Both aggregators read agent_recommendation_outcomes and both expose positiveRate, with opposite treatment of the same two states.

src/services/outcome-calibration.ts:133-137 (doc at :65: "Positive (accepted/merged/improved) vs negative (rejected/closed) vs pending (stale/ignored) split") computes negative from rejected/closed only, pending from stale/ignored, resolved = positive + negative, and returns positiveRate: resolved > 0 ? round(positive / resolved) : null.

src/services/recommendation-quality-report.ts:75 and :160-169 declares NEGATIVE_STATES = ["closed", "rejected", "stale", "ignored"] and returns positiveRate: total > 0 ? roundRate(positive / total) : 0.

A ledger of 5 accepted + 5 stale yields positiveRate: 1.0 from buildRecommendationOutcomeCalibration and positiveRate: 0.5 from qualityTotals — two contradictory numbers for the same window, shown to the same operator. recommendation-quality-report.ts additionally lists stale/ignored under failureCategories, while outcome-calibration.ts:150 renders them as "still pending" in its signal string.

There is a second, independent divergence in the same pair: the zero-denominator arm. outcome-calibration.ts:137 returns null (typed number | null, correctly "no sample"); recommendation-quality-report.ts:169 returns 0, which is indistinguishable from a genuine 0% positive rate.

Requirements

  • Define the state classification once, in src/services/outcome-calibration.ts (the older module, and the one whose semantics the calibration signal strings already describe):
    • export const RECOMMENDATION_POSITIVE_STATES = ["accepted", "merged", "improved"]
    • export const RECOMMENDATION_NEGATIVE_STATES = ["rejected", "closed"]
    • export const RECOMMENDATION_PENDING_STATES = ["stale", "ignored"]
      each typed readonly AgentRecommendationOutcomeState[].
  • buildRecommendationOutcomeCalibration uses those constants instead of its inline literals at :134-136.
  • src/services/recommendation-quality-report.ts deletes POSITIVE_STATES/NEGATIVE_STATES (:74-75) and imports the shared constants; qualityTotals counts negative from RECOMMENDATION_NEGATIVE_STATES only, so stale/ignored leave the positiveRate denominator.
  • failureCategoryRows must stop attributing stale/ignored as failures; they are reported as pending. Add a pending count to RecommendationQualityTotals mirroring RecommendationOutcomeCalibration.pending so the information is not lost.
  • qualityTotals.positiveRate becomes number | null and returns null (not 0) on a zero denominator, matching outcome-calibration.ts:137. Update RecommendationQualityTotals's type and every render/consumer of that field accordingly, including the OpenAPI schema if it is specced (regenerate with npm run ui:openapi if so).
  • No change to AgentRecommendationOutcomeState itself or to the ledger's write path.

⚠️ Required pattern: outcome-calibration.ts's three-way positive/negative/pending split, and its resolved > 0 ? ... : null zero-denominator arm at :137, are the semantics both modules must share. It does NOT satisfy this issue to change outcome-calibration.ts to match recommendation-quality-report.ts instead (its own signal string at :150 and its pending field already commit to the pending semantics); to keep two constant lists that happen to agree today; or to unify the states but leave positiveRate returning 0 for an empty ledger in one module and null in the other.

Deliverables

  • The three RECOMMENDATION_*_STATES constants are exported from src/services/outcome-calibration.ts and are the only place those state literals are grouped (grep-verifiable — no NEGATIVE_STATES in recommendation-quality-report.ts).
  • Named regression test: a ledger of 5 accepted + 5 stale produces the same positiveRate (1.0) from buildRecommendationOutcomeCalibration and from buildRecommendationQualityReportFromOutcomes — asserted in one test that calls both.
  • RecommendationQualityTotals.pending is 5 for that same fixture.
  • qualityTotals over an empty ledger returns positiveRate: null, matching buildRecommendationOutcomeCalibration — asserted for both.
  • failureCategories for that fixture contains no stale or ignored row.
  • If RecommendationQualityTotals is specced in src/openapi/, the regenerated artifact is committed (npm run ui:openapi) and CI's drift check passes.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example unifying the state lists while leaving positiveRate returning 0 on an empty ledger — does not resolve this issue.

Test Coverage Requirements

99%+ Codecov patch coverage, branch-counted, on src/**; both files are inside coverage.include. Both arms of the changed zero-denominator conditional need a test in each module, and every state in all three constants must appear in at least one fixture so the classification change is fully exercised.

Expected Outcome

The recommendation-quality report and the outcome-calibration signal agree on what a positive rate means. A window of unactioned (stale/ignored) recommendations no longer reads as a 50% failure rate on one surface and a 100% success rate on another, and an empty ledger reports "no sample" rather than 0%.

Links & Resources

src/services/outcome-calibration.ts:64-70, :126-157; src/services/recommendation-quality-report.ts:60-76, :90-100, :160-175; src/openapi/schemas.ts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions