Skip to content

feat(miner): opt-in cross-product calibration signal from the maintainer review stack #3015

Description

@JSONbored

Context

Part of the internal historical-replay calibration harness roadmap. When a target repo used in the replay harness also runs the maintainer review stack's own gate, and its owner opts in, the review stack's structured per-dimension gate verdict is a materially richer signal than a coarse merged/closed pr_outcome label. This is a differentiated capability only available because the same team operates both the maintainer review stack and the miner system, and it should be treated as strictly opt-in and privacy-respecting on the maintainer side, never assumed.

Goal

Ingest structured per-dimension gate verdicts from opted-in maintainer-side repos as an additional, richer calibration signal for the replay harness.

Requirements

  • Add an explicit per-repo opt-in flag on the maintainer side controlling whether that repo's structured gate verdicts may be used as a calibration signal for the miner's replay harness; default is opted out.
  • Restrict the ingested signal to structured per-dimension gate verdict data already produced by the review stack for that repo; do not ingest raw review text, secrets, or anything not already part of the gate's own output.
  • Enforce the opt-in check at the point of ingestion into the calibration pipeline, not only at data-collection time, so a later opt-out immediately stops further ingestion.
  • Combine the per-dimension verdict signal with the existing objective-anchor and pairwise-judge scores as an additional weighted input, keeping the combination configurable.
  • Document the opt-in mechanism and scope of ingested data in the per-repo config surface (.gittensory.yml) alongside existing gate/review settings.
  • Add an audit trail recording which opted-in repos contributed calibration data to which replay runs.

Deliverables

  • Per-repo opt-in flag for structured gate-verdict calibration sharing, default off
  • Ingestion path restricted to structured per-dimension verdict fields, enforced at ingestion time against current opt-in state
  • Weighted combination of the per-dimension verdict signal with existing composite calibration score
  • .gittensory.yml documentation for the opt-in flag and data scope
  • Audit trail of opted-in repos contributing to each replay run
  • Unit tests covering opt-in, opt-out, and mid-flight opt-out during an in-progress calibration run

Expected outcomes

  • Zero structured gate-verdict data is ingested for any repo without an active opt-in flag
  • Opted-in repos' per-dimension verdicts measurably tighten the composite calibration score's correlation with real outcomes versus pr_outcome alone

Part of the historical-replay calibration roadmap: #2994

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions