feat(outcome): #543 outcome-learning calibration (measurement-only)

[JSONbored]

What

Closes the outcome-learning loop (#543): measure whether gittensory's deterministic signals actually predict reality. Measurement only — it never auto-adjusts a live score (like time-decay, recalibration stays an owner-reviewed decision, so it can't silently move rankings).

Two questions a maintainer can act on, over data that already exists (no new collection):

• Is the slop score predictive? For resolved PRs carrying a persisted slop band (feat(slop): per-PR slop score row in the maintainer panel #726), do higher-slop bands merge less often? discriminates is true iff merge rate is non-increasing as band severity rises (given ≥5 samples/band), false if it inverts, null if there isn't enough resolved data to judge.
• Are recommendations panning out? Positive (accepted/merged/improved) vs negative (rejected/closed) vs pending (stale/ignored) split from the agent_recommendation_outcomes ledger, scoped to the repo.

How

• src/services/outcome-calibration.ts — pure builders (buildSlopOutcomeCalibration, buildRecommendationOutcomeCalibration, buildOutcomeCalibrationSignals) + the buildRepoOutcomeCalibration env loader. Recommendation scoping (by outcome repo, falling back to target repo, case-insensitive) lives in the pure function so it's fully unit-testable without the DB.
• GET /v1/repos/:owner/:repo/outcome-calibration — maintainer-gated (requireRepoMaintainer), read-only; optional ?windowDays bounds the recommendation window. Path added to the session allowlist (isRepoOutcomeCalibrationPath).

Privacy / safety

• No public surface, no score mutation, no reward/payout/trust-score fields (asserted in tests).
• Read-only; maintainer-scoped.

Verification

• typecheck ✅ · test:coverage ✅ (branches 97.02%, all metrics ≥97) · ui:openapi:check ✅ · git diff --check ✅ · test:workers ✅
• New test/unit/outcome-calibration.test.ts (11 cases: discriminates true/false/null, open/unassessed exclusion, recommendation split, repo scoping incl. target-repo fallback + case-insensitivity, env loader) + integration coverage (401 unauthenticated, 200 windowed/unwindowed shape).

[ghost]

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal	Result	Evidence	Action
Gate result	⚠️ Skipped	#740 is no longer open.	No action.

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[ghost]

gittensory · advisory review

Reviewed 4 changed file(s) — two independent AI reviewers.

Suggested action: ✅ Safe to merge — both reviewers found no blocking issues.

Reviewer A · gpt-oss-120b — recommends ✅ merge
The PR adds a new read‑only outcome‑calibration service and API endpoint, integrates it into routing and permission checks, and provides comprehensive unit and integration tests. The implementation follows existing patterns and does not introduce forbidden public terms.

Suggestions

• Document the new /outcome-calibration endpoint in the API reference.
• Consider adding pagination or limits for large repositories to avoid excessive response sizes.
• Add a brief comment on the windowDays parameter behavior for future maintainers.

Worth double-checking

• Potential performance impact on very large repos due to loading all PRs and recommendation outcomes in one call.
• Ensure the endpoint remains read‑only; future changes should not inadvertently allow mutations.

Reviewer B · nemotron-3-120b-a12b — recommends ✅ merge
This PR adds a measurement-only outcome calibration feature (#543) to assess whether Gittensory's slop scores are predictive and whether agent recommendations pan out. It introduces a new service (outcome-calibration.ts), adds a maintainer-protected API route (/v1/repos/:owner/:repo/outcome-calibration), and includes comprehensive unit/integration tests. The implementation correctly avoids adjusting live scores (as noted) and focuses purely on historical analysis. Code is clean, well-tested, and respects the public/private boundary (all outputs are maintainer-only and free of forbidden terms).

Suggestions

• Consider adding a TODO comment in buildSlopOutcomeCalibration to revisit the MIN_BAND_SAMPLE constant (currently hardcoded to 5) if operators find it too restrictive for low-activity repos.
• In buildRecommendationOutcomeCalibration, the sameRepo helper could be moved to a shared utils file if reused elsewhere (minor).
• The integration test could assert a specific signal string (e.g., when discrimination is true) to improve precision, but current structure checks are sufficient.

Worth double-checking

• Ensure the listPullRequests repository function doesn't inadvertently expose excessive data for very large repos (though pagination isn't shown in the snippet, assume it's handled elsewhere).
• Verify that the windowDays parameter correctly bounds only the recommendation outcomes (not slop data) as intended—current implementation matches this.
• Double-check that the discriminates logic's 0.001 tolerance is appropriate for floating-point comparisons (it is).