Skip to content

feat(miner): add pairwise calibration scoring - #3151

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
joaovictor91123:feat/objective-anchor-scoring
Jul 4, 2026
Merged

feat(miner): add pairwise calibration scoring#3151
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
joaovictor91123:feat/objective-anchor-scoring

Conversation

@joaovictor91123

Copy link
Copy Markdown
Contributor

Summary

  • Adds deterministic objective-anchor feature extraction/scoring for replayed plans versus revealed history, including per-item history aggregation and Markdown audit rendering.
  • Adds the pairwise calibration scorer for order-swapped judge attempts, retry caps, instability metrics, and composite scoring with objective-anchor results.
  • Fixes feat(miner): pairwise-judge scoring layer with order-swap bias defense #3013

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

  • git diff --check HEAD~1..HEAD
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires >=99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • npm test --workspace @jsonbored/gittensory-engine passed after rebase: 102 tests.
  • npm run build:miner passed after rebase.
  • npm run test:coverage and npx vitest run --coverage --reporter=dot both timed out locally on Windows after long runs and ended with EPIPE after the session killed the output pipe, not with a reported assertion failure.
  • npm run test:mcp-pack failed locally on Windows/Node 24 in scripts/check-mcp-package.mjs while writing an undefined spawn diagnostic for npm pack; this feature does not touch MCP packaging.
  • UI checks were not run because this PR changes only the engine package and its package README.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

N/A - no visible UI, frontend, or extension change.

Notes

  • The engine package owns the deterministic scoring and stability logic. The miner runtime can supply order-swapped judge attempts from its own model invocation layer.

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 4, 2026
@loopover-orb

loopover-orb Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-04 17:15:34 UTC

6 files · 1 AI reviewer · no blockers · readiness 80/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
The PR adds a pure objective-anchor scorer, Markdown audit rendering, and a pairwise calibration combiner with public exports and direct tests. The core scoring paths are deterministic, covered, and the pairwise order-swap handling is coherent for stable replay/revealed/tie outcomes. I do not see a reachable correctness break in the visible diff, but there are a couple of contract edges worth tightening before this becomes a calibration API callers depend on.

Nits — 5 non-blocking
  • nit: packages/gittensory-engine/src/objective-anchor.ts:378 aggregates per-item `unknown` change kinds alongside known kinds, so one sparse history item can dilute an otherwise exact known-kind match; either drop `unknown` when any concrete kind is present in `combineFeatures`, or document that missing metadata intentionally penalizes aggregate history scores.
  • nit: packages/gittensory-engine/src/pairwise-calibration.ts:91 accepts non-finite `maxAttempts` values, so `Infinity` yields `maxAttempts: Infinity` and an unstable finite attempt list reports `exhausted: false`; clamp this the same way weights are normalized or require a finite positive integer.
  • nit: packages/gittensory-engine/src/pairwise-calibration.ts:126 uses `input.objectiveAnchor.score` as-is for `ObjectiveAnchorScore` inputs while numeric inputs are rounded/clamped; normalize both branches unless the API intentionally trusts score objects as already canonical.
  • packages/gittensory-engine/src/objective-anchor.ts:191 should consider filtering `unknown` out of the aggregated `changeKinds` when concrete kinds exist, e.g. by applying that rule inside `combineFeatures`.
  • packages/gittensory-engine/src/pairwise-calibration.ts:89 should add a small `finitePositiveInteger` helper for `maxAttempts` and cover `0`, negative, `NaN`, and `Infinity` in tests.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #3013
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 90 registered-repo PR(s), 49 merged, 8 issue(s).
Contributor context ✅ Confirmed Gittensor contributor joaovictor91123; Gittensor profile; 90 PR(s), 8 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: joaovictor91123
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 90 PR(s), 8 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Await review-lane availability.
  • Refresh registry data or choose a registered active repo.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gittensory approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit a2dcf6a into JSONbored:main Jul 4, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(miner): pairwise-judge scoring layer with order-swap bias defense

1 participant