Skip to content

fix(review): meter tie-break judge budget - #3671

Merged
JSONbored merged 2 commits into
mainfrom
codex/fix-unmetered-ai-calls-in-tie-break
Jul 6, 2026
Merged

fix(review): meter tie-break judge budget#3671
JSONbored merged 2 commits into
mainfrom
codex/fix-unmetered-ai-calls-in-tie-break

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Closes #3679

Summary

  • The order-swapped dual-AI tie-break path (fix(review): add order-swapped stability check to the dual-AI tie-break pass #2997) performs additional judge calls — with retries and
    per-model fallback — that were never counted against the free daily AI neuron budget, letting a
    contributor spend extra provider usage beyond the configured limit by triggering repeated consensus
    disagreements.
  • Reserves a worst-case tieBreakAiCalls budget (2 judge calls x up to 3 attempts x a per-model
    fallback multiplier) and adds it to estimatedNeurons in runGittensoryAiReview
    (src/services/ai-review.ts), gated on mode === "block" && dual && combine === "consensus" — the
    only path that can actually reach the tie-break judge.
  • Two regression tests: the default two-distinct-models fallback multiplier (x2), and a self-host
    single-model configuration where a reviewer has no distinct fallback (primaryFallback === primary.model), exercising the x1 multiplier branch.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused — one source file, one test file.
  • Follows CONTRIBUTING.md; no site//CNAME/VitePress changes.
  • Linked issue: Closes #3679.

Validation

  • git diff --check
  • npm run actionlint — not run; no workflow files touched.
  • npm run typecheck (clean)
  • npm run test:coverage (full/unsharded) — not run locally; ran the affected test file
    (test/unit/ai-review.test.ts, 153 tests, all green) plus the full queue.test.ts integration
    suite (732 tests). Cross-referenced the exact new-diff line/branch ranges against a fresh
    coverage-final.json (--coverage.include='src/services/ai-review.ts'): 100% of the new
    statements and branches are covered (the original patch was missing the x1-fallback-multiplier
    branch; closed with a second new test). GitHub CI runs the full suite/gate on push.
  • npm run test:workers / npm run build:mcp / npm run test:mcp-pack — not run; nothing in
    those surfaces touched.
  • npm run ui:openapi:check / ui:lint / ui:typecheck / ui:build — not run; no API/schema or
    apps/gittensory-ui/** changes.
  • npm audit --audit-level=moderate — not run; no dependency changes.
  • New/changed behavior has regression tests for both branches of the new fallback-multiplier
    ternary, plus the existing quota-exceeded assertion pattern already used elsewhere in this file.

If any required check was skipped, explain why:

  • Local validation here was typecheck + the specific affected test files rather than a full local
    test:ci/test:coverage/npm audit pass, since GitHub CI runs the complete gate on push and
    re-running the whole suite by hand for every change is redundant.

Safety

  • No secrets, wallets, hotkeys, coldkeys, trust scores, or private scoring values are touched or
    exposed.
  • Not a public-facing text change (internal budget/neuron-estimation logic only).
  • Not an auth/CORS/session change; no negative-path tests needed for that reason.
  • Not an API/OpenAPI change.
  • Not a UI change (N/A) — no UI Evidence section included.
  • No changelog edit.

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 6, 2026
@loopover-orb

loopover-orb Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-06 04:11:47 UTC

2 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR adds a `tieBreakAiCalls` reservation (worst-case 2 judges × 3 attempts × fallback multiplier) to `estimatedNeurons` in `runGittensoryAiReview`, gated on `mode === "block" && dual && combine === "consensus"`, closing the gap where retried/fallback tie-break judge calls could spend AI budget uncounted. The multiplier logic (x2 for distinct fallback model, x1 when `primaryFallback === primary.model`) is tested with two focused regression tests hitting `quota_exceeded`, and the change is additive/small in scope. The main gap is that only the non-zero `tieBreakAiCalls` path is exercised — no test covers the `tieBreakAiCalls === 0` branch (e.g. non-consensus combine mode or non-block mode), which lines up with the reported codecov/patch shortfall.

Nits — 6 non-blocking
  • Neither new test exercises the `tieBreakAiCalls === 0` branch (e.g. `combine !== "consensus"` or `mode !== "block"`), which is likely why codecov/patch reports only 75% diff coverage on this file.
  • The literals `3` (max attempts) and `512` (judge token estimate) in src/services/ai-review.ts:1736 and :1746 are unexplained magic numbers — consider naming constants (e.g. `TIE_BREAK_MAX_ATTEMPTS`, `TIE_BREAK_TOKEN_ESTIMATE`) so the worst-case math is self-documenting.
  • The comment says '2 judge calls x up to 3 attempts x per-model fallback multiplier' but doesn't reference where the retry count of 3 or the 2-judge count come from in the tie-break implementation — a pointer/comment link to that code would help future readers verify the worst-case math stays in sync.
  • Add a test where `combine !== "consensus"` (or `mode !== "block"`) with a dual reviewer config to confirm `tieBreakAiCalls` stays 0 and the reservation doesn't over-count budget for paths that can't reach the tie-break judge.
  • Extract the `3` and `512` literals into named constants near the tie-break judge implementation so the reservation formula and the actual judge-call site can't silently drift out of sync.
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #3679
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 55 registered-repo PR(s), 47 merged, 456 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 55 PR(s), 456 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 55 PR(s), 456 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • No action.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.34%. Comparing base (ddf68c3) to head (b84aec0).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3671   +/-   ##
=======================================
  Coverage   93.34%   93.34%           
=======================================
  Files         315      315           
  Lines       32107    32111    +4     
  Branches    11769    11772    +3     
=======================================
+ Hits        29969    29973    +4     
  Misses       1507     1507           
  Partials      631      631           
Files with missing lines Coverage Δ
src/services/ai-review.ts 96.99% <100.00%> (+0.02%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 6, 2026
JSONbored added 2 commits July 5, 2026 19:12
The reserved worst-case tieBreakAiCalls budget had an untested branch:
a self-host reviewer pair with no distinct fallback model
(primaryFallback === primary.model) falls to the x1 multiplier instead
of x2, which the original single regression test never exercised.
@JSONbored
JSONbored force-pushed the codex/fix-unmetered-ai-calls-in-tie-break branch from 7071bcc to b84aec0 Compare July 6, 2026 02:18
@JSONbored
JSONbored merged commit 2eba2ae into main Jul 6, 2026
10 checks passed
@JSONbored
JSONbored deleted the codex/fix-unmetered-ai-calls-in-tie-break branch July 6, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(review): meter the order-swapped tie-break judge calls against the daily AI budget

1 participant