feat(orb): risk-control thresholds with actuation, per-repo calibration, and the public guarantee - #8848
Merged
Merged
Conversation
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
loopover-orb
Bot
added
the
gittensor:bug
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-26 13:15:42 UTC
Review summary Blockers
Nits — 4 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | e2f08a4 | Commit Preview URL Branch Preview URL |
Jul 26 2026, 01:21 PM |
JSONbored
changed the title
…ccuracy mechanism Fixed-sequence calibration (Learn-then-Test; Trust or Escalate, ICLR 2025) of the per-arm act/hold threshold over the human-adjudicated labels: sweep lambda in DESCENDING-coverage order, exact Clopper-Pearson upper bound per candidate, stop at the first certification -- then with probability >= 1-delta, P(decision wrong | confidence >= lambda) <= alpha. No distributional assumptions. Separate Neyman-Pearson arms (close alpha=0.015, merge alpha=0.002 -- a wrong merge costs more). Advances #8835. Honesty guards, each pinned by a test: - insufficient labels is a REFUSAL, never a degraded guess: even a zero-error set cannot certify alpha until n >= ln(delta)/ln(1-alpha) (598 clean labels at alpha=0.005) -- and a passing-but-tiny high-confidence clique refuses too - 'uncertain' adjudications are excluded from both sides (the rubric's contract); rule-only decisions (no confidence) cannot join a confidence-thresholded guarantee and are skipped - a stale guarantee is a lie: an under-powered recalibration RETRACTS the published lambda and audits the label burn-down (have/needed) - the coverage-descending sweep direction matters: conservative-first dies on sample-size POWER at small-n candidates, not on errors -- documented in the module Daily recalibration tick (07:00 UTC, flag LOOPOVER_RISK_CONTROL, default OFF, self-host only, stale-queued-job re-check). CONSULT-ONLY deliberately: ai_review_close_confidence already has an automatic writer (backtest-gated knob loosening #8121/#8158), and two auto-writers on one knob need an explicit precedence rule first -- that actuation decision is the tracked remainder on #8835.
…ic guarantee Completes #8835 end to end and resolves #8849's precedence question in the same change, per the decided objective (maximum autonomy, minimum standing human involvement): - ACTUATION with the decided precedence: among the AUTOMATIC writers of the AI close-confidence floor, a live calibrated lambda outranks the backtest-gated knob loosening; retraction restores the loosening chain with no human step in either direction. An EXPLICIT per-repo gate.aiReview.closeConfidence still wins over both -- operator config-as-code outranks every automatic writer. The low-confidence hold names its floor's source (calibrated risk-control threshold vs configured floor) so a held contributor sees why. - PER-REPO calibration where a repo's own labels clear the floor (repo-scoped flag keys, independently retractable; the actuation read prefers the repo key, global fallback). - PUBLIC GUARANTEE: the exporter ships live global calibrations in the fleet payload; ingest stores them ONLY for registered instances (the strongest homepage claim must not be plantable via open ingest) and retracts arms the sender stops publishing; fleetAccuracy.guaranteed carries per-arm {alpha, lambda, coveragePct, n}; the hero hint reads 'closes >=98.5% guaranteed at N% coverage' while -- and only while -- a guarantee is live. - BUDGETS as instance-level env config with clamps (close 0.015, merge relaxed 0.005 per the decided objective -- the 0.002 draft needed ~1,497 labels, a year of adjudication for the last 3x of strictness; 0.005 keeps a real 3x asymmetry and is reachable), delta 0.05. Closes #8835. Closes #8849.
JSONbored
force-pushed
the
feat/risk-control-thresholds
branch
from
July 26, 2026 13:19
95cc0bb to
e2f08a4
Compare
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
…rd per PR decision_records keys one row per (PR, head sha), so a PR reviewed across several pushes accumulates several rows; joining labels on target_id alone fanned one adjudicated label into N pairs at different confidences, breaking the one-label-one-trial contract the Clopper-Pearson guarantee depends on. The join now selects the latest record per target (created_at DESC, id DESC tie-break), matching the acted decision the label adjudicates.
JSONbored
added a commit
that referenced
this pull request
Jul 26, 2026
…s enforce Both of these are broken on main right now and fail every PR branched from it, including this one: 1. .release-please-manifest.json still pinned packages/loopover-engine at 3.15.0 while its package.json says 3.15.1 -- the manual bump in #9107 (which the engine twin-parity guard required) never synced the release manifest, so release-manifest:sync:check has failed on every commit since. Regenerated via `npm run release-manifest:sync`. 2. apps/loopover-ui's prettier gate failed on two files last touched by #8848 (proof-of-power-stats-model.ts and proof-of-power-stats.test.tsx) -- two over-long object literals prettier wants wrapped. Applied `prettier --write`; pure formatting, no behavior change.
JSONbored
added a commit
that referenced
this pull request
Jul 26, 2026
…erdicts, fix holdout misattribution (#9110) * fix(orb): retry the disposition on lock contention, retry inconclusive verdicts, and stop misattributing holdout holds Three disposition-integrity fixes: #9025 -- maybeRunAgentMaintenance returned silently when it lost the per-PR actuation lock: the job completed "successfully", nothing re-queued the disposition, and no audit row recorded that a planned action was abandoned. That silently amplified every restart incident -- the recovered job re-ran, hit its own dead predecessor's orphaned lock, and lost the disposition a SECOND time with no trace at all. Now throws PrActuationLockContendedError (the same contract review-evasion.ts's withPrActuationLock already used for this exact condition; the queue honors its 5s retryAfterMs via consumingRetryDelayMs) and records a named audit event. Both maintenance call sites' catch handlers now re-throw retryable/rate-limit errors instead of logging-and-dropping them, matching the review pipeline's own propagation contract; a plain non-retryable failure is still swallowed and logged. #9019 -- `cacheable=0` conflates two unrelated things: a DYNAMIC review context (grounding/RAG), where the verdict is conclusive but not durable across time, and a genuinely INCONCLUSIVE verdict (a provider outage, a consensus-disputed roll). Because published rows were exempt from the non-cacheable cooldown, a transient outage verdict became FINAL for that head the moment it surfaced -- the bot never retried, directly contradicting the finding's own "re-evaluates on the next update" text, while a green PR gave the contributor no reason to push the commit that would force one. Worse, the head-AGNOSTIC one-shot lookup pinned that same outage verdict across ALL future heads, so a contributor pushing new code could not escape it either. Records the review's own verdict as metadata.inconclusive and keys both behaviors on it: the publish exemption still applies to dynamic-context rows (#2119 unchanged) but not to inconclusive ones, and the one-shot cadence skips inconclusive rows entirely -- that PR never got its one real shot. The cooldown still bounds retries to at most one attempt per window. #9040 -- every "auto-action held by precision circuit breaker" audit row was wrong. agentHoldAuditDetail inferred the breaker purely from "a terminal action was planned but is not in the final plan", but the call site passes the POST-HOLDOUT plan, so every ε-holdout adjudication hold (#8831) was attributed to a breaker that had never engaged -- 6 of 6 live rows paired 1:1 (within 20ms) with decision_audit_holdout events while system_flags contained no engaged breaker at all. Each transform now REPORTS its own engagement, derived from its own before/after pair, and the holdout gets its own reason string; the residual set-difference case returns an honest generic reason instead of a false specific attribution. Closes #9019 Closes #9025 Closes #9040 * fix(ci): repair two pre-existing main breakages the drift/format gates enforce Both of these are broken on main right now and fail every PR branched from it, including this one: 1. .release-please-manifest.json still pinned packages/loopover-engine at 3.15.0 while its package.json says 3.15.1 -- the manual bump in #9107 (which the engine twin-parity guard required) never synced the release manifest, so release-manifest:sync:check has failed on every commit since. Regenerated via `npm run release-manifest:sync`. 2. apps/loopover-ui's prettier gate failed on two files last touched by #8848 (proof-of-power-stats-model.ts and proof-of-power-stats.test.tsx) -- two over-long object literals prettier wants wrapped. Applied `prettier --write`; pure formatting, no behavior change.
JSONbored
added a commit
that referenced
this pull request
Jul 26, 2026
…ses (#9111) * fix(orb): retry the disposition on lock contention, retry inconclusive verdicts, and stop misattributing holdout holds Three disposition-integrity fixes: #9025 -- maybeRunAgentMaintenance returned silently when it lost the per-PR actuation lock: the job completed "successfully", nothing re-queued the disposition, and no audit row recorded that a planned action was abandoned. That silently amplified every restart incident -- the recovered job re-ran, hit its own dead predecessor's orphaned lock, and lost the disposition a SECOND time with no trace at all. Now throws PrActuationLockContendedError (the same contract review-evasion.ts's withPrActuationLock already used for this exact condition; the queue honors its 5s retryAfterMs via consumingRetryDelayMs) and records a named audit event. Both maintenance call sites' catch handlers now re-throw retryable/rate-limit errors instead of logging-and-dropping them, matching the review pipeline's own propagation contract; a plain non-retryable failure is still swallowed and logged. #9019 -- `cacheable=0` conflates two unrelated things: a DYNAMIC review context (grounding/RAG), where the verdict is conclusive but not durable across time, and a genuinely INCONCLUSIVE verdict (a provider outage, a consensus-disputed roll). Because published rows were exempt from the non-cacheable cooldown, a transient outage verdict became FINAL for that head the moment it surfaced -- the bot never retried, directly contradicting the finding's own "re-evaluates on the next update" text, while a green PR gave the contributor no reason to push the commit that would force one. Worse, the head-AGNOSTIC one-shot lookup pinned that same outage verdict across ALL future heads, so a contributor pushing new code could not escape it either. Records the review's own verdict as metadata.inconclusive and keys both behaviors on it: the publish exemption still applies to dynamic-context rows (#2119 unchanged) but not to inconclusive ones, and the one-shot cadence skips inconclusive rows entirely -- that PR never got its one real shot. The cooldown still bounds retries to at most one attempt per window. #9040 -- every "auto-action held by precision circuit breaker" audit row was wrong. agentHoldAuditDetail inferred the breaker purely from "a terminal action was planned but is not in the final plan", but the call site passes the POST-HOLDOUT plan, so every ε-holdout adjudication hold (#8831) was attributed to a breaker that had never engaged -- 6 of 6 live rows paired 1:1 (within 20ms) with decision_audit_holdout events while system_flags contained no engaged breaker at all. Each transform now REPORTS its own engagement, derived from its own before/after pair, and the holdout gets its own reason string; the residual set-difference case returns an honest generic reason instead of a false specific attribution. Closes #9019 Closes #9025 Closes #9040 * fix(ci): repair two pre-existing main breakages the drift/format gates enforce Both of these are broken on main right now and fail every PR branched from it, including this one: 1. .release-please-manifest.json still pinned packages/loopover-engine at 3.15.0 while its package.json says 3.15.1 -- the manual bump in #9107 (which the engine twin-parity guard required) never synced the release manifest, so release-manifest:sync:check has failed on every commit since. Regenerated via `npm run release-manifest:sync`. 2. apps/loopover-ui's prettier gate failed on two files last touched by #8848 (proof-of-power-stats-model.ts and proof-of-power-stats.test.tsx) -- two over-long object literals prettier wants wrapped. Applied `prettier --write`; pure formatting, no behavior change. * fix(ci): fail closed on truncated CI reads and partial GraphQL responses Two silent-failure classes on the wrong-merge path, both in the live CI/review readers that gate every merge decision. #9051 -- a FAILED check-runs page fetch already set checkRunsIncomplete, but EXHAUSTING the 10-page cap with `rel="next"` still present did not: the loop just exited and reduceLiveCiAggregate treated a truncated set as complete. A red check on page 11+ was therefore invisible -> ciState "passed" -> planner reviewGood -> MERGE. The executor's act-boundary recheck calls the same function so it reproduced the wrong verdict rather than catching it, and the false "passed" was persisted into the durable cross-job CI cache. Fixed for both the check-runs and classic-status loops. Separately, the check-suites backstop -- the LAST gate before a commit is certified settled -- read page 1 only with no Link follow, so a first-party suite still running on page 2 never set anyPending; it now paginates and returns null (which the reducer already fails closed on) when its own cap is exhausted. The GraphQL twin's `checkSuites` selection had no hasNextPage guard either, unlike its `contexts` sibling one line above; added. #9052 -- fetchLiveReviewThreadBlockers read `connection?.nodes` and returned [] without ever checking the GraphQL top-level `errors` array. GitHub's standard partial-failure shape under load is HTTP 200 with `reviewThreads: null` plus `errors`, which yielded [] -- indistinguishable from a genuinely thread-free PR -- so a maintainer's unresolved blocking thread was dropped from the findings and the gate could conclude success and merge over the open objection. Unlike a transport error (nothing read at all -> fail open, unchanged), a partial result means the answer is known-unreliable, so it now fails CLOSED with a synthetic blocker. The sibling readers in this file already guarded this; this one was the outlier. Same class, second instance: fetchLivePullRequestReviewDecision also had no errors check and returned undefined, which let the caller's `liveReviewDecision ?? pr.reviewDecision` substitute a STALE stored APPROVED for a read that failed -- so a PR later flipped to CHANGES_REQUESTED still merged. It now returns an explicit REVIEW_DECISION_UNREADABLE sentinel that survives the ?? fallback, matches no real enum value (so every === comparison is correctly false), and is checked by name at the one approval-queue site that would otherwise have read it as "confirmed no changes requested". Closes #9051 Closes #9052 Tests: 3 cap-exhaustion regressions (check-runs, statuses, check-suites), a GraphQL checkSuites-truncation guard, a 200-with-errors review-threads fail-closed test, two review-decision sentinel tests, and an approval-queue test proving an unreadable decision no longer clears a conflict-justified close. 100% coverage on all 88 added lines; 1368/1368 across the 12 affected suites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The complete risk-control system: distribution-free calibration, actuation under a decided precedence rule, per-repo thresholds, and the live guarantee on the public homepage. Closes #8835. Closes #8849.
The mechanism
Fixed-sequence calibration (Learn-then-Test; Trust or Escalate, ICLR 2025) over the human-adjudicated labels, per arm: sweep λ in descending-coverage order with an exact Clopper–Pearson bound per candidate, stop at the first certification — then with probability ≥ 1−δ, P(decision wrong | confidence ≥ λ̂) ≤ α. The sweep direction is load-bearing (conservative-first dies on sample-size power, not errors — found by the tests, documented in the module).
The decided precedence (#8849), implemented
Per the decided objective — maximum autonomy, minimum standing human involvement:
gate.aiReview.closeConfidencestill wins over both: operator config-as-code outranks every automatic writer.Everything else #8835 named
fleetAccuracy.guaranteedcarries per-arm{alpha, lambda, coveragePct, n}→ the hero hint reads "closes ≥98.5% guaranteed at N% coverage" while — and only while — a guarantee is live.uncertainexcluded both sides; retraction on power loss (stale guarantees lie); every read fails open to the static chain.Verification
TSC clean; 952 tests green across risk-control, wire, public-stats, ingest, collector, gate-check-policy, rules, github-commands, queue suites. Changed-line coverage: 0 uncovered statements/branches across all eight touched src files plus both new modules. Branding, schema-drift, and OpenAPI-parity checks green.