fix(orb): bind identity to credentials instead of renameable logins or trusted request bodies - #9177
Merged
Merged
Conversation
…r trusted request bodies (#9121, #9125, #9126, #9131) Four contributor-scoped controls trusted a mutable, attacker-influenced identity signal instead of a stable, authenticated one: - The risk-control ingest path trusted a body-supplied instance_id against a fleet-wide shared token, letting any fleet member plant or retract another instance's published guarantee. Ingest now requires a per-instance credential minted at registration, stores guarantees per-instance (aggregated at read time), and only an explicit null retracts an arm. - The contributor blacklist, PR/issue open-item caps keyed on the renameable GitHub login alone. pull_requests/issues now persist the author's immutable id, and matching is id-when-present union login, so a rename can no longer clear a ban or a cap. - Fleet-operator trust keyed on login alone, so a released-and- reregistered handle could grant operator access. ADMIN_GITHUB_IDS binds trust to the session's immutable id when configured, with a behavior-preserving login-only fallback. - submitter_stats counted webhook passes, not submissions: no idempotency key meant every re-gate (or a third party's review comment) bumped a rival's counter. Recording is now idempotent per (project, submitter, pull_number, outcome) and the burst signal reads a decaying window instead of an all-time total.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | c9585cf | Commit Preview URL Branch Preview URL |
Jul 27 2026, 05:48 AM |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
loopover-orb
Bot
added
the
gittensor:bug
❌ 7 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
4 tasks
JSONbored
added a commit
that referenced
this pull request
Jul 27, 2026
…ction, pool decisionAccuracy (#9068) orb_signals ingest stored any object-shaped risk_control payload verbatim, so a registered instance with a stale or misconfigured alpha (or an outright refused calibration) could reach the public guarantee unchecked. handleOrbIngest now runs the payload through validateCalibrationPayload (status === "calibrated", alpha/ lambda/coverage in range, nAtLambda clearing the zero-error floor for its own alpha/delta) before it reaches orb_risk_control_arms at all. gamingPatternFlags compared each eligible instance against the fleet median, which is structurally unfireable below 3 eligible instances (an instance IS the median at n=1) and can never flag "low reversal" once the fleet's own reversal-rate median is exactly 0 (a common, healthy-fleet case) since a fraction of zero can never be undercut. computeFleetAnalytics now gates detection on eligible.length >= 3 (surfaced via the new gamingDetectionEligible field) and falls back to an absolute reversal-rate floor when the fleet median is zero. fleet.decisionAccuracy published the per-instance MEDIAN while accuracyCiPct (public-stats.ts) is a Wilson interval over the POOLED counts — different estimands that only coincide at equal per-instance volumes. fleet.decisionAccuracy now publishes the pooled proportion directly (the same population the interval describes); the per-instance median survives as the new decisionAccuracyMedian diagnostic field. The underlying per-instance last-writer-wins fleet-key bug this issue also describes was already fixed by #9177 (orb_risk_control_arms is keyed per instance_id/arm and public-stats already aggregates across registered instances at read time) — this change covers the remaining validation-before-publish and detection-floor gaps.
JSONbored
added a commit
that referenced
this pull request
Jul 27, 2026
…ntee's fleet integrity (#9228) * fix(review): split threshold-refusal outcomes and Bonferroni-correct calibration delta (#9048, #9066) calibrateActThreshold returned "insufficient_labels" from two branches with different "have" semantics: a genuine label shortfall (have = total pairs) and a residual high-confidence stratum too small to certify despite ample total labels (have = that stratum's size). The latter now returns a distinct "no_certifiable_threshold" status carrying totalPairs/bestN/bestLambda/bestUpperBound, rendered through its own message and a distinct risk_control_no_certifiable_threshold audit event so the label burn-down no longer conflates "needs labels" with "needs a better error rate". Separately, the ascending-lambda scan reported whichever of K observed-confidence candidates passed first without correcting for testing K of them — the advertised 1-delta confidence overstated what the scan actually delivered. Each candidate is now tested at a Bonferroni-split delta/K, so the certified lambda is valid at the full, originally-advertised delta regardless of which candidate passes. Chosen over a true fixed-sequence rewrite as the smaller, safer diff against the existing ascending scan. Also adds AND dr2.action = dal.verdict to the calibration join (latent-risk hardening: today's data is not mis-joined, but a later HOLD/MERGE record on the same PR could otherwise shadow the acted CLOSE record a label adjudicates) and tags each calibration pair's provenance (backfilled vs live, via the backfill's configDigest sentinel) so a published guarantee can later say how much of its evidence is reconstructed history. * fix(orb): validate risk-control payloads at ingest, floor gaming detection, pool decisionAccuracy (#9068) orb_signals ingest stored any object-shaped risk_control payload verbatim, so a registered instance with a stale or misconfigured alpha (or an outright refused calibration) could reach the public guarantee unchecked. handleOrbIngest now runs the payload through validateCalibrationPayload (status === "calibrated", alpha/ lambda/coverage in range, nAtLambda clearing the zero-error floor for its own alpha/delta) before it reaches orb_risk_control_arms at all. gamingPatternFlags compared each eligible instance against the fleet median, which is structurally unfireable below 3 eligible instances (an instance IS the median at n=1) and can never flag "low reversal" once the fleet's own reversal-rate median is exactly 0 (a common, healthy-fleet case) since a fraction of zero can never be undercut. computeFleetAnalytics now gates detection on eligible.length >= 3 (surfaced via the new gamingDetectionEligible field) and falls back to an absolute reversal-rate floor when the fleet median is zero. fleet.decisionAccuracy published the per-instance MEDIAN while accuracyCiPct (public-stats.ts) is a Wilson interval over the POOLED counts — different estimands that only coincide at equal per-instance volumes. fleet.decisionAccuracy now publishes the pooled proportion directly (the same population the interval describes); the per-instance median survives as the new decisionAccuracyMedian diagnostic field. The underlying per-instance last-writer-wins fleet-key bug this issue also describes was already fixed by #9177 (orb_risk_control_arms is keyed per instance_id/arm and public-stats already aggregates across registered instances at read time) — this change covers the remaining validation-before-publish and detection-floor gaps. * fix(review): disambiguate the published guarantee's coverage and validate it before serving (#9050) readGuarantee published "coveragePct" as if it were a share of all closes, adjacent to a different fleetAccuracy.coveragePct that IS a share of all decided signals -- one word, two denominators. The guarantee's own field is actually the share of the arm's AI-JUDGED sub-population the threshold covers (loadCalibrationPairs can only join a confidence to decisions an AI-judgment blocker ran on, a minority of real closes). Renamed to aiJudgedCoveragePct and the homepage string now names the sub-population explicitly instead of leaving a bare percentage next to its sibling. readGuarantee also now re-validates every stored orb_risk_control_arms row through risk-control.ts's validateCalibrationPayload (defense in depth alongside the ingest-side check from the companion #9068 fix) and walks all registered rows for an arm in nAtLambda-descending order instead of trusting only the top one, so a single malformed or stale peer can no longer hide a good row behind it. Each calibration pair now carries whether it's backfilled (the 2026-07 calibration-corpus backfill's configDigest sentinel) or live; calibrateActThreshold surfaces the split as backfilledPairs, and the public guarantee renders it as backfilledPct so a guarantee resting mostly on reconstructed history says so. Regenerated apps/loopover-ui/public/openapi.json for the schema rename/nullability. * style(ui): run prettier over the guarantee coverage rename/test additions npm run ui:lint's format:check caught unformatted lines from the #9050 fleetAccuracy.guaranteed rename in proof-of-power-stats-model.ts and its new test.
This was referenced Jul 27, 2026
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four contributor-scoped controls trusted a mutable, attacker-influenced identity signal instead of a stable, authenticated one. This PR binds each to something the subject cannot unilaterally change or forge.
Closes #9121
Closes #9125
Closes #9126
Closes #9131
Details
/v1/orb/ingest) trusted a body-suppliedinstance_idchecked only against the shared fleet-wideORB_INGEST_TOKEN, so any fleet member could plant or (via an absent arm) retract another registered instance's published ≥95% guarantee.POST /v1/internal/orb/instances/registernow mints a per-instance credential (returned once, only its hash persisted); the ingest path requires it before accepting arisk_controlwrite, guarantees are stored per-instance (orb_risk_control_arms, replacing the sharedsystem_flagscell) and aggregated at read time preferring the largest sample size, and only an explicitnullretracts an arm — an absent key is a no-op.pull_requests/issuesnow persistauthor_github_id(the immutable id) at upsert; blacklist matching and both cap queries are id-when-present union login, so a rename no longer clears either control.ADMIN_GITHUB_LOGINS) keyed on login alone, so a released-and-re-registered GitHub handle would grant operator access.ADMIN_GITHUB_IDS, when configured, binds trust to the session's immutablegithubUserId; a matching login with a non-matching id is denied. Unset falls back to the existing login-only behavior, unchanged.submitter_stats.submissionscounted webhook passes, not submissions: no idempotency key meant every re-gate of a held PR (a push, a body edit, or a third party's review comment) bumped the counter again, and reputation was read frompull_request_review*events at all. Recording is now idempotent per(project, submitter, pull_number, outcome)via a newsubmitter_outcome_logtable, review-family events no longer feed the reputation signal, and the burst-detection aggregate reads a decaying 90-day window instead of an all-time total.Scope
Two new migrations (
0187_orb_instance_credentials.sql,0188_pull_request_issue_author_github_id.sql,0189_submitter_outcome_log.sql),scripts/check-schema-drift.tsallowlist updated for the two new raw-SQL tables,apps/loopover-ui/src/lib/selfhost-env-reference.tsregenerated for the newORB_COLLECTOR_INSTANCE_SECRETenv var.Validation
npm run typecheck— cleannpm run db:migrations:check/npm run db:schema-drift:check— cleannpm run selfhost:env-reference:check— cleannpm run manifest:drift-check/npm run ui:openapi:check/npm run coverage-boltons:check— cleanvitest run --coverageacross every changedsrc/**file (auth/security.ts,orb/ingest.ts,review/{submitter-reputation,reputation-wire,public-stats}.ts,services/control-panel-roles.ts, plus the touched slices ofdb/repositories.tsandqueue/processors.ts) — 100% line coverage on every changed line, only pre-existing untouched branches remain uncoverednpm run test:ci/npm auditwere not run locally — left to CINote
One pre-existing, unrelated test failure was observed after rebasing onto the now-updated
main(api.test.ts > serves installation repair diagnostics and refreshes installation health, failing on required-webhook-event health checks insrc/github/app.ts/backfill.ts— code this PR never touches). It reproduces identically without this branch's changes and appears tied to the same root cause as the already-flagged#9169regression.