Skip to content

epic(review): unrelated merges rebase and re-run CI on other PRs — confirmed, with no processing-order guarantee to fall back on #9500

Description

@JSONbored

Summary

Investigation of the reported behaviour: "some PRs re-run CI during the middle of their runs — whenever a separate unrelated PR is merged, it's retriggering CI on some or all PRs again, rebasing them before it's even necessary", and the expectation that ORB should work through PRs one at a time, oldest to newest.

The report is correct. The causal claim — unrelated merges cause rebases of other PRs — is confirmed by code path and production data. The serialization expectation is not how the system currently behaves, and no mechanism exists that could deliver it.

Verdict on each point

Report Verdict
Unrelated merges re-trigger CI on other PRs Confirmed. Trigger is an absolute staleness threshold in the readiness gate, delivered by the merge-wake fan-out
Rebases happen "before it's even necessary" Confirmed. The rebase precedes the CI wait, the gate verdict, the disposition plan, and the merge-train check
ORB works one PR at a time, oldest→newest Refuted. No focus concept exists; concurrency is 8; the ordering that does exist is defeated by 5 of 8 producers

Sub-issues

Evidence summary

Statistical (audit ledger, 7 days to 2026-07-27, JSONbored/loopover):

  • Rebases cluster within 60s of a merge at 17.5× the rate expected by chance (20k-resample null model); median delay-since-merge 169s vs null median 6,338s. Not explained by merge density — median inter-merge gap is 488s.
  • Delay-since-sweep, by contrast, is roughly uniform over 0–240s — the signature of independence from the sweep clock.
  • The sweep is structurally excluded: at 2026-07-24T14:06:15Z the sweep logged examined:12, flaggedPulls:[8273,8457], but the seven PRs rebased 8–20s later were 8301, 8302, 8342, 8337, 8340, 8431, 8453 — six of seven were never examined. The 15:40 burst had no loopover sweep at all.
  • Intra-burst offsets show the index * 10s fan-out stagger directly: 03.5, 13.7, 24.0, 31.9, 32.5 and 03.8, 15.8, 25.2, 31.8, 33.2, 34.6.

Mechanism:

  • All completed update_branch events carry the same reason: "default branch is N commits ahead of this PR's head (threshold 15)".
  • Only 34 of 625 merges (5.4%) produced any rebase — consistent with "roughly every fifteenth merge", not every merge.
  • Cohort {8301, 8302, 8337, 8340, 8342} was force-rebased 3× in 2.5 hours; merge counts between bursts were 15, 10, 5, 14 — the phase-locking signature of an absolute threshold with no jitter.

Correction on record

Two earlier analyses disagreed about where staleBaseAheadByThreshold = 15 comes from; both were wrong. It is not a code default and not a repository_settings row — repository_settings has rows only for awesome-claude and metagraphed, both NULL, and no row for JSONbored/loopover. The value is set in the global private config, /opt/loopover/loopover-config/.loopover.yml:51 (source: selfhost_private_global).

This matters for the mitigation: because it is the global layer, clearing it affects all three repos, so a per-repo override is needed to scope it to loopover.

Regression window: production update_branch count was 0 on 2026-07-20 and begins on 07-21, matching both the code path landing (#7686) and the oldest config backup containing the setting (…-20260721T111127Z).

Recommended order of work

  1. review: readiness gate force-rebases speculatively — unrelated merges cancel and restart CI on a phase-locked cohort of PRs #9497 — move the staleness rebase to the merge boundary. This alone eliminates the reported behaviour, because a rebase then only happens to the PR about to merge.
  2. queue: no 'current focus' concept exists, and oldest-first ordering is defeated by 5 of 8 regate producers omitting prCreatedAt #9499 Phase 1 — thread prCreatedAt, add the missing coalesce key, fix the head-SHA-keyed repair budget. Low risk, and may deliver enough of the desired ordering on its own.
  3. review(hardening): update_branch has no non-retryable failure class and no per-class metric (workflow-permission cause resolved 2026-07-24) #9498 — stop retrying impossible rebases.
  4. queue: no 'current focus' concept exists, and oldest-first ordering is defeated by 5 of 8 regate producers omitting prCreatedAt #9499 Phase 2 — an explicit focus gate, only if still wanted after measuring the effect of 1–2. It is a genuine behaviour change with a starvation cost, not just a tightening.

Two self-feeding loops worth noting

  • Rebase → repair priority → rebase. A successful update_branch mints a new head SHA, so lastPublishedSurfaceSha !== headSha makes the PR surface-repair priority (processors.ts:1194-1195), which bypasses both the 2-minute freshness guard and #never-endless-reregate (agent-sweep.ts:148-151, 172-174). It is re-gated within one sweep tick and re-checks aheadBy.
  • The repair budget resets on every rebase, because isRegateRepairExhausted keys on repo#pr#headSha (processors.ts:1121-1123) — the exact bug already fixed for the fresh-rebase counter by keying on PR number (:4787-4790).

Both are addressed in the sub-issues; recorded here because they explain why the behaviour is self-sustaining rather than a one-off burst.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions