Skip to content

fix(queue): cap backlog-convergence re-review attempts per head SHA - #5859

Merged
JSONbored merged 1 commit into
mainfrom
fix/backlog-convergence-retry-cap
Jul 14, 2026
Merged

fix(queue): cap backlog-convergence re-review attempts per head SHA#5859
JSONbored merged 1 commit into
mainfrom
fix/backlog-convergence-retry-cap

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • sweepRepoBacklogConvergence re-selects any open PR whose public review surface never converged to its current head (lastPublishedSurfaceSha != headSha) — the same signal the main sweep's outage-repair path already guards with a per-SHA attempt cap. This sweep had no such guard: when a PR's gate-check finalize silently failed while its comment still published, the marker never advanced and this sweep re-dispatched a full agent-regate-pr job for it every ~30 minutes indefinitely.
  • Confirmed live on the three PRs an ops-anomaly detector flagged: the same sticky PR comment was edited 11–100 times over 1.3–23.5 hours, long after the head SHA had stopped changing (in one case, zero pushes for the entire window).
  • Extracted the existing repair-attempt-cap check (previously inlined in surfaceRepairPriorityPullNumbers) into isRegateRepairExhausted and shared it from both sweeps against the SAME per-(repo, PR, headSha) budget — deliberately not an independent counter, since both sweeps re-select on the identical signal and would otherwise double the wasted spend the cap exists to prevent. Passing repairHeadSha on backlog-convergence's dispatched jobs reuses the existing execution-time (not dispatch-time) attempt recording, so a deferred or dropped job still doesn't count against the cap.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • Linked issue — N/A. Maintainer-authored fix found via a production ops-anomaly detector, not a contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally — 853 test files passed, 0 failed (full unsharded run), including the 3 pre-existing #orb-retry-storm regression tests (verifying the extracted helper is behavior-preserving) plus 2 new tests covering the backlog-convergence half: a SHA that already exhausted the shared budget is not re-dispatched, and a fresh candidate is dispatched with repairHeadSha set and correctly charges the shared budget on execution.
  • npm run test:engine-parity — clean, unaffected (this change doesn't touch packages/loopover-engine).
  • npm audit --audit-level=moderate — clean, 0 vulnerabilities.
  • npm run test:workers / npm run build:mcp / npm run test:mcp-pack / npm run ui:* — not run, see note below.

If any required check was skipped, explain why:

  • This change is confined to src/queue/processors.ts and its unit tests — no Workers runtime, MCP package, API route, OpenAPI schema, or UI code is touched.

Safety

  • No secrets, wallets, hotkeys, coldkeys, trust scores, or reward values touched.
  • Auth/CORS/GitHub App/Cloudflare/session, API/OpenAPI/MCP, UI, and docs/changelog: not applicable, none touched.

UI Evidence

N/A — no UI/frontend/docs change.

Notes

  • Root cause was confirmed against real GitHub data on all three flagged PRs (comment edit history via GraphQL userContentEdits) before writing the fix, not from the ops-anomaly log alone.

sweepRepoBacklogConvergence re-selects any open PR whose public review
surface never converged to its current head (lastPublishedSurfaceSha !=
headSha) -- the same signal the main sweep's outage-repair path already
guards with a per-SHA attempt cap. This sweep had no such guard: when a
PR's gate-check finalize silently failed while its comment still
published, the marker never advanced and this sweep re-dispatched a full
agent-regate-pr job for it every ~30 minutes indefinitely.

Confirmed live on the three PRs an ops-anomaly detector flagged: the
same sticky PR comment was edited 11-100 times over 1.3-23.5 hours, long
after the head SHA had stopped changing (in one case, zero pushes for
the entire window).

Extract the existing repair-attempt-cap check (previously inlined in
surfaceRepairPriorityPullNumbers) into isRegateRepairExhausted and share
it from both sweeps against the SAME per-(repo, PR, headSha) budget --
deliberately not an independent counter, since both sweeps re-select on
the identical signal and would otherwise double the wasted spend the cap
exists to prevent. Passing repairHeadSha on backlog-convergence's
dispatched jobs reuses the existing execution-time (not dispatch-time)
attempt recording, so a deferred or dropped job still doesn't count
against the cap.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.12%. Comparing base (08d7049) to head (c9c6928).
⚠️ Report is 8 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5859   +/-   ##
=======================================
  Coverage   95.11%   95.12%           
=======================================
  Files         587      587           
  Lines       46509    46513    +4     
  Branches    14870    14871    +1     
=======================================
+ Hits        44238    44245    +7     
+ Misses       1515     1513    -2     
+ Partials      756      755    -1     
Flag Coverage Δ
shard-1 43.31% <100.00%> (-0.45%) ⬇️
shard-2 36.29% <0.00%> (+0.10%) ⬆️
shard-3 32.01% <0.00%> (-0.12%) ⬇️
shard-4 31.91% <50.00%> (-0.91%) ⬇️
shard-5 32.48% <0.00%> (+0.86%) ⬆️
shard-6 44.86% <0.00%> (+0.34%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/processors.ts 95.91% <100.00%> (+0.10%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 14, 2026
@loopover-orb

loopover-orb Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-14 20:15:54 UTC

2 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR extracts the existing per-SHA repair-attempt-cap check out of surfaceRepairPriorityPullNumbers into a shared isRegateRepairExhausted helper and applies it to sweepRepoBacklogConvergence, which previously had no cap and re-dispatched agent-regate-pr jobs indefinitely for PRs whose surface never converged. The refactor of the original inline check is behavior-preserving (same attempts/alreadyFlagged sequence, same audit-event side effects), and the new backlog-convergence usage filters exhausted candidates before dispatch while passing repairHeadSha so the budget is charged at job-execution time rather than dispatch time, avoiding double-counting for deferred/dropped jobs. Two new tests exercise the real dispatch and execution paths (not fabricated payloads) and confirm the shared-budget behavior end-to-end; CI is green across the full matrix.

Nits — 5 non-blocking
  • src/queue/processors.ts:1730 `pr.headSha!` in the Promise.all over allCandidates relies entirely on selectBacklogConvergenceCandidates's invariant that needsSurfaceConvergence PRs always have a truthy headSha — worth a defensive filter or an explicit type on the candidate list rather than a bare non-null assertion.
  • isRegateRepairExhausted issues two separate countRecentAuditEventsForActorAndTarget queries (attempts, then alreadyFlagged) sequentially per candidate; could be combined into one query keyed by event_type IN (...) for less DB round-trip overhead under a large backlog, though this mirrors the pre-existing pattern so it's not a regression.
  • The long, dense comment blocks (e.g. processors.ts:1055-1062, 1729-1735) are informative but could be trimmed to just the non-obvious 'why' per the repo's own comment convention.
  • Consider adding a lightweight assertion or dev-mode invariant check that selectBacklogConvergenceCandidates never returns a PR with an empty headSha, so the `!` assertion doesn't silently mask a future regression in that selector.
  • If backlog size grows large, batch the exhausted-check queries (e.g. one query per repo instead of one per PR) to bound DB calls during the sweep.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue — Link the relevant issue (for example `Closes #123`) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 45 registered-repo PR(s), 37 merged, 260 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 45 PR(s), 260 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence. LLM value judgment: significant — This closes a confirmed production incident (PRs re-dispatched every ~30 minutes for up to 23.5 hours) by sharing an existing, tested attempt-cap mechanism across both sweeps rather than inventing a parallel one, which is a well-targeted, narrow fix with real operational impact.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 45 PR(s), 260 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 14, 2026
@JSONbored
JSONbored merged commit 5851c07 into main Jul 14, 2026
17 checks passed
@JSONbored
JSONbored deleted the fix/backlog-convergence-retry-cap branch July 14, 2026 20:23
This was referenced Jul 14, 2026
This was referenced Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

1 participant