Part of #5016. Depends on #5028 (isInstalled reliability). This is the direct, durable fix for the originally-reported "job queue pending is high, traffic is low" symptom — land this as early as possible after #5028.
Problem
This is the actual root cause of the reported queue-growth incident. backfillRegisteredRepositories (src/github/backfill.ts:395-463) and enqueueRepositoryOpenDataBackfill (src/github/backfill.ts:465-480) — the two entry points for open-data backfill (labels/open issues/open PRs/recent-merged PRs, src/github/backfill.ts:1969+) — both filter on repo.isRegistered. Any repo on the gittensor subnet's registry file gets swept into backfill every 30 minutes, REGARDLESS of whether this self-host instance ever installed the app on it.
Live-verified on edge-nl-01 (the dedicated self-host server): 15 of 18 gittensor-subnet repos were unrelated third-party miner repos, none installed on this instance, all still getting backfill-repo-segment jobs dispatched every cron cycle — the direct driver of the unbounded queue growth. Two compounding gaps made it worse (not in scope here, just context): unauthenticated fallback calls when no installation token exists bypass GitHub rate-limit admission tracking, and rate_limited status isn't in syncFreshnessSkipReason's backoff set, so a stuck repo gets retried every cron tick forever.
Target
Retarget both entry points from isRegistered to isInstalled. Consider renaming the job type backfill-registered-repos → backfill-installed-repos for clarity — #2 (fidelity repair) already re-enqueues this exact job type for repair, which is circular/confusing under the current name.
Requirements
- Backfill only ever runs for repos with
isInstalled=true.
- A repo that is subnet-registered but not installed never gets a segment job dispatched, on any cron tick, regardless of gittensor-plugin opt-in status.
- If renaming the job type: update every dispatcher/handler reference (
src/queue/job-dispatch.ts:72-115, the trigger at src/queue/processors.ts:4261-4264, src/index.ts's cron push) and the OpenAPI-documented internal endpoints (/v1/internal/jobs/backfill-registered-repos*) consistently — config-as-code parity applies here (see CLAUDE.md's contributing skill).
Deliverables
- Updated filters + optional rename, consistently applied.
- Regression test reproducing the live incident shape: N repos registered (gittensor-subnet), only a subset installed → only the installed subset gets segment jobs.
- Full local gate green, 99% patch coverage on the diff.
Operational note
Landing this code fix does not retroactively clean up rows already marked isRegistered=true on edge-nl-01 from before this fix — that's a one-time manual DB cleanup (already handled separately, not part of this issue's deliverable).
Part of #5016. Depends on #5028 (isInstalled reliability). This is the direct, durable fix for the originally-reported "job queue pending is high, traffic is low" symptom — land this as early as possible after #5028.
Problem
This is the actual root cause of the reported queue-growth incident.
backfillRegisteredRepositories(src/github/backfill.ts:395-463) andenqueueRepositoryOpenDataBackfill(src/github/backfill.ts:465-480) — the two entry points for open-data backfill (labels/open issues/open PRs/recent-merged PRs,src/github/backfill.ts:1969+) — both filter onrepo.isRegistered. Any repo on the gittensor subnet's registry file gets swept into backfill every 30 minutes, REGARDLESS of whether this self-host instance ever installed the app on it.Live-verified on edge-nl-01 (the dedicated self-host server): 15 of 18 gittensor-subnet repos were unrelated third-party miner repos, none installed on this instance, all still getting
backfill-repo-segmentjobs dispatched every cron cycle — the direct driver of the unbounded queue growth. Two compounding gaps made it worse (not in scope here, just context): unauthenticated fallback calls when no installation token exists bypass GitHub rate-limit admission tracking, andrate_limitedstatus isn't insyncFreshnessSkipReason's backoff set, so a stuck repo gets retried every cron tick forever.Target
Retarget both entry points from
isRegisteredtoisInstalled. Consider renaming the job typebackfill-registered-repos→backfill-installed-reposfor clarity — #2 (fidelity repair) already re-enqueues this exact job type for repair, which is circular/confusing under the current name.Requirements
isInstalled=true.src/queue/job-dispatch.ts:72-115, the trigger atsrc/queue/processors.ts:4261-4264,src/index.ts's cron push) and the OpenAPI-documented internal endpoints (/v1/internal/jobs/backfill-registered-repos*) consistently — config-as-code parity applies here (see CLAUDE.md's contributing skill).Deliverables
Operational note
Landing this code fix does not retroactively clean up rows already marked
isRegistered=trueon edge-nl-01 from before this fix — that's a one-time manual DB cleanup (already handled separately, not part of this issue's deliverable).