perf(scheduler): phase-spread and backpressure the whole hourly job fan-out

[JSONbored]

Parent: #1936

Problem

The self-host stack shows a recurring GitHub rate-limit + queue-backlog spike "every hour on the hour." The root cause is temporal, not (only) duplication: enqueueScheduledJobs (src/index.ts:91-190) keys every cadence off minute === 0, so ~10 scheduled job types — several of them per-repo fan-outs — enqueue at the same instant. This is distinct from #1942 (which targets duplicate work and terminal-state jitter); even with perfect dedup, the top-of-hour tick creates a burst that drains at max worker rate for minutes and trips GitHub's secondary (burst) rate limit.

Compounding factors:

• No cross-fan-out phase offset. Each per-repo fan-out staggers internally (src/queue/processors.ts: signal index*10 cap 600, regate index*10 cap 600, rag index*30 cap 900, backfill index*45|15 cap 900), but all start at delay 0 on the same tick with overlapping windows, so repo[0] of every type + all singleton jobs coincide and one repo is hit by multiple fan-outs at once.
• Tail thundering-herd. Math.min(index*step, cap) collapses every repo past cap/step onto the same max-delay instant instead of continuing to spread.
• Un-gated hourly jobs. Only agent-regate-sweep and backfill-registered-repos yield to shouldWaitForGitHubRateLimit(MAINTENANCE_RESERVED_HEADROOM) + regate backlog (src/index.ts:121-152). refresh-registry/refresh-scoring-model/refresh-upstream-drift, rollup-product-usage, generate-signal-snapshots, build-*, file-upstream-drift-issues, rag-index-repo enqueue regardless of budget.

Requirements

• Spread the hourly enqueue across the available cron slots so the top-of-hour instant is not a burst.
• Give each scheduled job type (and each per-repo fan-out item) a deterministic phase offset so different fan-outs do not collide and the same repo is not multiply-hit at one instant.
• Gate all scheduled fan-outs behind the maintenance rate-limit headroom, not just sweep + backfill; a low bucket should defer non-urgent maintenance uniformly.
• Replace the min(index*step, cap) clamp with round-robin/modulo slot assignment so the tail keeps spreading at scale.
• Keep webhook-triggered work unaffected (it drives timely reviews and must never pre-yield).

Deliverables

• Deterministic phase-slot assignment for scheduled job types and per-repo fan-out items (hash → slot), applied in src/index.ts enqueue and/or the processors.ts fan-outs.
• Extend the maintenance-headroom backpressure check to every scheduled fan-out type.
• Fix the tail clamp so per-repo delay continues to spread past cap.
• Tests: burst-shape assertions (no all-at-minute-0 fan-out), phase-offset determinism, headroom-gating for each job type, tail-spread beyond the old cap.

Acceptance criteria

• Under normal load, the top-of-hour tick no longer creates an avoidable queue spike or a secondary-rate-limit event.
• With N repos, per-repo fan-out jobs are spread across time (no max-delay thundering herd).
• When the shared REST budget is low, all non-urgent scheduled fan-outs defer; webhooks are unaffected.
• No behavior change to what work runs — only when it is enqueued.

Expected outcome

The self-host queue stays smooth across the hour instead of spiking on the hour, eliminating the recurring rate-limit stalls without dropping any scheduled work.

[linear-code]

JSO-2030

[JSONbored]

Audit note: the highest-leverage form of this is small — a deterministic-jitter helper already exists (deterministicJitterMs, src/selfhost/queue-common.ts:540-548); it is currently applied only on process-start/recovery/rate-limit paths, never to the fresh cron enqueue at src/index.ts:189 (await Promise.all(jobs.map((job) => env.JOBS.send(job))) sends every job with no delaySeconds). The fix is to compute delaySeconds = deterministicJitterMs(job.type, INTERVAL)/1000 per hourly/full-sync job (and hash(repoFullName) -> slot for per-repo fan-out children) at the send site. The 10-agent audit ranked this #1 of 17 by leverage-per-effort and a prerequisite for budget-gating to function (a gate cannot throttle work that all arrives before any drains). See the full table on #1936.