Webhook endpoint / DNS cutover plan with rollback for the self-host GPU host migration

[JSONbored]

Part of #4325.

Context

GitHub delivers webhooks to a fixed URL configured on the GitHub App. The cutover moment — the instant live traffic actually starts hitting the new host instead of the current one — is the highest-risk single step in this epic and needs its own explicit plan, separate from "the new host is ready" (which the earlier sub-issues establish).

Requirements

• Decide the cutover mechanism: a DNS/reverse-proxy layer that gets repointed (preferred — the GitHub App's configured webhook URL never changes, only what's behind it) vs. directly editing the GitHub App's webhook URL at cutover time. Write down which, and why.
• Define the rollback: if the new host misbehaves shortly after cutover, what is the exact, fast action to revert traffic to the current host? This must be tested, not just described — a documented rollback that has never been executed is not a validated rollback.
• The current host must remain fully able to resume serving traffic immediately after a rollback — nothing in earlier sub-issues should leave it in a state where it can't (e.g. don't stop its queue consumer or let its Postgres fall permanently behind before cutover is confirmed stable).

Deliverables

• Cutover mechanism chosen and documented, with the reasoning
• Rollback procedure written down AND executed at least once in a drill (not just on paper)
• Confirmed the current host can resume serving traffic within the rollback's stated time budget
• Cutover runbook reviewed against the actual current-host webhook configuration (App settings, not assumptions about them)

Expected outcome

A cutover that can be executed calmly, and reversed quickly and confidently if the new host underperforms in the first hours of real traffic.

Effort

M

[linear-code]

JSO-3308

[JSONbored]

Status check (2026-07-09): Cutover has clearly happened — the new host is processing real, live GitHub webhook traffic right now (confirmed via its own logs and continuously-growing database rows).

One correction to this issue's assumed mechanism: I found no DNS/reverse-proxy layer or Tailscale Funnel/Serve config on this host, and the app's port isn't publicly exposed at all — traffic appears to arrive via the central Orb relay/broker (orb_relay_drained events recurring in the logs), not a directly-edited GitHub App webhook URL or a host-level DNS repoint. Worth updating this issue's requirements to match the actual architecture rather than the DNS/webhook-URL mechanism it assumed.

Still outstanding, with no evidence found either way: whether a rollback was actually drilled (this issue explicitly requires it be executed, not just documented), and whether the old host could still resume traffic within a stated time budget. Leaving open.