host-service never re-registers after the relay answers 4403 not_registered, so a running host stays invisible until restarted #8445

Activity

1. github-actions commented on Oct 13, 2026

   

   github-actionsBot

   on Oct 13, 2026 – with GitHub Actions

   Contributor

   More actions

   Summary

   The reporter's host-service registered fine at boot and then lost its cloud access row while it was still running. After that, the relay refused every control connect with 4403 forbidden … host not registered to this account. The process kept reconnecting every ~5 s for five days and never called host.ensure again. A manual superset stop && superset start re-registered it at once. The report holds up against the code: registration runs once per process, and nothing on the reconnect path treats a 4403 differently from any other drop. Whatever removed the row, the host has no way to recover without a restart, and superset status keeps reporting it as registered the whole time.

   Affected code

   • packages/host-service/src/tunnel/connect.ts:58-92: host.ensure runs in a retry loop only until the first success. It then builds the TunnelClient, returns it, and is never called again for the life of the process. The comment at connect.ts:65-66 states this design ("Registration runs once per process and a restart re-registers").
   • packages/host-service/src/tunnel/tunnel-client.ts:154-168: the close listener only logs, and partysocket reconnects with maxReconnectionDelay: 5_000 (tunnel-client.ts:124-129). A 4403 is treated like any transport drop. That gives the steady ~5 s 4403 loop in the logs, even though RELAY_CLOSE.forbidden is documented as "retry slowly" (packages/shared/src/tunnel-protocol.ts:41-42). TunnelClientOptions (tunnel-client.ts:51-62) has no API client and no callback, so the client cannot re-register even if it wanted to.
   • packages/host-service/src/tunnel/registration-state.ts:26-30: recordRegistrationSuccess() sets registered = true at boot. Nothing ever resets it, so health.check / superset status keep reporting the host as registered while the relay rejects it.
   • Relay side: apps/relay/src/index.ts:125-137 checks access on every control connect through checkHostAccessForRegister (apps/relay/src/access.ts:75-89). That calls host.checkAccess (packages/trpc/src/router/host/host.ts:240-278), which only looks for a v2_users_hosts row for (user, org, machine). If the row is missing, the result is not_registered, and the 4403 reason is the text from accessDenialMessage (access.ts:38-39). The reason travels only as free text; 4403 is also used for not_in_org and invalid_host.

   What could have removed the row

   checkAccess and host.list both depend on v2_users_hosts, so the trigger was the loss of that row (or of the v2_hosts row it belongs to). I found these deletion paths:

   • An explicit host delete by a host owner or org owner: packages/trpc/src/router/host/management.ts:238-246.
   • Removal of one user's access to a host: management.ts:~370 (allowed only when another owner remains).
   • Member removal: packages/db/src/utils/member-removal.ts:95-113. This deletes all of the member's v2_users_hosts rows in the org, plus any hosts left without an owner.
   • Account purge: packages/trpc/src/lib/account-purge/purgeAccount.ts.

   The reporter is still a member, so member removal is unlikely unless they were removed and then re-invited. I can't tell which path ran from the code alone. Server logs or the nudge(…, "hosts") events around 2026-10-08 15:39Z would show it. The 4401 just before it is probably unrelated: the relay sends 4401 for an expired JWT, and the next connect with a fresh token reached the access check.

   Reproduction

   1. Start a host-service (superset start --daemon) and confirm it appears in superset hosts list.
   2. Delete the host's access row, for example with the host delete action in Settings → Hosts or by removing your own v2_users_hosts row in a dev DB.
   3. Watch the host-service log. You get repeated control closed { code: 4403, reason: "Forbidden: host not registered…" } about every 5 s, no registered as host line, and superset status still reports the host as registered.

   A bun:test reproduction could start a local Bun.serve WebSocket server that accepts the upgrade and closes with 4403 and the not_registered reason. Point a TunnelClient at it, then assert that a new onNotRegistered/reRegister callback (see below) is called once, before the next connect attempt. Today the test fails because no such hook exists. Pass the delays in rather than waiting on real timers, per the test guidelines.

   Suggested fix direction

   1. Add an option to TunnelClientOptions, for example onForbidden?: (reason: string) => Promise<void>. Call it from the close listener when event.code === RELAY_CLOSE.forbidden.
   2. In connect.ts, move the host.ensure call into a helper that the boot loop and that callback both use. On a not_registered close, call recordRegistrationFailure(...) so health.check reports the truth, then retry ensure with the existing REGISTER_RETRY_* backoff. Guard it with a single in-flight promise so repeated closes don't stack calls. The machineId/org don't change, so the existing tunnel and routing key can stay. It only needs control.reconnect() after ensure succeeds.
   3. Make the relay send a structured denial reason (a short fixed token, or a separate close code for not_registered) so the host doesn't have to string-match accessDenialMessage. A stale JWT not_in_org should probably refresh the token rather than call ensure.
   4. Separately, apply "retry slowly" to 4403: a longer backoff while forbidden would have cut the 2,095 refused connects.

   Design question for maintainers: automatic re-registration means a host that an owner (especially an org owner) deleted on purpose comes back by itself while its process is still running. That already happens on the next restart, but it would now be immediate. If that's unwanted, host.ensure would need a tombstone or "deleted" marker to refuse, and the host-service would then report registered: false instead of recreating the row.
2. RohithReddy20

   added a commit that references this issue on Oct 26, 2026

   fix(host-service): re-register the host when the relay refuses it

   ...

   This issue will close once commit 14956eb is merged into the 'main' branch.

   14956eb