Flaky e2e: gateway-fleet reads a worker's capacity before the worker has left starting

What fails

e2e/gateway-fleet.test.ts > "gateway fleet > two workers join, status aggregates them, a kill disconnects one, drain flags the other" fails with AssertionError: expected 0 to be greater than 0 at expect(iosLimit(joined[0])).toBeGreaterThan(0). Seen once in CI on PR #409 (run 37477061691).

Cause: a race on main, not in a PR

The test waits only for both workers to be connected, then reads capacity from that first view. A worker that dials the gateway while its daemon is still starting answers status.get with health and host only (#398), so the gateway builds its view without capacity (worker-link.ts #rebuildView -> #refreshStarting). The view stays without capacity until the next refresh, and none is triggered when the worker becomes running: daemon.started is not a lease./device./install event (changesCapacityOrLeases), so the next read is a device or lease event or the 30s periodic tick. The test's iosLimit falls back to 0.

Evidence

On origin/main (37188b5), unmodified: 6 of 6 runs of the file pass under CPU load (16 busy loops on 12 cores), 5 of 5 unloaded.
On origin/main with await new Promise((r) => setTimeout(r, 1500)) as the first line of convergeStartup in src/daemon/main.ts (a slower startup): 3 of 3 runs fail with this exact assertion (2 tests failed each time).
The window is the worker's startup time, so anything that lengthens startup (PR feat(leasing): startup ends every lease whose device is not running #409 adds a serial read, doctor pass and lease reconcile) makes it likelier. It is not specific to that PR.

Two things to decide in triage

Product: should the gateway refresh a worker's view as soon as it leaves starting? Today a freshly joined worker takes no routed work for up to 30s after its startup ends if no device or lease event happens first (check FleetCoordinator/routing on a view with no catalog).
Test: waitForWorkers in the file should wait for capacity (and catalog) to be present before the test reads it.

Reproduction

The flaky e2e is fixed on main; the product gap is real, with another trigger. A gap: the expectation is a proposal.

src/gateway/service.test.ts > "builds the full view of a worker that was starting when first read, once its startup ends, before the next tick", on bug/414-repro:

pnpm exec vitest run --project unit src/gateway/service.test.ts -t "once its startup ends"

AssertionError: expected { connection: 'connected', …(7) } to match object { …(7) }
-   "capacity": { "ios": { "limit": 2 } }, "catalog": [...], "devices": [...],
-   "health": "running", "leases": [...], "queueDepth": 1,
+   "health": "starting",

Correction: the e2e never read a starting answer. A starting worker holds events.subscribe (dispatch.ts:312-314), which the gateway awaits after marking it connected (worker-link.ts:248-280). The old test read the view before the first refresh; #411 now waits for capacity:

c881500 + 1500 ms sleep in convergeStartup: e2e/gateway-fleet.test.ts  8 passed (8)

Root cause

src/gateway/worker-link.ts:253-280. A starting view is written only when startup outlasts WORKER_CALL_TIMEOUT_MS (10 s; startup's read alone can take a minute): events.subscribe times out and the refresh at :280 reads starting (:362-364). When startup ends, nothing reads the worker again. The worker answers the held subscribe then, but #withTimeout has discarded it, and daemon.started is emitted (server.ts:563) before that subscribe resumes. No capacity or catalog until a lease, device or install event, or the 30 s tick, as #398 assumed.

Simplest fix

On a subscribe timeout, keep the promise; once it resolves, store the handle and refresh. The file then passes 54/54:

const subscription = client.subscribeEvents((push) => this.#onWorkerEvent(push.event));
// in the catch, for WorkerCallTimeoutError:
void subscription.then((unsubscribe) => {
  this.#unsubscribeEvents = unsubscribe;
  void this.refresh({ includeCatalog: true });
}, () => undefined);

Pinned shape: the late subscribe answer means startup ended.

Alternatives rejected

Refresh on daemon.started: a gateway connected during startup never receives it.
Poll status.get while starting: extra round trips for a signal already sent.
A longer subscribe timeout: it bounds every call (D2/D3).

Risk

The late handler may run after close(); it must not keep a handle or refresh a closed or replaced link.
The fake worker's parking and event order come from server.ts:530-535, not a real 10 s startup.
The flaky-test label no longer matches anything on main.

Side findings

A comment in the daemon's startup says the gateway's first status.get parks on startup readiness, which it no longer does #420

Written by an agent.

fix(gateway): a worker that was starting when first read gets its ful…

refactor(gateway): the late subscribe refresh reads the catalog its v…

Navigation Menu

Repository navigation

Flaky e2e: gateway-fleet reads a worker's capacity before the worker has left starting #414

What fails

Cause: a race on main, not in a PR

Evidence

Two things to decide in triage

Reproduction

Root cause

Simplest fix

Alternatives rejected

Risk

Side findings

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Flaky e2e: gateway-fleet reads a worker's capacity before the worker has left starting #414

Description

What fails

Cause: a race on main, not in a PR

Evidence

Two things to decide in triage

Activity

V3RON commented on Oct 6, 2026

Reproduction

Root cause

Simplest fix

Alternatives rejected

Risk

Side findings

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions