Skip to content

Queen CI: server-api and server-pglive fail on Linux only (leaked fake timers, two clocks, a logger mock that restores itself) #1738

Closed

Description

@gHashTag

Part of #1712. Blocks the ship PR, see #1730 (actors-next -> queen).

What fails

Tests / server-api and Tests / server-pglive on #1730 (runs 38003333371 and 38051321192) fail on ubuntu with Bun 1.3.6. The same groups pass on macOS with Bun 1.3.6 and 1.3.11, in CI's file order.

  1. tests/api/queen-bee-actors-round.test.ts: the three tests on jest.useFakeTimers() time out at 5000 ms.
  2. tests/api/queen-review-actors-start.test.ts: the give-up test on jest.useFakeTimers() times out at 5000 ms.
  3. tests/pglive/queen-actors-pg-beat-live.test.ts: "renews a lease its incarnation holds" reads a renewal sent 56 s before the test began (the send time is read under fake timers, the start on the real clock).
  4. tests/api/queen-turn-stop.test.ts: "ignores its SIGTERM ..." fails in 4 ms with no pid in .../grandchild, although its wait for the pid has a 10 s deadline.
  5. Two (unnamed) failures: logger.setLevel is not a function in the beforeAll and afterAll of tests/api/queen-bee-actors-bench.test.ts.
  6. A cascade after the timeouts: tasksContract SIGTERM/SIGINT, WaitForHelper (Date.now() - before is 0), tokenManager, createAgentRoutes /chat/active, Klavis, ContainerRuntime.waitForReady, probeGatewayReady.

Why

  • 1, 2, 4, 6: a test that times out keeps running in the background, and its finally { jest.useRealTimers() } has not run. The fake clock stays installed for every file after it. Item 4 can only fail in 4 ms on a 10 s deadline if the clock it reads was advanced by a test body that resumed later. Item 6 is the same leak after item 2.
  • 3: the beat's send time and the test's start come from two different clocks.
  • 5: tests/api/task-queue-service.test.ts captured the "real" logger inside afterAll, after its own mock.module. So its restore put the mock back, and every later file got a logger without setLevel. Its kept pg namespace has the same defect: mock.module rewrites a loaded namespace in place. This one reproduces on macOS too, with those two files run in that order.

Plan

  • Give the runtime the clock instead of faking the process's timers:

    • startBeeDispatcher(deps, clock = realClock);
    • a test-only restartBeeActorsOn(clock) for the round's module singleton;
    • startReviewerActors(..., clock = realClock).

    The tests then walk a VirtualClock.

  • Split the beat into nodeBeat(start, post, now = monoNow). The live test calls it on a clock it holds.

  • Copy the real pg and logger modules before mocking them.

  • Production behaviour is unchanged: every new parameter defaults to the real clock.

Activity

  1. gHashTag commented on Oct 10, 2026

    @gHashTag
    CollaboratorAuthor

    Shipped: #1741 merged into actors-next, which went into queen as 56c5df7 via #1730 (2026-10-10 13:38Z). Every new behaviour is behind a flag that is off. Closes fires only on the default branch main, so this is closed by hand. Ledger: #1729.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions