Skip to content

fix(engine): module-singleton shared-ownership — joiner must not tear down shared pool - #1599

Closed
jalagrange wants to merge 2 commits into
garrytan:masterfrom
jalagrange:fix/module-singleton-shared-ownership
Closed

jalagrange wants to merge 2 commits into
garrytan:masterfrom
jalagrange:fix/module-singleton-shared-ownership

Conversation

@jalagrange

Copy link
Copy Markdown
Contributor

Problem

Companion to #1593 — that PR repairs the batch-retry path so existing data loss stops. This PR fixes the root cause: a temp engine that joins the shared module-level singleton silently kills it for the whole cycle on disconnect().

Repro (full dream cycle on direct Postgres)

  1. Cycle's main engine connects without poolSize → module path. db.ts singleton initialized; _connectionStyle='module'.
  2. Lint phase's resolveLintContentSanity (src/commands/lint.ts:298) creates a temp PostgresEngine to read 4 config keys, calls engine.connect({}).
  3. db.connect() no-ops because singleton is already alive. Temp engine sets _connectionStyle='module' regardless — even though it did nothing.
  4. Lint's try/finally calls await engine.disconnect() on the temp engine. Pre-fix: falls through to db.disconnect() because _connectionStyle === 'module' — and silently nulls the shared singleton for the rest of the cycle.
  5. sync, synthesize, and embed phases that run AFTER lint flunk with "No database connection: connect() has not been called."

extract survives because #1593's batchRetry onRetry calls this.reconnect() — but the non-batch phases have no such cushion.

Live evidence

Traced db.disconnect() on a v0.41.26.0 install with a temporary stack-trace probe:

[DBG] db.disconnect called, sql=alive
Error
    at disconnect (db.ts:228)
    at disconnect (postgres-engine.ts:211)
    at resolveLintContentSanity (lint.ts:319)
    at async runLintCore (lint.ts:395)
    at async runPhaseLint (cycle.ts:676)
    at async runCycle (cycle.ts:1421)

The first db.disconnect() of the cycle comes from lint, not from anything the cycle author would expect.

Fix

Track per-engine whether THIS instance actually initialized the singleton (_ownsModuleSingleton) by snapshotting db.isConnected() BEFORE calling db.connect(). Only the initializer is allowed to call db.disconnect() on cleanup. Joiners no-op — the CLI's main engine retains exclusive teardown rights.

Two-file change

src/core/db.ts — export new helper isConnected():

export function isConnected(): boolean {
  return sql !== null;
}

src/core/postgres-engine.ts — track ownership in connect(), gate teardown in disconnect():

+ private _ownsModuleSingleton = false;

  async connect(...) {
    ...
    } else {
+     const singletonWasLive = db.isConnected();
      await db.connect(config);
      this._connectionStyle = 'module';
+     this._ownsModuleSingleton = !singletonWasLive;
    }
  }

  async disconnect() {
    ...
    if (this._connectionStyle === 'module') {
+     if (this._ownsModuleSingleton) {
        await db.disconnect();
+     }
      this._connectionStyle = null;
+     this._ownsModuleSingleton = false;
    }
  }

Tests

test/postgres-engine-module-ownership.test.ts pins the contract hermetically via mock.module on db.ts. Four cases:

  • first engine initializes; joiner does NOT disconnect on cleanup
  • joiner.disconnect() BEFORE primary.disconnect() leaves singleton alive (the exact lint-phase repro)
  • joiner.disconnect() is idempotent
  • solo engine round-trips correctly
$ bun test test/postgres-engine-module-ownership.test.ts
 4 pass, 0 fail, 16 expect() calls

$ bun test test/postgres-engine-module-ownership.test.ts test/engine-factory.test.ts \
          test/connection-resilience.test.ts test/connection-manager.serial.test.ts \
          test/core/retry.test.ts
 99 pass, 0 fail

$ bun run typecheck  # clean

End-to-end verification

Deployed both PRs to the same install + ran gbrain dream:

Before (master + nothing):

✗ sync         sync phase failed
   [InternalError/UNKNOWN] No database connection: connect() has not been called.
✗ synthesize   synthesize phase failed
   [InternalError/SYNTH_PHASE_FAIL] No database connection: connect() has not been called.
✓ extract     285 link(s), 0 timeline entries  ← 13 rows lost in batch

After (PR #1593 + this PR):

✓ sync         +0 added, ~0 modified, -0 deleted
✓ synthesize   73 transcript(s) synthesized in 0.4s     ← drained the backlog
✓ extract      285 link(s), 37 timeline entries         ← full recovery

Zero "No database connection" in the log. The remaining ✗ embed is an unrelated OPENAI_API_KEY config issue, not the singleton bug.

Why ship both PRs

#1593 and this PR are independently load-bearing:

  • This PR alone — batchRetry's pre-existing retry loop has no reconnect, so even though the singleton no longer dies mid-cycle, any real (non-leak) connection blip still loses batches. #1593 is the defense-in-depth.
  • #1593 alone — extract recovers, but sync and synthesize keep flunking on every cycle. This PR is the root-cause fix.

Either can land first; merging both gives a fully clean cycle.

Risk

Minimal — the only behavior change is that passive sharers stop tearing down the shared pool. The CLI's main engine, autopilot's long-lived engine, and any first-connecting engine still teardown exactly as before. Second/third-connecting engines (lint's content-sanity, frontmatter audit, any future similar pattern) now correctly no-op on cleanup.

…ull repair

PR garrytan#1416 added "No database connection" to retry-matcher's retryable
patterns, but the retry path never re-established the pool. When
engine.disconnect() runs mid-cycle (e.g. between dream phases) the
module-level singleton goes null; the in-flight batch flunks with
"No database connection: connect() has not been called.", retries fire
3/3 against the same null singleton, and the batch is lost.

Live repro on v0.41.26.0 against direct Postgres (not Supavisor):

  [extract.timeline_fs] connection blip, retrying (attempt 1/3): No database connection ...
  [extract.timeline_fs] connection blip, retrying (attempt 2/3): No database connection ...
  [extract.timeline_fs] connection blip, retrying (attempt 3/3): No database connection ...
    batch error (13 timeline rows lost): No database connection ...

Symptoms: 192 exhausted batch retries in 24h (doctor batch_retry_health
FAIL), cycle_freshness FAIL because conversation_facts_backfill's first
engine.getConfig() also hits the dead singleton (immediate, no retry path).

Two-file change:

  src/core/retry.ts
    Widen WithRetryOpts.onRetry to allow Promise<void> and await it in
    withRetry. The old fire-and-forget call meant async pre-retry repair
    work (e.g. engine.reconnect()) could not run atomically with the
    retry — the next attempt raced against unfinished repair.

  src/core/postgres-engine.ts
    In batchRetry's onRetry, call this.reconnect() before the next
    attempt sleeps. Mirrors the supervisor pool-restart pattern at
    minions/supervisor.ts:600. engine.reconnect() no-ops when
    _savedConfig is null (module-singleton mode without a saved engine
    config) so the call is defensively safe. withRetry only invokes
    onRetry after isRetryableConnError matches, so every onRetry call
    is for a connection-class error where reconnect is the right repair.

Tests added in test/core/retry.test.ts:

  - async onRetry is awaited before next attempt (pins the contract
    batchRetry now depends on; without await, asserts event ordering
    would break)
  - onRetry that throws propagates (best-effort repair contract — the
    reconnect failure surfaces visibly rather than being swallowed)

All 39 retry.test.ts tests pass; 94/94 across retry-adjacent suites
(retry, retry-stress, retry-matcher, connection-resilience,
doctor-batch-retry). Typecheck clean.

Does not address the non-batch-wrapped path (conversation_facts_backfill's
direct engine.getConfig fails with the same root cause). That's a
separate, smaller fix — likely engine.ensureConnected() at phase start
— intentionally left out of this PR to keep it surgical.
… down shared pool

Repro (cycle on direct Postgres):
- Cycle's main engine connects without poolSize → module path. db.ts
  singleton initialized; _connectionStyle='module'.
- Lint phase's resolveLintContentSanity (src/commands/lint.ts:298) creates
  a temp PostgresEngine to read 4 config keys, calls engine.connect({}).
  Routes to module path; db.connect() early-returns because singleton is
  already alive. Temp engine sets _connectionStyle='module' regardless,
  having done nothing.
- Lint's try/finally calls await engine.disconnect() on the temp engine.
  Pre-fix: falls through to db.disconnect() because _connectionStyle is
  'module' — and silently nulls the shared singleton for the cycle.
- Sync, synthesize, and embed phases that run AFTER lint flunk with
  "No database connection: connect() has not been called." Extract
  survives because batchRetry's onRetry now reconnects (PR garrytan#1593) — but
  the non-batch phases have no such cushion.

Live evidence captured by tracing db.disconnect() with a stack-trace probe
on a v0.41.26.0 install:

  [DBG] db.disconnect called, sql=alive
  Error
      at disconnect (db.ts:228)
      at disconnect (postgres-engine.ts:211)
      at resolveLintContentSanity (lint.ts:319)
      at async runLintCore (lint.ts:395)
      at async runPhaseLint (cycle.ts:676)
      at async runCycle (cycle.ts:1421)

Fix: track per-engine whether THIS instance actually initialized the
singleton (`_ownsModuleSingleton`) by snapshotting `db.isConnected()`
BEFORE calling `db.connect()`. Only the initializer is allowed to call
`db.disconnect()` on cleanup. Joiners no-op — the CLI's main engine
retains exclusive teardown rights.

Two-file change:

  src/core/db.ts
    Export new helper `isConnected()` returning whether the singleton
    is live. Pure read, no side effect.

  src/core/postgres-engine.ts
    Add private `_ownsModuleSingleton` flag. In connect()'s module path,
    snapshot `db.isConnected()` BEFORE `db.connect()` and set
    `_ownsModuleSingleton = !singletonWasLive`. In disconnect(), gate
    the `db.disconnect()` call on `_ownsModuleSingleton`. Reset both
    flags on disconnect so the engine is reusable.

Test: test/postgres-engine-module-ownership.test.ts pins the contract
hermetically via mock.module on db.ts — 4 cases:
  - first engine initializes; joiner does NOT disconnect on cleanup
  - joiner.disconnect() before primary.disconnect() leaves singleton alive
    (the exact lint-phase repro)
  - joiner.disconnect() is idempotent
  - solo engine round-trips correctly

99/99 tests pass across postgres-engine-module-ownership, engine-factory,
connection-resilience, connection-manager, retry. Typecheck clean.

Companion to PR garrytan#1593: that one repairs the batchRetry path so existing
data loss stops; this one fixes the ROOT cause so non-batch phases stop
flunking too. Together they close the dream-cycle singleton-null hole.
@garrytan

garrytan commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Thanks for this! The Postgres module-singleton / "connect() has not been called" ownership bug was fixed canonically in #1805 (atomic owner-only-disconnect), with follow-ups in #1608 and #1572 — all merged to master. Closing as already-resolved. If you still reproduce after upgrading, please reopen, and thank you for the contribution.

@garrytan garrytan closed this Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants