[Bug] routing-history index rebuild runs synchronously on the serving event loop (487 s proxy-wide freeze) #6990

Client or integration

Codex App

Area

Proxy and routing

Summary

The routing-history index rebuild runs synchronously on the serving event loop. On a large ledger one rebuild froze the whole proxy for 487 s: no request on any provider completed and Codex clients showed "Connection failed: error sending request". I expected the rebuild to run in the background without blocking requests.

One call into the request-history index froze the proxy for 487 s. No request completed on any provider.
Codex clients then showed "Connection failed: error sending request". The ledger usage.jsonl was 2.34 GB (1.12 M rows).
The index file routing-history.sqlite was rebuilt to 4.93 GB.

Reproduction

1. Let usage.jsonl grow to about 2 GB (no usageLedgerMaxBytes).
2. Trigger a schema/identity mismatch or delete routing-history.sqlite, then send any request (routing calls openRequestHistoryIndexSync() from src/routing/health.ts:236).
3. Observe: the event loop is blocked for the whole rebuild (scratch measurements below).

Version

2.83.0 (same code in 2.81.0)

Operating system

macOS (Bun 1.4.2)

Provider and model

All providers (proxy-wide)

Logs or error output

rebuilt after identity/schema mismatch
routing-history.sqlite rebuilt to 4.93 GB from usage.jsonl 2.34 GB (1,119,276 rows); proxy unresponsive 487 s

Source-level analysis (v2.83.0)

1. refreshLockedSync() (line 432) is synchronous. ensureSchemaAndIdentity() may call destroyAndRecreate(), then
   fullRebuild() (line 412) reads the whole ledger and inserts every row in the same call stack. No await, no time slice.
2. The sync entry openRequestHistoryIndexSync() is called at routing time from src/routing/health.ts:236.
   The async entry openRequestHistoryIndex() wraps the same sync function in Promise.resolve(...), so it blocks too.
3. isHealthy() runs PRAGMA quick_check on every refresh (line 394). On a 4.8 GB file it takes 2.3 s per call, and 12 s on a cold cache.
4. Line 438 overwrites the specific reason that destroyAndRecreate() stored (schema version, integrity, identity, unreadable)
   with the text "rebuilt after identity/schema mismatch". The operator cannot tell which check fired.

Rebuild time grows faster than the row count (random primary-key inserts). The live freeze took 487 s on a loaded machine.

Proposed change

• Make the ingest a generator that yields after each read chunk. Drive it with a time slice (25 ms) and setImmediate for async callers.
• Give the sync entry a small budget (50 ms). When work remains, hand the generator to a background flight and throw a typed "busy" error.
  health.ts already treats a throw as "evidence unknown".
• Build a full rebuild in a private sidecar file (routing-history.sqlite.building.<pid>) and publish it with one atomic rename.
  No reader, including another process, sees a partial index, and no table is dropped under a reader.
• Commit rows, indexed_rows and indexed_offset in the same transaction for each batch, so an aborted ingest resumes exactly.
• Run no quick_check on the refresh path. Flag the index when any statement returns a corruption error and rebuild on the next refresh.
• Keep the specific rebuild reason.
• Keep one refresh in flight across closeRequestHistoryIndex().

Result on the 2.4 GB ledger: the sync call returns in 0.1 s, the largest event-loop stall is 0.2 s, the build takes 98 s in the background,
and a warm call costs 0 ms. A reference patch and 25 bun tests are available on request.

Also consider a default for usageLedgerMaxBytes. With no limit the ledger and the index grow without bound, and every rebuild is proportional to that size.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.