Skip to content

Bound the request-history integrity check so a silent worker cannot stall history readers #7984

Closed

Description

@lidge-jun

Area

Proxy and routing

What are you trying to accomplish?

Request-history readers (dashboard history and analytics, async history queries) should stay responsive even if the background integrity scan introduced by #7972 stops making progress.

What prevents this today?

After #7972, the PRAGMA quick_check on the request-history index runs in a disposable worker (src/routing/history/integrity.ts, RebuildWorkerJob.check() in src/routing/history/rebuild-job.ts). The check has no time limit of its own: the 5-second CLOSE_TIMEOUT_MS applies only after dispose(). A worker that crashes or exits rejects the pending check through fail(), but a worker that stays alive and never replies leaves inspect() pending indefinitely. Because the async open path shares one openPromise (indexer.ts around lines 568–575), every async history and analytics reader then waits on it. Routing callers are not affected; they get RequestHistoryIndexBusyError immediately.

This is not a regression. Before #7972, the same scan ran synchronously on the serving thread and would have blocked every request under the same conditions (for example frozen disk I/O). The worker rebuild path (refreshWaitingForWorker awaiting handedOff.rebuild.promise) has the same unbounded contract.

What should OpenCodex do?

An integrity check that has not answered within a bound should be treated as an operational failure. The worker should be disposed, the existing failure cooldown should apply, and waiting readers should get a bounded error rather than waiting forever. The file-replacement barrier must still be held until the worker has actually exited, which afterExit already guarantees.

Example usage or interface

Regression test sketch: inject a createJob whose check() never settles and never exits (RequestHistoryIntegrityOptions.createJob). Assert that inspect() moves from pending to unavailable within the bound, that dispose() was called, and that a later inspect() after the cooldown starts a fresh check.

Alternatives or workarounds

Apply the same bound to RebuildWorkerJob.build() waits in one change, since both share send().

Additional context

Raised by the independent review of #7972 (merged as a strict liveness improvement) and filed as its agreed follow-up. Related: #7966.

Checks

  • I searched existing issues and documentation.
  • This request describes a concrete OpenCodex workflow rather than merely naming a desired technology.
  • I removed secrets and personal data.

Activity

  1. coderabbitai commented on Oct 30, 2026

    @coderabbitai
    Contributor
    🔗 Related PRs

    #7972 - perf(history): move integrity scans off the serving thread [open]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. added
    proxyHTTP proxy, routing, reverse-proxy / management auth
    on Oct 30, 2026
  3. added a commit that references this issue on Oct 30, 2026
    227811f
  4. lidge-jun commented on Oct 30, 2026

    @lidge-jun
    OwnerAuthor

    Fixed by #7997, merged into dev as 227811f. The integrity check now has a 30 s bound: on timeout the worker is disposed, the failure cooldown applies, and waiting async readers get a bounded error, while index replacement still waits for the worker's actual exit. The worker rebuild wait (build()) intentionally keeps its contract, because a large-ledger rebuild can legitimately exceed 30 s; that would need its own bound if it ever becomes a problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestproxyHTTP proxy, routing, reverse-proxy / management auth

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions