Skip to content

orb(selfhost): enabling PostHog silently disables crash-on-unhandled-rejection — a dead worker loop now survives inside a 'healthy' process #9133

Description

@JSONbored

Problem

Enabling PostHog silently disables Node's crash-on-unhandled-rejection, converting a fatal, self-healing
crash into a live process with a dead worker loop.

server.ts deliberately registers no process handlers, and says why (src/server.ts:322-326):

enableExceptionAutocapture (set inside initPostHog) already installs its own
uncaughtException/unhandledRejection handlers … so no manual process.on wiring is needed here for
the crash case.

That holds for uncaughtException — posthog-node's handler counts foreign listeners and calls
onFatalErrorprocess.exit(1) when it is the only one. It does not hold for rejections
(node_modules/posthog-node/src/extensions/error-tracking/autocapture.ts:51-59):

process.on('unhandledRejection', reason => captureFn(...))

It never rethrows and never exits. Node 22's default --unhandled-rejections=throw escalates a rejection
to an uncaught exception only when no unhandledRejection hook is set. Installing one downgrades
every unhandled rejection to a captured telemetry event.

Confirmed: grep -rn "uncaughtException\|unhandledRejection" src/ finds no handler in the ORB server —
the only hits are in packages/loopover-miner and packages/discovery-index, which are different
processes.

Trigger

Operator sets POSTHOG_API_KEY — or is a hosted tenant, where LOOPOVER_CENTRAL_POSTHOG_KEY is injected
automatically by control-plane/src/container-driver.ts:104, so this is on by default for hosted.
A promise then rejects unhandled inside the queue drain loop.

  • Before PostHog: process dies → Docker restarts → loopover_jobs_recovered_total reclaims the stale
    in-flight job.
  • After PostHog: one telemetry event, loop dead, process alive, container reports healthy, job stays
    claimed.

Impact

An absorbing state with no autonomous exit, created as a side effect of turning on observability. It also
inverts the meaning of the container's own liveness: /health keeps returning 200 for a process whose
worker is gone, so neither Docker's restart policy nor the healthcheck intervenes.

This is the mirror image of #8997/#9007: those cover a restart that kills a live pass; this is the restart
that should happen and doesn't.

Requirements

  • Register an explicit process.on("unhandledRejection", …) in src/server.ts that logs, captures,
    flushes telemetry, then rethrows or process.exit(1) — so the crash-and-restart contract is
    independent of whether telemetry is configured.
  • Correct the comment at server.ts:322-326, which asserts a property posthog-node does not provide for
    rejections.
  • Verify the same reasoning for uncaughtException: the listener-count heuristic means adding our own
    handler changes posthog's behaviour, so both must be designed together.
  • Consider asserting worker-loop liveness in /health so a dead pump cannot report healthy.

Test Coverage Requirements

99%+ patch coverage, branch-counted. Both arms (telemetry configured / not) asserting an unhandled
rejection terminates the process.

Links & Resources

maintainer-only — process lifecycle.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions