Problem
Enabling PostHog silently disables Node's crash-on-unhandled-rejection, converting a fatal, self-healing
crash into a live process with a dead worker loop.
server.ts deliberately registers no process handlers, and says why (src/server.ts:322-326):
enableExceptionAutocapture (set inside initPostHog) already installs its own
uncaughtException/unhandledRejection handlers … so no manual process.on wiring is needed here for
the crash case.
That holds for uncaughtException — posthog-node's handler counts foreign listeners and calls
onFatalError → process.exit(1) when it is the only one. It does not hold for rejections
(node_modules/posthog-node/src/extensions/error-tracking/autocapture.ts:51-59):
process.on('unhandledRejection', reason => captureFn(...))
It never rethrows and never exits. Node 22's default --unhandled-rejections=throw escalates a rejection
to an uncaught exception only when no unhandledRejection hook is set. Installing one downgrades
every unhandled rejection to a captured telemetry event.
Confirmed: grep -rn "uncaughtException\|unhandledRejection" src/ finds no handler in the ORB server —
the only hits are in packages/loopover-miner and packages/discovery-index, which are different
processes.
Trigger
Operator sets POSTHOG_API_KEY — or is a hosted tenant, where LOOPOVER_CENTRAL_POSTHOG_KEY is injected
automatically by control-plane/src/container-driver.ts:104, so this is on by default for hosted.
A promise then rejects unhandled inside the queue drain loop.
- Before PostHog: process dies → Docker restarts →
loopover_jobs_recovered_total reclaims the stale
in-flight job.
- After PostHog: one telemetry event, loop dead, process alive, container reports healthy, job stays
claimed.
Impact
An absorbing state with no autonomous exit, created as a side effect of turning on observability. It also
inverts the meaning of the container's own liveness: /health keeps returning 200 for a process whose
worker is gone, so neither Docker's restart policy nor the healthcheck intervenes.
This is the mirror image of #8997/#9007: those cover a restart that kills a live pass; this is the restart
that should happen and doesn't.
Requirements
- Register an explicit
process.on("unhandledRejection", …) in src/server.ts that logs, captures,
flushes telemetry, then rethrows or process.exit(1) — so the crash-and-restart contract is
independent of whether telemetry is configured.
- Correct the comment at
server.ts:322-326, which asserts a property posthog-node does not provide for
rejections.
- Verify the same reasoning for
uncaughtException: the listener-count heuristic means adding our own
handler changes posthog's behaviour, so both must be designed together.
- Consider asserting worker-loop liveness in
/health so a dead pump cannot report healthy.
Test Coverage Requirements
99%+ patch coverage, branch-counted. Both arms (telemetry configured / not) asserting an unhandled
rejection terminates the process.
Links & Resources
maintainer-only — process lifecycle.
Problem
Enabling PostHog silently disables Node's crash-on-unhandled-rejection, converting a fatal, self-healing
crash into a live process with a dead worker loop.
server.tsdeliberately registers no process handlers, and says why (src/server.ts:322-326):That holds for
uncaughtException— posthog-node's handler counts foreign listeners and callsonFatalError→process.exit(1)when it is the only one. It does not hold for rejections(
node_modules/posthog-node/src/extensions/error-tracking/autocapture.ts:51-59):It never rethrows and never exits. Node 22's default
--unhandled-rejections=throwescalates a rejectionto an uncaught exception only when no
unhandledRejectionhook is set. Installing one downgradesevery unhandled rejection to a captured telemetry event.
Confirmed:
grep -rn "uncaughtException\|unhandledRejection" src/finds no handler in the ORB server —the only hits are in
packages/loopover-minerandpackages/discovery-index, which are differentprocesses.
Trigger
Operator sets
POSTHOG_API_KEY— or is a hosted tenant, whereLOOPOVER_CENTRAL_POSTHOG_KEYis injectedautomatically by
control-plane/src/container-driver.ts:104, so this is on by default for hosted.A promise then rejects unhandled inside the queue drain loop.
loopover_jobs_recovered_totalreclaims the stalein-flight job.
claimed.
Impact
An absorbing state with no autonomous exit, created as a side effect of turning on observability. It also
inverts the meaning of the container's own liveness:
/healthkeeps returning 200 for a process whoseworker is gone, so neither Docker's restart policy nor the healthcheck intervenes.
This is the mirror image of #8997/#9007: those cover a restart that kills a live pass; this is the restart
that should happen and doesn't.
Requirements
process.on("unhandledRejection", …)insrc/server.tsthat logs, captures,flushes telemetry, then rethrows or
process.exit(1)— so the crash-and-restart contract isindependent of whether telemetry is configured.
server.ts:322-326, which asserts a property posthog-node does not provide forrejections.
uncaughtException: the listener-count heuristic means adding our ownhandler changes posthog's behaviour, so both must be designed together.
/healthso a dead pump cannot report healthy.Test Coverage Requirements
99%+ patch coverage, branch-counted. Both arms (telemetry configured / not) asserting an unhandled
rejection terminates the process.
Links & Resources
src/server.ts~322-326;src/selfhost/posthog.ts~161;control-plane/src/container-driver.ts~104maintainer-only — process lifecycle.