Skip to content

Recycle the self-host server before workerd's per-request memory leak OOMs the container - #408

Open
Adityakk9031 wants to merge 1 commit into
every-app:mainfrom
Adityakk9031:fix/self-host-request-memory-growth
Open

Adityakk9031 wants to merge 1 commit into
every-app:mainfrom
Adityakk9031:fix/self-host-request-memory-growth

Conversation

@Adityakk9031

Copy link
Copy Markdown

Root cause: workerd GC accounting, not the app

The app is not retaining the memory — the runtime serving it is. The self-host image serves the built app through vite preview with the Cloudflare Vite plugin (@cloudflare/vite-plugin 1.42.3 → Miniflare → workerd), so every HTTP request — including GET /api/health — is a request into a workerd isolate.

workerd has an open garbage-collection accounting bug (cloudflare/workerd#6894): each request allocates V8-wrapped native objects (per-invocation trace-span async-context frames and a promise-context reference in worker-entrypoint.c++ / io-context.c++), but the receiver isolate's native weight is never reported to V8. V8 therefore never sees enough heap pressure to schedule a GC under steady request load, and the objects pile up as uncollected garbage:

Why the app code is exonerated:

Because the leak sits inside workerd's own request path, no serving configuration of the current stack avoids it: any mode that dispatches through workerd leaks, and the app is a Cloudflare Workers app that cannot run under plain Node. The options are waiting for upstream, or recycling the process before the kernel does. This PR does the latter.

Fix: cgroup memory watchdog in the entrypoint

deploy/docker/docker-entrypoint.sh now runs the preview server in a supervised loop:

  1. A watchdog subshell polls the container's cgroup memory (memory.current vs memory.max on cgroup v2; memory.usage_in_bytes vs memory.limit_in_bytes on v1 fallback) every SELF_HOST_MEMORY_CHECK_INTERVAL seconds (default 60).
  2. When usage crosses SELF_HOST_MEMORY_RECYCLE_BYTES (default: 90% of the cgroup limit), it terminates the server — and its workerd child, whose leaked native memory is the point — and the loop starts a fresh server. Downtime is a few seconds; restart: unless-stopped / Kubernetes RestartPolicy recover from everything else as before.
  3. Anti-thrash: two recycles within 10 minutes means the limit is likely below the app's ~600 MiB baseline, so the entrypoint logs it and exits 1 rather than flapping forever.
  4. A real stop or crash still propagates its exit status to the restart policy, and the docker stop/SIGINT trap takes down the server, workerd, and the watchdog cleanly.

The threshold and interval are configurable, and the watchdog disables itself (with a startup log line) when no cgroup memory limit is detectable or the override is not a positive integer. SELF_HOST_CGROUP_PATH repoints it at a nonstandard cgroup mount. New knobs are documented in docs/SELF_HOSTING_DOCKER.md (new "Memory limits and the recycle watchdog" section) and .env.example; compose.yaml needed no change since env_file: .env forwards them.

Verification

  • sh -n and shellcheck -s sh clean on the entrypoint; npx knip and prettier --check clean.
  • A behavioral harness (stub server + fake cgroup files + shims) verified: the recycle fires at the threshold and the server restarts; a second recycle within 10 minutes exits 1 with the "giving up" message; a server that exits on its own propagates its status with no recycle; TERM reaches the stub server and workerd child and leaves no orphaned processes.

Honest limitation: Docker is not available on the machine this was developed on, so there is no live before/after measurement. The confirming test would be re-running #335's probe loop against this image: memory should climb to ~90% of the limit, drop back to baseline on recycle, and keep serving indefinitely instead of OOM-killing on a cycle.

Fixes #335

…he container

workerd, the runtime behind `vite preview`, retains ~80 MiB per 1000
requests in V8-invisible native memory (cloudflare/workerd#6894), so even a
health probe drives a memory-limited container to its cgroup limit and the
kernel OOM-kills it. Until upstream fixes GC accounting, the entrypoint
watches the container's cgroup memory and gracefully recycles the server at
90% of the limit — a few seconds of downtime instead of a hard kill.
Threshold and interval are configurable, and a real stop still propagates
its exit status to the restart policy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-host: every HTTP request retains ~80 MiB per 1000 requests; Kubernetes health probes alone OOM-kill the container

1 participant