Repository navigation
Recycle the self-host server before workerd's per-request memory leak OOMs the container - #408
Open
Adityakk9031 wants to merge 1 commit into
Open
Adityakk9031 wants to merge 1 commit into
Adityakk9031 wants to merge 1 commit into
Conversation
…he container workerd, the runtime behind `vite preview`, retains ~80 MiB per 1000 requests in V8-invisible native memory (cloudflare/workerd#6894), so even a health probe drives a memory-limited container to its cgroup limit and the kernel OOM-kills it. Until upstream fixes GC accounting, the entrypoint watches the container's cgroup memory and gracefully recycles the server at 90% of the limit — a few seconds of downtime instead of a hard kill. Threshold and interval are configurable, and a real stop still propagates its exit status to the restart policy.
emresari89
pushed a commit
to emresari89/open-seo
that referenced
this pull request
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause: workerd GC accounting, not the app
The app is not retaining the memory — the runtime serving it is. The self-host image serves the built app through
vite previewwith the Cloudflare Vite plugin (@cloudflare/vite-plugin1.42.3 → Miniflare → workerd), so every HTTP request — includingGET /api/health— is a request into a workerd isolate.workerd has an open garbage-collection accounting bug (cloudflare/workerd#6894): each request allocates V8-wrapped native objects (per-invocation trace-span async-context frames and a promise-context reference in
worker-entrypoint.c++/io-context.c++), but the receiver isolate's native weight is never reported to V8. V8 therefore never sees enough heap pressure to schedule a GC under steady request load, and the objects pile up as uncollected garbage:@cloudflare/vite-plugindev/preview servers (workers-sdk#14701). A mitigation PR (workers-sdk#14702) passes V8 flags through to workerd but is not yet released — and it only raises the V8 heap limit, which does not help memory V8 cannot see.Why the app code is exonerated:
withPgClientis a no-op) still leaked ~73 MiB/1000 — the Postgres path is not involved.GET /api/healthexecutes onlygetSelfHostSetupStatus()— a singleselect count(*) from projects— and self-host telemetry early-returns for that route. The request path insrc/server.tsis clean; there is nothing per-request to retain.Because the leak sits inside workerd's own request path, no serving configuration of the current stack avoids it: any mode that dispatches through workerd leaks, and the app is a Cloudflare Workers app that cannot run under plain Node. The options are waiting for upstream, or recycling the process before the kernel does. This PR does the latter.
Fix: cgroup memory watchdog in the entrypoint
deploy/docker/docker-entrypoint.shnow runs the preview server in a supervised loop:memory.currentvsmemory.maxon cgroup v2;memory.usage_in_bytesvsmemory.limit_in_byteson v1 fallback) everySELF_HOST_MEMORY_CHECK_INTERVALseconds (default 60).SELF_HOST_MEMORY_RECYCLE_BYTES(default: 90% of the cgroup limit), it terminates the server — and its workerd child, whose leaked native memory is the point — and the loop starts a fresh server. Downtime is a few seconds;restart: unless-stopped/ KubernetesRestartPolicyrecover from everything else as before.docker stop/SIGINTtrap takes down the server, workerd, and the watchdog cleanly.The threshold and interval are configurable, and the watchdog disables itself (with a startup log line) when no cgroup memory limit is detectable or the override is not a positive integer.
SELF_HOST_CGROUP_PATHrepoints it at a nonstandard cgroup mount. New knobs are documented indocs/SELF_HOSTING_DOCKER.md(new "Memory limits and the recycle watchdog" section) and.env.example;compose.yamlneeded no change sinceenv_file: .envforwards them.Verification
sh -nandshellcheck -s shclean on the entrypoint;npx knipandprettier --checkclean.TERMreaches the stub server and workerd child and leaves no orphaned processes.Honest limitation: Docker is not available on the machine this was developed on, so there is no live before/after measurement. The confirming test would be re-running #335's probe loop against this image: memory should climb to ~90% of the limit, drop back to baseline on recycle, and keep serving indefinitely instead of OOM-killing on a cycle.
Fixes #335