Unbounded native memory growth from repeated cross-worker fetch() subrequests (uncollected receiver-isolate garbage)

Summary

Under a sustained stream of cross-worker fetch() subrequests, workerd's RSS grows
without bound and the process is eventually OOM-killed. The growth is native (C++)
memory — per-subrequest object graphs in the receiver isolate whose V8 wrappers are
never collected, because that isolate's JS heap stays tiny and the native weight does not
appear to be reported to V8, so V8 never schedules a GC.

It is uncollected garbage, not a true leak: forcing a GC on the receiver isolate frees
all of it and the memory is reused.

This surfaced originally as an OOM in @cloudflare/vite-plugin dev servers
(workers-sdk#14701), but it
reproduces in plain workerd with two trivial Workers and no wrangler / vite / miniflare.

Reproduction (dependency-free)

Three files + a config. Worker main issues N fetch() subrequests to worker target.

config.capnp:

using Workerd = import "/workerd/workerd.capnp";

const config :Workerd.Config = (
  services = [
    (name = "main", worker = .mainWorker),
    (name = "target", worker = .targetWorker),
  ],
  sockets = [
    (name = "http", address = "*:8080", http = (), service = "main"),
  ],
);

const mainWorker :Workerd.Worker = (
  modules = [ (name = "main.js", esModule = embed "main.js") ],
  bindings = [ (name = "TARGET", service = "target") ],
  compatibilityDate = "2024-12-30",
);

const targetWorker :Workerd.Worker = (
  modules = [ (name = "target.js", esModule = embed "target.js") ],
  compatibilityDate = "2024-12-30",
);

main.js:

export default {
  async fetch(request, env) {
    const count = Number(new URL(request.url).searchParams.get("n") ?? "100");
    for (let i = 0; i < count; i++) {
      const res = await env.TARGET.fetch("http://target/");
      await res.text();
    }
    return new Response(`did ${count} subrequests`);
  },
};

target.js:

export default {
  async fetch(request) {
    return new Response("ok");
  },
};

Run it and drive load:

workerd serve --inspector-addr=localhost:9230 config.capnp
# in another terminal — 20,000 subrequests total:
for i in $(seq 1 200); do curl -s "http://localhost:8080/?n=100" >/dev/null; done
# sample RSS:
ps -o rss= -p "$(pgrep -f 'workerd serve')" | awk '{printf "%d MB\n", $1/1024}'

Observed (`workerd` 2026-07-21, macOS arm64)

RSS grows linearly and never comes back down on its own:

subrequests	RSS	live `DeleteQueue` (proxy for per-request graphs)
0	28 MB	2
4,000	79 MB	4,042
8,000	121 MB	8,082
12,000	165 MB	12,122
16,000	201 MB	16,162
20,000	224 MB	20,202

≈ one leaked per-request object graph per subrequest. DeleteQueue was counted with
macOS heap --addresses=workerd::DeleteQueue --noContent <pid> and is used only as an
easy-to-count proxy for the number of un-GC'd per-subrequest graphs (it is not itself the
bulk of the memory).

The memory is reclaimable garbage

HeapProfiler.takeHeapSnapshot forces a full mark-compact GC. Applying it via CDP after
20,000 subrequests:

GC the receiver (target) isolate → DeleteQueue → 0, RSS returns toward
baseline, and the freed memory is reused by subsequent load.
GC the caller (main) isolate → no effect.

So the retained objects are unreachable garbage in the receiver isolate; V8 simply never
schedules a GC there under sustained subrequest load.

The same behaviour was confirmed in a real multi-isolate app (the @cloudflare/vite-plugin
dev server, ~21 isolates): after driving HMR load the live DeleteQueue count was ~5,700,
and forcing GC specifically on the isolate that receives the module-fetch subrequests
dropped it to ~87 while every other isolate's GC did nothing. The catch that made this hard
to see: that receiver isolate's inspector debug path contains a literal #, which
URL-based WebSocket clients (and Chrome DevTools) silently truncate as a fragment, so a
naive "force GC via DevTools" always lands on the wrong (caller) isolate and appears to do
nothing.

What is allocated per subrequest (root cause)

Each env.TARGET.fetch(...) sets up an IoContext in the receiver isolate and runs the
entrypoint. Besides the expected Request/Response/Headers/ReadableStream, each
invocation allocates:

Two trace-span async-context frames. worker-entrypoint.c++ builds both a
traceScope and a userTraceScope per fetch (worker-entrypoint.c++:449-451, also
:671-672, :790-791, :864-866, :962-963), via
IoContext::makeAsyncTraceScope (io-context.c++:1136) and
IoContext::makeUserAsyncTraceScope (io-context.c++:1162). Each heap-allocates a
SpanParent, wraps it in an IoOwn (addObject, io-context.h:702), wraps that in an
opaque V8 handle (jsg::wrapOpaque), and builds an AsyncContextFrame::StorageScope
holding a v8Ref. makeAsyncTraceScope short-circuits the span when tracing is off
(io-context.c++:1148) but still performs the wrapper allocations — there's an explicit
TODO(cleanup) about this at io-context.c++:1145-1146; makeUserAsyncTraceScope has
no short-circuit at all.
A promise-context tag holding a DeleteQueue ref. runInContextScope installs a
v8::Isolate::PromiseContextScope via getPromiseContextTag
(io-context.c++:1287-1288), which creates an IoCrossContextExecutor holding
deleteQueue.queue.addRef() (io-context.c++:1652-1657, io-own.h:124-126), stored in
an opaque V8 object owned by the IoContext. One DeleteQueue per subrequest.

After the subrequest completes the IoContext is torn down and all of the above become
unreachable, but they are V8-wrapped native objects, so their C++ memory is only
released when V8 collects the wrappers in the receiver isolate. The receiver isolate's JS
heap stays ~17-22 MB and the native weight is apparently not reported to V8, so V8's
JS-heap-growth-driven GC scheduler never fires under sustained load → unbounded RSS.

Possible directions

Report the native memory behind these per-request wrappers to V8 (external-memory
accounting) so GC is scheduled under subrequest load.
Elide the per-invocation trace-scope allocations when tracing is disabled — the
TODO(cleanup) at io-context.c++:1145-1146, extended to makeUserAsyncTraceScope.
Prompt GC on idle receiver isolates that have accumulated many completed IoContexts.

Ruled out

Vite / JS retention — heap snapshots of all inspectable isolates are flat.
Durable Objects — reproduces with no DO; only one IoContext is live after the load.
Cross-request promise resolution — toggling
no_handle_cross_request_promise_resolution had zero effect (it only gates a
SetPromiseCrossContextResolveCallback, not the promise-context tag / DeleteQueue).

Environment

workerd 2026-07-21 (@cloudflare/workerd-darwin-arm64@1.20260721.1)
macOS, arm64
Line numbers above are from the workerd main branch around that date.

Follow-up: prototyped direction #1, and a second (dominant) layer on Linux

I built workerd from main on Linux (Debian bookworm, arm64, default tcmalloc) and
measured the proposed fixes. Two findings worth recording before anyone invests in a fix:
external-memory accounting (direction #1) is inert as-is, and on Linux the RSS growth
has a second layer — allocator fragmentation — that no GC or release call reclaims.

1. External-memory accounting (direction #1) does not schedule a GC

I prototyped reporting the per-subrequest native weight to V8 via
AdjustAmountOfExternalAllocatedMemory (both a caller-side estimate on the incoming
Request and the existing IoContext accounting path). Measured RSS was identical to
baseline at every heap configuration.

Reason, from the V8 source workerd builds against: a positive external-memory delta only
triggers a GC through Heap::ReachedHeapLimit() (needs an old-generation or global heap
limit) or through StartIncrementalMarkingIfAllocationLimitIsReached. But workerd:

runs V8 with --noincremental-marking (jsg/setup.c++), so the incremental-marking
branch is dead; and
in OSS the IsolateLimitEnforcer returns empty v8::ResourceConstraints
(server.c++, NullIsolateLimitEnforcer::getCreateParams()), so there is no
old-generation limit and enforce_global_heap_limit is off.

So external memory never reaches a live GC trigger. Direction #1 can only work if it is
paired with an actual heap limit (or incremental marking) — on its own it changes nothing.

Confirming the mechanism: adding --max-old-space-size=64 (via v8Flags) does make V8
GC — the accumulating JS wrappers eventually hit the limit — which substantially slows the
growth. But it still does not bound RSS, which brings us to layer 2.

2. Layer 2 (Linux): freed memory is retained as allocator fragmentation

With --max-old-space-size=64 (so V8 GCs regularly and the JS heap stays small), I drove
the dependency-free repro to 100k subrequests while, on every mark-compact GC, calling
tcmalloc::MallocExtension::ReleaseMemoryToSystem(SIZE_MAX) plus draining per-CPU and
thread caches (SetMaxPerCpuCacheSize(0), SetMaxTotalThreadCacheBytes(0)), and logging
tcmalloc's own counters alongside process RSS:

subreqs	RSS	tcmalloc `physical_memory_used`	non-tcmalloc (RSS − physical)
10k	171 MB	89 MB	82 MB
20k	191 MB	104 MB	87 MB
30k	226 MB	121 MB	105 MB
50k	305 MB	175 MB	130 MB
80k	327 MB	201 MB	126 MB
100k	374 MB	238 MB	136 MB

Both components grow without bound. The larger one (~73% of growth) is tcmalloc's physical
footprint, and it grows despite releasing on every GC. At 100k, immediately after a
forced full GC:

generic.current_allocated_bytes (live) → 42 MB (V8 heap is 0 MB used / 1 MB total)
generic.physical_memory_used → 238 MB
tcmalloc.page_heap_free → ~40 MB, tcmalloc.cpu_free → 0

So ~150 MB is neither live nor in the free page list — it is stuck in partially-used
spans. This is classic fragmentation from a churny stream of many short-lived,
variable-size native allocations (the per-subrequest IoContext object graphs). No release
API can return a span that still holds a single live object, so ReleaseMemoryToSystem
recovers almost nothing. The remaining ~27% of growth is non-tcmalloc (V8's own mmap'd
heap/code arenas + cages + thread stacks), which also creeps up.

Why macOS looked fully reclaimable but Linux OOMs

On macOS the system allocator returns freed memory promptly, so "force a GC and RSS comes
back" holds (as in the original report). On Linux with tcmalloc the same GC frees the
graphs back to the allocator, but the allocator keeps the fragmented pages, so RSS does
not come down — which is why this bites as a hard OOM in Linux dev servers and
production, not just as transient high-water-mark on macOS.

Implications for the proposed directions

Rename BUILD -> BUILD.bazel #1 (external-memory accounting): inert unless combined with a heap limit / incremental
marking. Even then it only addresses layer 1 (getting V8 to GC); on Linux the freed
memory stays resident due to layer 2.
Fix clang 11 and make icudata embed portable #3 (prompt GC on idle receiver isolates): necessary but not sufficient on Linux —
GC hands the memory to the allocator, which retains it as fragmentation. Verified: forcing
full GC drops live to 42 MB but RSS only 374→324 MB.
Add tips for developing with vscode #2 (reduce the per-invocation allocations): the only lever that attacks the actual
cause — fewer/pooled per-subrequest native allocations (the two trace-span async-context
frames, the DeleteQueue promise-context tag, the wrapper objects) means less churn, so
less fragmentation and less V8-side growth. Note the "elide only when tracing is off"
variant has limited shelf life since tracing is expanding to local dev; the durable win is
structural reduction/reuse regardless of tracing state.

Net: this is fundamentally an allocation-pattern problem in the receiver isolate's
per-subrequest path, not something an allocator-release or external-memory tweak can fix on
its own.

(Measured with a from-main Linux build; V8 flag / IsolateLimitEnforcer references are
from the same tree. Happy to share the instrumented diff and the harness if useful.)

Run metric queries inside AccountMetricCoordinator instead of per-que…

Serve the Worker from Miniflare directly, not wrangler dev (#3)

Navigation Menu

Repository navigation

Unbounded native memory growth from repeated cross-worker fetch() subrequests (uncollected receiver-isolate garbage) #6894

Summary

Reproduction (dependency-free)

Observed (`workerd` 2026-07-21, macOS arm64)

The memory is reclaimable garbage

What is allocated per subrequest (root cause)

Possible directions

Ruled out

Environment

Follow-up: prototyped direction #1, and a second (dominant) layer on Linux

1. External-memory accounting (direction #1) does not schedule a GC

2. Layer 2 (Linux): freed memory is retained as allocator fragmentation

Why macOS looked fully reclaimable but Linux OOMs

Implications for the proposed directions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unbounded native memory growth from repeated cross-worker fetch() subrequests (uncollected receiver-isolate garbage) #6894

Description

Summary

Reproduction (dependency-free)

Observed (workerd 2026-07-21, macOS arm64)

The memory is reclaimable garbage

What is allocated per subrequest (root cause)

Possible directions

Ruled out

Environment

Activity

petebacondarwin commented on Jul 22, 2026

Follow-up: prototyped direction #1, and a second (dominant) layer on Linux

1. External-memory accounting (direction #1) does not schedule a GC

2. Layer 2 (Linux): freed memory is retained as allocator fragmentation

Why macOS looked fully reclaimable but Linux OOMs

Implications for the proposed directions

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Observed (`workerd` 2026-07-21, macOS arm64)