MemoryAdapterFactory.QueueMessageBatchAsync throws OutOfMemoryException when serializing large message batches

What is failing

MemoryAdapterFactory.QueueMessageBatchAsync throws an OutOfMemoryException when JSON-serializing a message batch via MessageDeliveryConverter. The failure occurs inside Utf8JsonWriter.Grow() when the PooledByteBufferWriter requests more memory from the shared array pool but allocation fails because the process has exhausted its available memory.

Probable cause

The stack trace shows the OEM originates in System.Text.Json.Utf8JsonWriter.Grow() → PooledByteBufferWriter.GetMemory() → SharedArrayPool<byte>.Rent() → GC.AllocateNewArray(). The message batch being serialized was large enough that, combined with existing memory pressure on the pod, the .NET GC could not allocate the next buffer chunk.

Confidence: High — the stack trace is unambiguous about the allocation path. What is unclear is whether the root cause is (a) a single abnormally large batch, (b) a leak accumulating memory over time until a routine serialization tip over, or (c) the pod simply being provisioned too small for the workload.

Impact

Single occurrence on one pod (memex-portal-deployment-585f6d7b98-z7ktk), one timestamp, no repeats. Not an active outage, but if the condition recurs the pod could become unstable or be OOMKilled by Kubernetes.

Where to look

MemoryAdapterFactory.QueueMessageBatchAsync — the entry point where the OOM is caught and logged.
MeshWeaver.Messaging.Serialization.MessageDeliveryConverter.Write — the custom JsonConverter<IMessageDelivery> that triggers the buffer growth. Check whether the serialized payload includes unbounded data (e.g. large attachment or blob inline) that could be streamed instead.
Pod memory limits — verify the memory limit/request for the memex-portal deployment and check whether it is consistently near capacity.

Evidence


Fingerprint	`4a50cc9363466ae2`
Category	`Microsoft.Extensions.Logging.ILogger`
Severity	Error
Exception	`System.OutOfMemoryException`
Top frame	`MeshWeaver.Messaging.Serialization.MessageDeliveryConverter.Write(Utf8JsonWriter writer, IMessageDelivery value, JsonSerializerOptions options)`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-585f6d7b98-z7ktk`
Occurrences	1
First seen	2026-08-12 21:00:36Z
Last seen	2026-08-12 21:00:36Z

Recent log lines

2026-08-12 21:00:36Z memex-portal-deployment-585f6d7b98-z7ktk fail: Microsoft.Extensions.Logging.ILogger[200401]
      Exception thrown in MemoryAdapterFactory.QueueMessageBatchAsync.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Buffers.SharedArrayPool`1.Rent(Int32 minimumLength)
         at System.Net.ArrayBuffer.EnsureAvailableSpaceCore(Int32 byteCount)
         at System.Text.Json.PooledByteBufferWriter.GetMemory(Int32 sizeHint)
         at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
         at System.Text.Json.Utf8JsonWriter.TranscodeAndWriteRawValue(ReadOnlySpan`1 json, Boolean skipInputValidation)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
         at MeshWeaver.Messaging.Serialization.MessageDeliveryConverter.Write(Utf8JsonWriter writer, IMessageDe…[truncated]

_{Opened automatically from Admin/_LogIncident/4a50cc9363466ae2. Recurrences are folded into this issue rather than opening new ones.}

Investigation: the batch hypothesis is falsified, and the serializer is not the defect

What the limit actually was

Not an unbounded batch. MemoryAdapterFactory.QueueMessageBatchAsync takes IEnumerable<T> events, so
"a batch that grew without limit" is the natural first read. It is wrong here: the only producer is
RoutingGrain.BuildStreamRoute → s.OnNextAsync(delivery) (src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs:222-225),
and OnNextBatchAsync appears nowhere in src/. Every batch on this path is exactly one message.

Not the serializer buffering something it could stream, either — or at least, that is not the defect.
The allocation that failed is a routine one: SharedArrayPool<byte>'s largest pooled bucket is 1 MB, so
any request above that is a fresh GC.AllocateUninitializedArray on the LOH. Against a heap sitting at
6289 MB of a 7.752 GiB hard limit (the same pod's GC stalls: 9.2 s and 11.4 s pauses), a perfectly
ordinary multi-MB serialization buffer cannot be allocated. The OOM is where the pressure surfaced, not
what caused it. Chasing this stack leads nowhere; the heap is the finding.

What the message was

MessageDeliveryConverter.Write → one ObjectDefaultConverter level → TranscodeAndWriteRawValue. That is
MessageDelivery<RawJson>.Message being written by RawJsonConverter.Write's writer.WriteRawValue(string)
(src/MeshWeaver.Messaging.Hub/Serialization/RawJsonConverter.cs:41-47).

So the payload was already a single UTF-16 string before Orleans ever saw it —
MessageDelivery.Package (MessageDelivery.cs:182-208) serialises the whole message to a string on the root
mesh hub route (MeshBuilder.cs:236-237), and the Orleans JSON codec then transcodes that same string back
to UTF-8 through a doubling PooledByteBufferWriter. The payload is therefore materialised at full size
twice, with the doubling growth on the LOH, for a hop that on the memory stream never leaves the process.
That is real LOH churn on the hottest path in the mesh — but it is a ~2x constant, not a cause of unbounded
growth, and shipping it as "the fix for this issue" would misrepresent an optimisation as a root cause.

Destination-wise the stream only carries portal / client / cache / mesh / import addresses
(MeshConfiguration.DefaultStreamRoutedAddressTypes), which puts this in the same family as #1200
("Import activity triggers unbounded message storm to cache hub via RawJson/DataChangedEvent loop").

The related-but-distinct amplifier worth recording

#1341 established the O(n²) CPU term and fixed it by batching, and correctly noted that the patch output
was never the problem — PatchExtensions.DiffArray is element-wise, so an append emits one O(1) add.

That holds for the own-stream (RFC 6902) path. It does not hold for the cross-hub path.
MeshNodeStreamExtensions.ComputeMergePatchDiff is an RFC 7396 merge patch and recurses into JSON
objects only — a changed array is emitted whole (updatedValue?.DeepClone()), and
MeshNodePatchMerge.ExtractBaseValues:303 clones the whole previous array into BaseValues to feed the
three-way MergeArray. So a single append to an N-element collection in MeshNode.Content produces a
PatchDataRequest carrying ~2N elements. Message size is O(collection size), and
ActivityLog.Messages / SubActivities / AffectedPaths and NodeTypeDefinition.CompilationDiagnostics
have no cap — unlike every other log-shaped collection in node content (LogIncident.Samples →
MaxSamples, CompletionMemory.Entries → MaxEntries = 300).

Note BaseValues is not dead payload — MergeArray genuinely consumes it — so dropping it for arrays
would regress the three-way merge. That option is closed.

Why no PR yet — a scope call

Every fix available inside this issue's boundary is a band-aid: capping the buffer, catching the OOM, or
capping ActivityLog.Messages (which silently drops audit lines — a retention-semantics decision, not a bug
fix). The genuine cure is the one already under discussion: move growing log-shaped content to the
satellite shape so an append is a new node rather than a rewrite of the whole collection. That is an
architectural change spanning many call sites and is not mine to decide unilaterally.

Recommendation: close this as a symptom and track the two real items separately — (1) the heap-pressure
investigation that the 6289 MB / 11.4 s GC stalls point at, and (2) the satellite migration for unbounded
in-content collections (successor to #1341 / #1359 / #1200). Happy to take either on request.

Merge pull request #1414 from Systemorph/perf/1284-cross-hub-string-s…

Disposition: fixed by the heap-pressure work — and the "~2x double materialisation" is measurably not 2x

Two separate findings. The first closes the issue; the second corrects a number this issue asserted
and, on the corrected number, argues against the optimisation it proposed.

1. The occurrence predates every fix, and it has not recurred

The previous comment established that the OOM is where the pressure surfaced, not what caused it —
the heap was at 6289 MB with 9.2 s / 11.4 s GC pauses, and SharedArrayPool's largest bucket is
1 MB, so any multi-MB buffer above it is a fresh LOH allocation that simply could not be satisfied.
That diagnosis holds. What was missing was the timeline:

	when
the OOM	2026-08-12 21:00:36Z
#1345 — `Workspace.GetStream` memoized	2026-08-12 21:51:30Z (51 min later)
#1398 / #1399 — activity-log satellites	2026-08-13 08:28Z / 09:11Z
#1415 — evicted mirrors reclaimed	2026-08-13 11:33Z
#1400	still open

Every named fix landed after the single occurrence. So this record is evidence from the pre-fix
build; it was never evidence that the OOM survives them.

Since then, on the driver those fixes target, measured on NodeTypeRecompileAlcLeakTest (Release,
three recompiles of a trivial type):

	retained per recompile
before #1345	20 MB · 63 hubs
after #1345	9.3 MB · 22 hubs
after #1415	7 MB · 12.7 hubs
after the intermediate-reduce fix (PR for #1324)	3–5 MB · 6.2 hubs

That is the growth term behind the 6.3 GB heap, down ~5x in memory and 10x in hubs.

Corroborating, read-only, on the deployment that produced the record:

Occurrences is still 1 and Last seen is unchanged at 2026-08-12 21:00:36Z. Recurrences are
folded into this issue rather than opening new ones, so ~15 h and two self-update rolls
(ci.3423 → ci.3428) later, it has not repeated.
memex-cloud portal is at 3075 Mi against a 28 Gi limit, no OOMKilled events; every restart
in the window is a self-update roll.

Recommend closing as fixed by #1345 / #1398 / #1399 / #1415 (+ the #1324 follow-up) — fixed, not
stale. The two items the previous comment asked to track separately remain valid and are unaffected
by this: the unbounded in-content collections (ActivityLog.Messages, SubActivities,
AffectedPaths, NodeTypeDefinition.CompilationDiagnostics) and the satellite migration.

2. The double materialisation is real — but removing it at the `RawJson` boundary nets ZERO

This issue recorded the Package → string → TranscodeAndWriteRawValue round trip as "a real ~2x
constant". I implemented the removal (payload held as UTF-8 end to end: SerializeToUtf8Bytes in
Package, WriteRawValue(ReadOnlySpan<byte>) in the converter, UTF-8 overloads at the two
.Content readers, public surface kept source-compatible) and measured it. It is not 2x, and as
implemented it is not a win at all.

In isolation, the pair behaves exactly as the issue describes — a standalone benchmark of
serialize-then-write, four payload sizes:

payload	string path	UTF-8 path	ratio	CPU ratio
5 KB	16 KB	11 KB	1.45x	1.05
83 KB	251 KB	167 KB	1.50x	0.96
1.3 MB	4112 KB	2742 KB	1.50x	1.15
5.6 MB	16799 KB	11199 KB	1.50x	1.27

3N → 2N, flat, plus real CPU above a megabyte where the LOH growth bites.

In the real pipeline it disappears. Same measurement on an actual
MessageDelivery<DataChangedEvent> carrying a 2539 KB RawJson, running Package() + the
transport serialize:

	allocated	vs payload
including construction — before	24983 KB	9.84x
including construction — after	24960 KB	9.83x
per hop only (delivery built once, outside) — before	24983 KB	9.84x
per hop only — after	22444 KB	8.84x

So the honest split:

Per hop, it saves exactly one payload — 2539 KB of 24983 KB, 10.2%, not 50%. The isolated
pair is a small fraction of what a hop actually costs; the doubling writer buffers dominate.
Construction pays it straight back. new RawJson(string) — the overwhelmingly common
construction path (JsonSynchronizationStream's patch JSON, ComputeMergePatchDiff,
EntityDelta, ~40 sites) — must now encode to UTF-8 eagerly, +1N. A message that takes one
hop therefore nets zero, which is exactly what the construction-inclusive row shows.

Recommendation: do not ship it in this form. A 10%-per-hop gain that is cancelled for
single-hop messages, bought by reshaping the type at the centre of every cross-hub message, is not
a good trade. Two ways it becomes one, either of which is a separate, properly-scoped change:

Make RawJson hold either form lazily — string-backed until Utf8 is read, byte-backed until
Content is read, each caching. That removes the +1N construction cost, so the per-hop saving
is kept with no offset. Strictly dominates what I measured above.
Move the producers to UTF-8, so the patch/diff path emits bytes and no conversion happens at
all. Bigger, and the only version that gets the full 1.5x.

The change is reverted and unshipped; the numbers above are the input for deciding whether either
follow-up is worth opening. Flagging rather than deciding, since it is a scope call: the premise the
optimisation rested on ("a real ~2x constant") is measurably 10% per hop and 0% end to end.

🤖 Measured, not inferred — including the measurement that falsified my own first benchmark.

Closing as fixed by #1345 / #1398 / #1399 / #1415 (plus the #1324 follow-up), per the evidence in the comment above: the single occurrence predates every one of those merges, it has not recurred in the ~15 h and two self-update rolls since, and the heap-pressure driver it was a symptom of is down ~5x in memory and 10x in hubs per recompile.

The two follow-ups the earlier investigation identified stay open on their own records: the unbounded in-content collections and the satellite migration. The RawJson double materialisation is measured in the comment above and deliberately NOT shipped — it is 10% per hop and 0% end to end in this form.

Merge pull request #3344 from Systemorph/fix/3342-withdraw-the-centra…

Navigation Menu

Repository navigation

MemoryAdapterFactory.QueueMessageBatchAsync throws OutOfMemoryException when serializing large message batches #1351

What is failing

Probable cause

Impact

Where to look

Evidence

Investigation: the batch hypothesis is falsified, and the serializer is not the defect

What the limit actually was

What the message was

The related-but-distinct amplifier worth recording

Why no PR yet — a scope call

Disposition: fixed by the heap-pressure work — and the "~2x double materialisation" is measurably not 2x

1. The occurrence predates every fix, and it has not recurred

2. The double materialisation is real — but removing it at the `RawJson` boundary nets ZERO

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MemoryAdapterFactory.QueueMessageBatchAsync throws OutOfMemoryException when serializing large message batches #1351

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

rbuergi commented on Aug 13, 2026

Investigation: the batch hypothesis is falsified, and the serializer is not the defect

What the limit actually was

What the message was

The related-but-distinct amplifier worth recording

Why no PR yet — a scope call

rbuergi commented on Aug 13, 2026

Disposition: fixed by the heap-pressure work — and the "~2x double materialisation" is measurably not 2x

1. The occurrence predates every fix, and it has not recurred

2. The double materialisation is real — but removing it at the RawJson boundary nets ZERO

rbuergi commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. The double materialisation is real — but removing it at the `RawJson` boundary nets ZERO