Repository navigation
Message delivery to the AgenticEngineering message hub exceeds the Orleans 100 MB MaxMessageBodySize and cannot be delivered #2897
Description
What is failing
A mesh message delivery routed through the Orleans transport (MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery)) to the messagehub/AgenticEngineering grain carries a body of 149,199,409 bytes (~142 MB) — above the configured Orleans MaxMessageBodySize of 104,857,600 bytes (100 MB). Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength refuses the frame, Connection.ProcessOutgoing tears the connection down, and the delivery fails.
Probable cause — medium confidence
One message produced for the AgenticEngineering space's message hub is far larger than the transport allows — most likely a delivery whose payload embeds a very large node/tree (a big document, export, or activity snapshot) serialized whole. All five observed occurrences carry the identical body size, at ~5 s intervals, each on a new connection (different local port every time) — the signature of a single oversized message in a retry loop, not five independent incidents. High confidence that the proximate cause is an oversized single-frame delivery; medium confidence on which producer built it (the log gives the destination grain, not the sending code path).
Impact
Small in raw numbers so far (5 occurrences over ~23 s on one portal pod), but the failure is a silent dead letter: the delivery to that space's hub can never succeed and, per the retry pattern, is likely repeating beyond the observed window. Whatever update or response that message carries to the AgenticEngineering hub is not arriving. Not an outage; a stuck data-delivery defect.
Where to look
- The delivery path:
MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessageoverMeshWeaver.Messaging.IMessageDelivery— find what serializes the delivery body and why it reaches ~142 MB, and whether large deliveries can be chunked or streamed instead of one frame. - The destination: the
messagehub/AgenticEngineeringhub on namespacememex(podmemex-portal-deployment-…) — inspect which space activity was in flight at the timestamps to identify the oversized payload's producer. - The limit itself: Orleans
MaxMessageBodySize(raising it would mask the symptom and make 142 MB frames normal traffic — prefer fixing/chunking the payload).
Not a duplicate: no existing incident covers this exception type or component (existing incidents are SiloUnavailableException/TimeoutException/rendering failures under different categories).
Evidence
| Fingerprint | 44c8a7988d01b764 |
| Category | Orleans.Runtime.Messaging.Connection |
| Severity | Error |
| Exception | Orleans.Runtime.Messaging.InvalidMessageFrameException |
| Namespace | memex |
| Pods | memex-portal-deployment-644ffc96b9-58pv4 |
| Occurrences | 5 |
| First seen | 2026-08-31 09:06:29Z |
| Last seen | 2026-08-31 09:06:52Z |
Recent log lines
2026-08-31 09:06:29Z memex-portal-deployment-644ffc96b9-58pv4 fail: Orleans.Runtime.Messaging.Connection[173314469]
Exception sending message Request [S10.244.4.229:11111:147160374 sys.client/hosted-10.244.4.229:11111@147160374]->[S10.244.2.146:11111:147160293 messagehub/AgenticEngineering] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #71C10FBE065881B0 to remote endpoint 10.244.2.146:11111 from local endpoint 10.244.4.229:33290
Orleans.Runtime.Messaging.InvalidMessageFrameException: Invalid body size: 149199409 (max configured value is 104857600, see MaxMessageBodySize)
at Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength(Int32 bodyLength) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 221
at Orleans.Runtime.Messaging.MessageSerializer.Write(PipeWriter writer, Message message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 193
at Orleans.Runtime.Messaging.Connection.ProcessOutgoing() in /_/src/Orleans.Core/Networking/Connection.cs:line 366
2026-08-31 09:06:34Z memex-portal-deployment-644ffc96b9-58pv4 fail: Orleans.Runtime.Messaging.Connection[173314469]
Exception sending message Request [S10.244.4.229:11111:147160374 sys.client/hosted-10.244.4.229:11111@147160374]->[S10.244.2.146:11111:147160293 messagehub/AgenticEngineering] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #71C10FBE065883BE to remote endpoint 10.244.2.146:11111 from local endpoint 10.244.4.229:60792
Orleans.Runtime.Messaging.InvalidMessageFrameException: Invalid body size: 149199409 (max configured value is 104857600, see MaxMessageBodySize)
at Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength(Int32 bodyLength) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 221
at Orleans.Runtime.Messaging.MessageSerializer.Write(PipeWriter writer, Message message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 193
at Orleans.Runtime.Messaging.Connection.ProcessOutgoing() in /_/src/Orleans.Core/Networking/Connection.cs:line 366
2026-08-31 09:06:39Z memex-portal-deployment-644ffc96b9-58pv4 fail: Orleans.Runtime.Messaging.Connection[173314469]
Exception sending message Request [S10.244.4.229:11111:147160374 sys.client/hosted-10.244.4.229:11111@147160374]->[S10.244.2.146:11111:147160293 messagehub/AgenticEngineering] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #71C10FBE065882FA to remote endpoint 10.244.2.146:11111 from local endpoint 10.244.4.229:60800
Orleans.Runtime.Messaging.InvalidMessageFrameException: Invalid body size: 149199409 (max configured value is 104857600, see MaxMessageBodySize)
at Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength(Int32 bodyLength) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 221
at Orleans.Runtime.Messaging.MessageSerializer.Write(PipeWriter writer, Message message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 193
at Orleans.Runtime.Messaging.Connection.ProcessOutgoing() in /_/src/Orleans.Core/Networking/Connection.cs:line 366
2026-08-31 09:06:44Z memex-portal-deployment-644ffc96b9-58pv4 fail: Orleans.Runtime.Messaging.Connection[173314469]
Exception sending message Request [S10.244.4.229:11111:147160374 sys.client/hosted-10.244.4.229:11111@147160374]->[S10.244.2.146:11111:147160293 messagehub/AgenticEngineering] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #71C10FBE06588D02 to remote endpoint 10.244.2.146:11111 from local endpoint 10.244.4.229:39776
Orleans.Runtime.Messaging.InvalidMessageFrameException: Invalid body size: 149199409 (max configured value is 104857600, see MaxMessageBodySize)
at Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength(Int32 bodyLength) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 221
at Orleans.Runtime.Messaging.MessageSerializer.Write(PipeWriter writer, Message message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 193
at Orleans.Runtime.Messaging.Connection.ProcessOutgoing() in /_/src/Orleans.Core/Networking/Connection.cs:line 366
2026-08-31 09:06:52Z memex-portal-deployment-644ffc96b9-58pv4 fail: Orleans.Runtime.Messaging.Connection[173314469]
Exception sending message Request [S10.244.4.229:11111:147160374 sys.client/hosted-10.244.4.229:11111@147160374]->[S10.244.2.146:11111:147160293 messagehub/AgenticEngineering] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #71C10FBE06588FE7 to remote endpoint 10.244.2.146:11111 from local endpoint 10.244.4.229:39778
Orleans.Runtime.Messaging.InvalidMessageFrameException: Invalid body size: 149199409 (max configured value is 104857600, see MaxMessageBodySize)
at Orleans.Runtime.Messaging.MessageSerializer.ThrowInvalidBodyLength(Int32 bodyLength) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 221
at Orleans.Runtime.Messaging.MessageSerializer.Write(PipeWriter writer, Message message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 193
at Orleans.Runtime.Messaging.Connection.ProcessOutgoing() in /_/src/Orleans.Core/Networking/Connection.cs:line 366
Opened automatically from Admin/_LogIncident/44c8a7988d01b764. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 31, 2026
Provenance for the trigger: the ~142 MB delivery to
messagehub/AgenticEngineeringcoincides with the forced GitSync re-imports of that partition on 2026-08-31 09:00–09:20Z (fourupdate/forceruns during the #2899 investigation) — a 371-node import shipped as ONE mesh delivery, content included, blows the 100 MB Orleans cap the moment a partition is big enough. The defect stands on its own: bulk imports must chunk deliveries rather than serialize the whole node set into one message. Note the partition's write lane also false-passes (#2899), so the oversized delivery was additionally a write that could never become visible.rbuergi commentedon Sep 1, 2026
Not currently recurring — measured on both production portals
Checked the live cluster for this signature across a 6-hour window:
Both namespaces, both running
memex-portal-ai:3.0.0-rc9.ci.7231, both healthy (memex-cloud 3/3 replicas, memex 2/2). Zero occurrences.What that does and does not establish
Does: the defect is not firing continuously, so nobody is losing deliveries right now, and there is no live incident to chase.
Does not: show it is fixed. This issue's occurrence correlated with forced GitSync re-imports of a large partition — an event, not a steady state. A 6-hour window with no such import in it cannot distinguish "fixed" from "the trigger did not occur". I did not run an import to find out, because doing so on a production portal to reproduce an OOM is not a reasonable test.
Suggested disposition
Leave open, but re-scope the next step to: run the trigger deliberately somewhere it is safe — a disposable mesh or a non-production instance — with a partition large enough to reproduce, rather than waiting for another production occurrence. The producing path was never pinned down (the leading candidate is a full-state snapshot on subscribe or a change-feed dump after a bulk import), and that is the thing a controlled reproduction would settle in one run.
Also worth carrying: whatever the fix, do not raise
MaxMessageBodySize— this issue already rejects that correctly. A 100 MB frame is the symptom of an unbatched producer, and a bigger limit just moves the cliff.Merge pull request #3018 from Systemorph/fix/2899-write-lane-cannot-t…
rbuergi commentedon Sep 1, 2026
Fixed by #3018 (merged). It did not auto-close: the PR body wrote the keyword inside a code span (
Fixes #2897), and GitHub does not parse closing keywords inside backticks. Closing by hand — the second time tonight this exact class of mistake left a landed fix's issue open, the other beingCloses #A, #Bwhere only the first number parses.What landed: a producer-side refusal, never a bigger limit —
MaxMessageBodySizeis untouched, as this issue itself insisted ("would mask the symptom and make 142 MB frames normal traffic").The mechanism is worth restating because it explains the blast radius this issue found puzzling. Orleans refuses the frame, not the message:
MessageSerializer.ThrowInvalidBodyLengththrows from insideConnection.ProcessOutgoing, the silo-to-silo write loop. A serializer fault there is not recoverable per-message, so Orleans tears the whole connection down — every unrelated delivery queued on it dies too — and the reconnect re-sends the same undeliverable body. That is why the five occurrences carried the identical body size ~5 s apart, each from a new local port: one undeliverable message in a loop that cannot converge, because the size is a property of the message and not of the attempt.It also explains why the symptom read as a partition that stopped answering rather than one request that timed out.
The guard is applied to both forward grain legs (
IMessageHubGrainandIPodHubGrain) — guarding only the first would have left the stream-routed half of forward traffic unprotected, which is the kind of partial cover that reads as a fix and is not.Still open and NOT fixed by this, so nobody reads the merge as covering them: #2899 (a write into a Store-plugin partition acknowledged, versioned and invisible) and #2885 (OOM while serialising a delivery to an import hub). #3018 was deliberately rescoped to the transport half; whether those two share this root cause is still an open question.
fix: the report, the ack and the log about a delivery stop carrying it (
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 11 more since the last update (16 total).
memex-portal-deployment-644ffc96b9-58pv4Recent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:53:22Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-08-31 09:24:00Z2026-09-01T23:57:02ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.