Repository navigation
Memory stream queue grain dequeue calls time out after 30s on memex, stalling stream message delivery (memory-7) #5301
Description
What is failing
The Orleans memory-stream receiver MemoryAdapterReceiver (queue memory-7) fails to dequeue messages from its IMemoryStreamQueueGrain: the inter-silo call to IMemoryStreamQueueGrain.Dequeue(Int32) from the stream agent silo to the grain-hosting silo does not complete within Orleans' 30-second response timeout. This delays or drops delivery of stream messages carried by the memory-7 queue until the receiver's retry loop recovers.
Probable cause
Cause unclear, but the evidence points at a wedged or overloaded queue grain / silo on the memex instance rather than a defect in the receiver code. The timeout is on the grain-to-grain call itself (sys.svc.stream.agent -> memorystreamqueue/... Dequeue), meaning the hosting silo did not answer in 30 s — classic silo thread-pool starvation or a stuck grain turn. This is consistent with the repeated memex restarts for "wedged read hubs" (see Ops/Actions/restart-memex-wedged-reads-20260921) and the large cluster of sibling incidents at the same log site over the past days (TimeoutException, SiloUnavailableException and OrleansMessageRejectionException across memory-0 through memory-7). Confidence: medium — the receiver is the reporter, not the culprit.
Impact
Minimal in isolation: a single occurrence on a single pod (memex-portal-deployment-5c99d79997-7qdb4), with the receiver retrying. The risk is that it is an early symptom of the same silo congestion that has repeatedly wedged memex and required operator restarts; if the underlying starvation recurs, stream delivery across all eight memory queues stalls.
Where to look
Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs— the receiver that logs this failure (GetQueueMessagesAsync, line ~45).- The queue grain hosting on the target silo (
memorystreamqueue/...in the sample) — check silo thread-pool queue length and grain-call latency around the timestamp. - Related ops history:
Ops/Actions/restart-memex-wedged-reads-20260921andOps/Actions/restart-memex-confirmed-20260921(MeshWeaver#5073).
Possible duplicate
This fingerprint was folded from incident 9dc66e5046d3e6e4 (same site, same symptom), and the broader memory-queue incident cluster should be treated as one investigation. If a ticket already covers the memex silo congestion / wedged memory-stream queue grains, this can be closed as a duplicate of it.
Evidence
| Fingerprint | 515d2d1921e52e4c |
| Category | Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7` |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex |
| Pods | memex-portal-deployment-5c99d79997-7qdb4 |
| Occurrences | 1 |
| First seen | 2026-09-22 19:15:39Z |
| Last seen | 2026-09-22 19:15:39Z |
| Routing | not determined — no configured route matches the category Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 19:15:39Z memex-portal-deployment-5c99d79997-7qdb4 fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7[200402]
Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.157:11111:149109240 sys.svc.stream.agent/10.244.2.157:11111@149109240+Memory_3_memory-7-0xE0000007]->[S10.244.5.235:11111:149109296 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #7DD7C24373E25A3A.
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
Re-addressed by the current identity function: this incident inherited 9dc66e5046d3e6e4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/515d2d1921e52e4c. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #4802.
Same code site: the Orleans memory-stream reader,
PersistentStreamPullingAgent.ReadFromQueue→MemoryAdapterReceiver.GetQueueMessagesAsync→IMemoryStreamQueueGrain.Dequeueon a queue grain hosted by another silo.This issue's own discriminator: memex 19:15:39Z, pod
5c99d79997-7qdb4, queuememory-7, 30 sTimeoutException. The event behind it: the target10.244.5.235:…:149109296is the newxs87x, which booted at 19:15:01Z, 38 s earlier; its bake sweep ran at 19:17:30Z.The reader recovers without help. In decompiled
Orleans.Streaming10.3.1,PumpQueueretriesReadFromQueueup to 6 times with backoff, and if it gives up, the agent's grain timer starts a new pump on its next tick. This issue's samples show noGiving up readingline. What is lost is the RAM-resident queue content on the silo that left or stalled, which is the structural exposure #4802 tracks (theAddMemoryStreamsretirement order is recorded there). The full per-issue table is on #4802.systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-5c99d79997-7qdb4,memex-portal-deployment-56fbdcd48f-sz5stRecent log lines
Re-addressed by the current identity function: this incident inherited
9dc66e5046d3e6e4(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-5c99d79997-7qdb4,memex-portal-deployment-56fbdcd48f-sz5stRecent log lines
Re-addressed by the current identity function: this incident inherited
9dc66e5046d3e6e4(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
4e28705e75c023b1(Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7`), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.f141b098a0af7b0c4e28705e75c023b1Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7`System.TimeoutExceptionResponse did not arrive on time in {n}:{n}:{n} for message: Request [S1{n}.{n}:{n}:{n} {path}:{n}@{n}+Memory_3_memory-{n}-{hex}]->[S1{n}.{n}:{n}:{n} {path}] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int3{n}, System.Threading.CancellationToken) #{hex}.memexmemex-portal-deployment-5f6b9fc9b-hrdhqAdmin/_LogIncident/f141b098a0af7b0cWhat triage wrote about this fingerprint
What is failing
The Orleans memory stream provider's receiver for queue
memory-7fails to dequeue messages: theIMemoryStreamQueueGrain.Dequeuegrain call issued fromMemoryAdapterReceiver.GetQueueMessagesAsyncdoes not get a response within the default 30 s grain-call timeout, and the receiver throwsSystem.TimeoutException. The adapter itself is not at fault — it is waiting on a queue grain hosted on another silo that did not answer in time.Probable cause
Silo overload or activation starvation on the silo hosting the
memorystreamqueuegrain, causing the dequeue call to exceed the 30 s response timeout — same mechanism as the previously filed cluster-wide timeout pattern. Confidence: medium; this instance is a single occurrence and carries only the one stack, but the target grain lives on a remote host and the identical fingerprint has a documented history (see duplicate below).Impact
Minimal for this instance: a single occurrence on a single pod. Stream message delivery on queue
memory-7is delayed until the receiver retries; no data loss is indicated since the queue grain retains the messages.Duplicate of an already-filed issue
This is a recurrence of the exact fingerprint filed as Systemorph/MeshWeaver #4802 (incident
Admin/_LogIncident/d61395c4eab35f38, 19 occurrences across 13 pods, Aug 22 – Sep 20) and should likely be closed as a duplicate after a human confirms. That ticket covers the same log site and the same dequeue-timeout mechanism, including the sibling queue incidents (memory-0throughmemory-6) seen across the same period.Where to look
Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs—GetQueueMessagesAsync, the dequeue call site and its error/retry handling.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5f6b9fc9b-hrdhqRecent log lines
systemorph-com commentedon Oct 10, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 6 more since the last update (7 total).
memex-portal-deployment-5f6b9fc9b-hrdhqRecent log lines
systemorph-com commentedon Oct 10, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
4e28705e75c023b1(Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7`), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.c1485fc26a61fef34e28705e75c023b1Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-7`System.TimeoutExceptionResponse did not arrive on time in {n}:{n}:{n} for message: Request [S1{n}.{n}:{n}:{n} {path}:{n}@{n}+Memory_5_memory-{n}-{hex}]->[S1{n}.{n}:{n}:{n} {path}] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int3{n}, System.Threading.CancellationToken) #{hex}.memex-cloudmemex-portal-deployment-85587f8869-5j9gxAdmin/_LogIncident/c1485fc26a61fef3What triage wrote about this fingerprint
What is failing. The
PersistentStreamPullingAgentfor memory stream queuememory-7(StreamProviders.Memory) callsIMemoryStreamQueueGrain.Dequeueon the queue grain as part of its regular pull loop. The queue grain was activated on a different silo from the agent, and that call received no response within the 30 s grain-call deadline — the receiver (MemoryAdapterReceiver.GetQueueMessagesAsync) surfaced aTimeoutExceptionand the pull iteration for that queue failed.Probable cause (moderate confidence): transient silo churn during a roll, not a defect in the receiver. The call failed as a timeout, not a fail-fast
SiloUnavailableException, meaning the target silo was still in cluster membership when the call was placed but had stopped answering — consistent with a silo departing or stalled mid-roll. The same burst window contains siblingTimeoutExceptionincidents at adjacent log sites (RoutingGrain,GrainCallCancellationManager,OrleansRoutingService, the plainMemorypulling-agent site,memory-1), and the churn window shortly before produced theroll-churn-*SiloUnavailableExceptioncluster — this reads as one echo of a rolling deploy. That is not conclusive from a single occurrence: if this fingerprint recurs outside churn windows, treat the queue grain's silo as wedged and apply the wedge signatures in Orleans Task Scheduler.Impact. Low today: one occurrence on one pod; the pull loop retries and self-heals, so the worst case is delayed delivery on the (now fallback) memory-stream leg for the duration of the stall. The class still matters:
MemoryStreamQueueGrainis RAM-backed and dies with its silo (the #2322 residual documented in Orleans Stream Pub-Sub Durability), so a stall that ends in a real departure can silently drop what the queue was holding.Where to look. Start at the Orleans receiver
MemoryAdapterReceiver.GetQueueMessagesAsync(the log site) and its callerPersistentStreamPullingAgent(sys.svc.stream.agent), then at cluster membership/departure events for the queue grain's silo around the burst. Background: Orleans Stream Pub-Sub Durability for the memory-stream leg and its residual loss windows; Pod-Hub Delivery Roll Plan for the transport that replaced it on the primary path.Possibly related. The same
memory-7site shows an earlier same-shape incident today (Admin/_LogIncident/f141b098a0af7b0c), and the same window carries sibling TimeoutException incidents (Admin/_LogIncident/49edeead95d1e6b5,f3443c4d60a505f2,c929cd46aec95c86,fb31e284cf14f42d). If those get filed or folded as duplicates of one churn event, this ticket can be closed against them in seconds.Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-85587f8869-5j9gxRecent log lines
systemorph-com commentedon Oct 10, 2026
Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.
Still occurring: 7 more since the last update (8 total).
memex-portal-deployment-85587f8869-5j9gxRecent log lines