Memory stream queue dequeue intermittently exceeds 30s response deadline (symptom of cluster-wide Orleans call timeouts)

What is failing

The Orleans memory-stream pulling agent's periodic read of queue memory-5-0xA0000005 timed out: the IMemoryStreamQueueGrain.Dequeue call between pods (10.244.5.235 → 10.244.2.157) exceeded the 30s response deadline, and MemoryAdapterReceiver.GetQueueMessagesAsync surfaced it through the agent's retry executor. This is an Orleans framework log site (Orleans.Streams.Memory), not application code.

Probable cause

Most likely a symptom of cluster-wide slow/pod-to-pod call latency or grain queue contention rather than a defect in the memory stream provider itself — the same window shows many sibling TimeoutException incidents across unrelated Orleans components (MessageHub, RoutingGrain, ActivityTracking, other memory-N queues), which points to general request-path pressure (CPU saturation, silo-to-silo messaging delays) rather than a fault in MemoryAdapterReceiver. Confidence: medium; a single occurrence with no inner exception or further stack cannot confirm it.

Impact

One occurrence on one pod (memex-portal-deployment-5c99d79997-xs87x) at 2026-09-22 19:13 UTC. The pulling agent retried and no message loss is indicated. Low standalone impact; its significance is as one data point in the broader timeout family.

Known / duplicate

This burst was folded from incident Admin/_LogIncident/3a8dfa6d2e5c4d2d (same log site, 61 revisions) — that incident is the primary record for this symptom. If that one is already filed, close this ticket as a duplicate.

Where to look

Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs (GetQueueMessagesAsync, line 45 in the frame)
Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs (ReadFromQueue)
Cluster health around firstSeen: check CPU/memory pressure and silo-to-silo latency on the memex pods.

Evidence


Fingerprint	`4a95a54311e22598`
Category	`Orleans.Streams.Memory`
Severity	Error
Exception	`System.TimeoutException`
Namespace	`memex`
Pods	`memex-portal-deployment-5c99d79997-xs87x`
Occurrences	1
First seen	2026-09-22 19:13:09Z
Last seen	2026-09-22 19:13:09Z
Routing	not determined — no configured route matches the category `Orleans.Streams.Memory`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-22 19:13:09Z memex-portal-deployment-5c99d79997-xs87x fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-5-0xA0000005
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.235:11111:149102958 sys.svc.stream.agent/10.244.5.235:11111@149102958+Memory_2_memory-5-0xA0000005]->[S10.244.2.157:11111:149103361 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #374A14BEFB3642A9. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

_{Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/4a95a54311e22598. Recurrences are folded into this issue rather than opening new ones.}

Still occurring: 2 more since the last update (3 total).

Last seen: 2026-09-22 19:13:09Z
Pods: memex-portal-deployment-5c99d79997-xs87x, memex-portal-deployment-5c444645f8-8pdzb

Recent log lines

2026-09-22 19:13:09Z memex-portal-deployment-5c99d79997-xs87x fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-5-0xA0000005
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.235:11111:149102958 sys.svc.stream.agent/10.244.5.235:11111@149102958+Memory_2_memory-5-0xA0000005]->[S10.244.2.157:11111:149103361 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #374A14BEFB3642A9. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-22 14:41:34Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-0-0x00000000
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.129:11111:149091044 sys.svc.stream.agent/10.244.5.129:11111@149091044+Memory_1_memory-0-0x00000000]->[S10.244.9.90:11111:149090968 memorystreamqueue/be8eb06fb48875590000000000000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #662B61156B4865E6. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 236
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-22 14:41:34Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-7-0xE0000007
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.129:11111:149091044 sys.svc.stream.agent/10.244.5.129:11111@149091044+Memory_2_memory-7-0xE0000007]->[S10.244.9.90:11111:149090968 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #662B61156B4865E5. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 236
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

_{Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Duplicate of #4802.

Same code site: the Orleans memory-stream reader, PersistentStreamPullingAgent.ReadFromQueue → MemoryAdapterReceiver.GetQueueMessagesAsync → IMemoryStreamQueueGrain.Dequeue on a queue grain hosted by another silo.

This issue's own discriminator: memex 19:13:09Z, pod 5c99d79997-xs87x, queue memory-5, 30 s TimeoutException. The event behind it: the target 10.244.2.157:…:149103361 is the incarnation of 7qdb4 from before its restart. Both memex pods show restarts: 1 and booted at 19:15:01Z / 19:15:07Z.

The reader recovers without help. In decompiled Orleans.Streaming 10.3.1, PumpQueue retries ReadFromQueue up to 6 times with backoff, and if it gives up, the agent's grain timer starts a new pump on its next tick. This issue's samples show no Giving up reading line. What is lost is the RAM-resident queue content on the silo that left or stalled, which is the structural exposure #4802 tracks (the AddMemoryStreams retirement order is recorded there). The full per-issue table is on #4802.

Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

Still occurring: 2 more since the last update (5 total).

Last seen: 2026-09-23 14:22:22Z
Pods: memex-portal-deployment-5c99d79997-xs87x, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-777677694-hjkc8

Recent log lines

2026-09-22 14:41:34Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-7-0xE0000007
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.129:11111:149091044 sys.svc.stream.agent/10.244.5.129:11111@149091044+Memory_2_memory-7-0xE0000007]->[S10.244.9.90:11111:149090968 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #662B61156B4865E5. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 236
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-23 14:22:21Z memex-portal-deployment-777677694-d4j8c fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-4-0x80000004
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.3.49:11111:149177715 sys.svc.stream.agent/10.244.3.49:11111@149177715+Memory_2_memory-4-0x80000004]->[S10.244.9.136:11111:149177569 memorystreamqueue/be8eb06fb48875590400008004000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3D3A579981D997A3. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-23 14:22:22Z memex-portal-deployment-777677694-hjkc8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 0th time reading from queue memory-7-0xE0000007
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.246:11111:149177785 sys.svc.stream.agent/10.244.5.246:11111@149177785+Memory_2_memory-7-0xE0000007]->[S10.244.9.136:11111:149177569 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #1C4882488141FB87. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

_{Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

Still occurring: 4 more since the last update (9 total).

Last seen: 2026-09-23 14:23:45Z
Pods: memex-portal-deployment-5c99d79997-xs87x, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-777677694-hjkc8

Recent log lines

2026-09-23 14:23:03Z memex-portal-deployment-777677694-hjkc8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 1th time reading from queue memory-7-0xE0000007
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.246:11111:149177785 sys.svc.stream.agent/10.244.5.246:11111@149177785+Memory_2_memory-7-0xE0000007]->[S10.244.9.136:11111:149177569 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #1C4882488141FFB0. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-23 14:23:44Z memex-portal-deployment-777677694-d4j8c fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-4-0x80000004
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.3.49:11111:149177715 sys.svc.stream.agent/10.244.3.49:11111@149177715+Memory_2_memory-4-0x80000004]->[S10.244.9.136:11111:149177569 memorystreamqueue/be8eb06fb48875590400008004000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3D3A579981D9A5B4. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-23 14:23:45Z memex-portal-deployment-777677694-hjkc8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-7-0xE0000007
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.246:11111:149177785 sys.svc.stream.agent/10.244.5.246:11111@149177785+Memory_2_memory-7-0xE0000007]->[S10.244.9.136:11111:149177569 memorystreamqueue/be8eb06fb4887559070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #1C4882488141E31C. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

_{Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Navigation Menu

Repository navigation

Memory stream queue dequeue intermittently exceeds 30s response deadline (symptom of cluster-wide Orleans call timeouts) #5298

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Memory stream queue dequeue intermittently exceeds 30s response deadline (symptom of cluster-wide Orleans call timeouts) #5298

Description

Evidence

Activity

systemorph-com commented on Sep 22, 2026

rbuergi commented on Sep 23, 2026

systemorph-com commented on Sep 23, 2026

systemorph-com commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions