Skip to content

Stream-subscription attach failure permanently disables a hub's cross-process routing instead of retrying the transient directory error #2633

Closed as not planned
Closed as not planned
Stream-subscription attach failure permanently disables a hub's cross-process routing instead of retrying the transient directory error#2633

Description

@meshweaver-cloud
meshweaver-cloud
bot
opened on Aug 29, 2026

What is failing

OrleansRoutingService.SubscribeWhenStreamingReadyAsync — the stream-subscription attach gate (#1081) — fails to attach a hub's Orleans memory-stream subscription when the grain-directory lookup behind the subscribe hits a momentarily unreachable silo, and then latches that hub into "cross-process routing disabled" (in-process routing stays active). Nothing re-attempts the attach, so the loss persists until the hub re-registers.

Probable cause

A transient cluster-membership / grain-directory failure treated as permanent by the subscribe path. Confidence: moderate.

  • The root exception in every sample is a ConnectionFailedException (Unable to connect to <silo> … will retry after ~0.5–0.9 s) surfaced as OrleansMessageRejectionException through LocalGrainDirectory.LookupAsync → DhtGrainLocator.Lookup → PlacementService.GetOrPlaceActivationAsync — the documented signature of grain-directory instability while cluster membership changes (see "What the swap traded" in Pod-Hub Delivery).
  • Orleans' own connection manager promises to retry the connection within a second, yet the gate latches the disable instead of riding that retry out.
  • The same exception type is already classified transient and retried on the delivery leg (RoutingGrain.IsTransientFailure matches OrleansMessageRejectionException); the subscribe leg has no equivalent — that inconsistency is the defect.
  • The affected pods span several ReplicaSet generations, consistent with rolling-deploy windows as the trigger.

I could not read the gate's source in the mesh to confirm there is no re-attach loop; the latch is inferred from the log's own "cross-process routing for this hub is DISABLED" wording and the absence of any later recovery for the affected hubs in the evidence.

Impact

Low volume, but a real functional loss each time: every occurrence disables cross-process routing for one distinct portal/… hub (a user circuit or an MCP hub), on a different pod, while in-process traffic keeps working. The affected circuit silently loses cross-silo delivery until it re-registers (reconnect or pod restart). Not an outage by the numbers — a per-user degradation, but one every rolling deploy re-triggers, logged at Critical by the framework.

If the affected pods already run the post-swap dual release this stream leg is only the fallback transport, so the disable costs the roll-window fallback; pre-swap it is the hub's only cross-silo route. Either way the attach should not latch a transient failure.

Where to look

Possibly related, not duplicates: #2357 (same unstable-directory class, but on the delivery leg — its cure does not touch the subscribe path) and #1081 (introduced this gate; this incident is about its failure handling).

Evidence
Fingerprint d8d032735045e4a5
Category MeshWeaver.Connection.Orleans.OrleansRoutingService
Severity Critical
Exception Orleans.Runtime.OrleansMessageRejectionException
Top frame MeshWeaver.Connection.Orleans.OrleansRoutingService.SubscribeWhenStreamingReadyAsync(Address address, AsyncDelivery callback, CancellationToken ct)
Namespace memex-cloud
Pods memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-856d4474c-4rzkz
Occurrences 6
First seen 2026-08-23 12:10:05Z
Last seen 2026-08-28 21:05:46Z
Recent log lines 
2026-08-23 12:10:05Z memex-portal-deployment-65fbcf6cd5-spgc9 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/xxA8-tWuVn9xmfeD4_ouGdEjKWGDmJo_Yh5yxAQxReI — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.150:11111:146475119, will retry after 582.6889ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement…[truncated]
2026-08-23 19:51:59Z memex-portal-deployment-999c9c79d-8ccj2 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/bCJCyqOYvxYbO2EGYRbnxTswnR8lhHWYbKCwlCFqgJE — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.87:11111:146498551, will retry after 900.918ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/P…[truncated]
2026-08-23 19:52:19Z memex-portal-deployment-999c9c79d-8ccj2 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/vm6N7X0ZFoSNijdO3ZF3JlQDv04pvSJppCrP1W5L1gs — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.87:11111:146498551, will retry after 884.0992ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/…[truncated]
2026-08-23 19:53:02Z memex-portal-deployment-999c9c79d-8ccj2 crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/mcp-rbuergi-5f4ce12b — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.87:11111:146498551, will retry after 875.717ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line…[truncated]
2026-08-28 21:05:46Z memex-portal-deployment-856d4474c-4rzkz crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/K2N-XE8vu55E54Sd2RKOqbrlu8IUQzkeQOrn2dTreCM — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.28:11111:146846418, will retry after 505.5975ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/…[truncated]
2026-08-28 21:05:46Z memex-portal-deployment-856d4474c-4rzkz crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/wUg6S86PSFdi9YxtcsEi02yhJw7ftQsIVSuUfdpIddQ — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.28:11111:146846418, will retry after 127.887ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/P…[truncated]

Opened automatically from Admin/_LogIncident/d8d032735045e4a5. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. rbuergi commented on Aug 29, 2026

    @rbuergi
    rbuergi
    on Aug 29, 2026
    Contributor

    PR #2645 — the attach now retries the transient, bounded and loud

    This issue's diagnosis holds exactly as written, including the part it flagged as unconfirmed:

    I could not read the gate's source in the mesh to confirm there is no re-attach loop; the latch is inferred…

    Confirmed from source. SubscribeWhenStreamingReadyAsync made one attach attempt inside a catch-all whose only outcomes were LogCritical and return null. Nothing re-attempted, and RegisterStream stores the completed task in subscriptionReady, so the disable persisted for that hub's whole life.

    The fix is the one this issue proposed — reuse the existing transient machinery rather than adding a new one:

    • the attach runs through a new AttachWithBoundedRetryAsync, gated on the existing OrleansRoutingService.IsTransientFailure (which already matches OrleansMessageRejectionException and IsDirectoryUnstable, i.e. the Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357 cure);
    • bounded — 5 re-attempts, 250 ms doubling to a 4 s cap, ≈7.75 s total. Against the ~0.5–0.9 s reconnect Orleans' own ConnectionManager promises in the very message it fails with, and well inside the 120 s readiness gate that precedes it;
    • loud — a Warning per retry naming the attempt;
    • still terminal — anything not classified transient gives up on the first attempt, and an exhausted budget still ends in the same LogCritical + latched disable. A permanent failure still fails.

    Each retry re-invokes the attach from scratch (fresh GetStream, fresh SubscribeAsync), so Orleans re-resolves the rendezvous grain rather than re-using a reference minted mid-handoff.

    One consequence worth stating

    While the attach is pending, DeliverMessage gates that hub's outbound dispatch on it (#1081), so a retrying attach holds outbound traffic for up to the budget. That is correct rather than a cost: the gate exists because a hub that cannot yet hear an answer must not ask a question. And it is a rounding error against the 120 s readiness gate already sitting in front of it.

    StreamAttachTransientRetryTest pins both directions (6 attempts for the transient, exactly 1 for a non-transient), plus a guard that the driver is the production exception type and is what the classifier calls transient — without which the retry tests could pass against nothing.

  2. rbuergi
    added 3 commits that reference this issue on Aug 29, 2026

    docs: the 'attach failed' residual is narrower now that the attach re…

    b75714b

    refactor(#2633): express the attach retry as Rx, matching AttachPodHu…

    d91493a

    Merge pull request #2645 from Systemorph/fix/orleans-stream-reliability

    67b4985

  3. rbuergi commented on Aug 30, 2026

    @rbuergi
    rbuergi
    on Aug 30, 2026
    Contributor

    Fixed by #2645, merged as 67b49859 and verified on main (merge-base --is-ancestor → on main).

    The defect was a transient grain-directory failure during
    OrleansRoutingService.SubscribeWhenStreamingReadyAsync being latched as permanent, leaving the hub
    with cross-process routing disabled until it re-registered. #2645 makes the attach retry the
    transient error instead of latching, and ships StreamAttachTransientRetryTest.cs (206 lines)
    covering it.

    Closing manually because the PR title reads fix(#2633): …, which is not a closing keyword —
    GitHub only auto-closes on Fixes #N / Closes #N. Worth noting since the same pattern left two
    other issues open after their fix merged tonight; an open issue is not evidence of an unfixed one.

    For the record, the red on that PR was unrelated: PandasExplorerLayoutAreaTest.Explorer_NoPythonNode_…,
    whose 35-project closure does not contain MeshWeaver.Connection.Orleans, so the changed assembly
    could not load in the process under test. It passed on re-run.

  4. rbuergi
    closed this as completedon Aug 30, 2026
  5. rbuergi
    added 2 commits that reference this issue on Aug 30, 2026

    fix(#2641): a re-subscribe must declare the same wire capabilities as…

    16d59d8

    fix(#2793): wait for the attach to terminate, not for its counter to …

    d61f461
  6. systemorph-com
    reopened this on Sep 17, 2026

  7. systemorph-com commented on Sep 17, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 17, 2026
    Contributor

    Reopened — this fault is happening again.

    Still occurring: 2 more since the last update (8 total).

    • Last seen: 2026-09-01 08:36:17Z
    • Pods: memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-856d4474c-4rzkz, memex-portal-deployment-85859db964-q5zcm
    Recent log lines 
    2026-08-28 21:05:46Z memex-portal-deployment-856d4474c-4rzkz crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/wUg6S86PSFdi9YxtcsEi02yhJw7ftQsIVSuUfdpIddQ — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.28:11111:146846418, will retry after 127.887ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/P…[truncated]
2026-09-01 08:36:13Z memex-portal-deployment-85859db964-q5zcm crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/Kgcye26gs8vZJg5kl2iIaHFN7aXE-Q2GO8v1s6fK9iM — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.154:11111:147250033, will retry after 65.9095ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/…[truncated]
2026-09-01 08:36:17Z memex-portal-deployment-85859db964-q5zcm crit: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Orleans 'Memory' stream subscription could not be attached for portal/claSupGaKx6ZSn2kLv-z_cSweFae3cfNt1rKYcm_gCU — cross-process routing for this hub is DISABLED (in-process routing remains active)
      Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.4.154:11111:147250033, will retry after 779.9341ms
         at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 104
         at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 229
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 527
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement…[truncated]

  8. rbuergi commented on Sep 18, 2026

    @rbuergi
    rbuergi
    on Sep 18, 2026
    Contributor

    Closing — the fix merged as #2645, and the fault has been quiet 17 days

    Dormancy. Newest occurrence Last seen: 2026-09-01 08:36:17Z (the bot's reopen comment above). Measured 2026-09-18T05:52Z that is 17 days quiet, with live log ingest (incidents updated 2026-09-18T05:42Z) and no portal roll in between.

    This issue was already closed on its fix (2026-08-30, "Fixed by #2645, merged as 67b49859") and the 2026-09-17 reopen was part of the stale-evidence sweep (#4577 / #4610). The fix is still in origin/main at 229930bf2f29845323e0f350e73829b1bdb4928e:

    $ git -C <core> grep -n '2633' origin/main -- 'src/**' 'test/**'
src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:1614:  /// How many times the stream ATTACH is re-attempted after a TRANSIENT failure — issue #2633.
src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:1665:  /// The attach, re-attempted while it fails TRANSIENTLY and the budget holds (issue #2633).
src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:1745:  // 🚨 THE ATTACH IS RETRIED, THE GATE IS NOT — issue #2633.
src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:1857:  // 🚨 Reaching here is now a REAL give-up, and that is the point of #2633: a transient ...
test/MeshWeaver.Hosting.Orleans.Test/StreamAttachTransientRetryTest.cs:20:
  /// 🚨 <b>Issue #2633 — a TRANSIENT stream-attach failure must not permanently disable a hub's ...

    So the exact defect this ticket named — a transient grain-directory rejection during SubscribeWhenStreamingReadyAsync being latched into "cross-process routing DISABLED" instead of retried — is fixed at the site the ticket pointed at, with StreamAttachTransientRetryTest.cs pinning it. Doc/Architecture/DataSyncAndCrdt records the same verdict in its log-line table: "(#2633 / #2692, fixed by #2645)".

    The reopen above cites 2026-09-01 08:36:17Z, i.e. after the 2026-08-30 close — the residual the delivery lag explains (a fix closes its issue when it merges; pods keep running the previous image for hours to days). Nothing has been seen since.

    Closing as redundant — this restores the close the fix earned. If the fault recurs, the recurrence bot reopens this issue with fresh evidence, which is the designed behaviour and why closing a dormant incident is safe.

  9. rbuergi
    closed this as not plannedon Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions