Skip to content

IRoutingGrain grain type not registered in Orleans cluster — all RouteMessage calls fail #1109

Closed as not planned

Description

@meshweaver-cloud

What is failing

Every message routed through the Orleans cluster targeting the MeshWeaver.Connection.Orleans.IRoutingGrain interface fails because the grain type was never registered in the cluster. The StatelessWorkerDirector placement strategy finds zero silos hosting this grain, so GetCompatibleSilos throws OrleansException: No active nodes are compatible.

Probable cause

The IRoutingGrain implementation (likely RoutingGrain) is not registered as a grain class in the Orleans silo configuration at startup. This is a configuration/registration gap — the grain interface is defined and messages are being sent to [routing/default], but no silo was told to host activations of that concrete type.

Confidence: High — the cluster's own placement service reports "Known nodes with grain type: none." This is definitive: the grain class was never added to the silo's grain-class registry.

Impact

Moderate. 734 failed message deliveries across 2 pods over a 6-hour window (01:08–07:09 UTC on 2026-08-10). Every message sent to /routing/default via IRoutingGrain.RouteMessage is permanently dropped. Any higher-level messaging that depends on the routing grain (e.g., cross-grain message delivery between mesh nodes) is silently failing.

Where to look

  1. Silo configuration — find where ISiloBuilder.ConfigureServices or .AddGrain is called for the memex deployment. The grain class implementing IRoutingGrain needs to be registered there (typically via siloBuilder.AddGrain<RoutingGrain>() or a dedicated extension method).
  2. MeshWeaver.Connection.Orleans — the IRoutingGrain interface and its expected implementation live here. Check if the implementation class (RoutingGrain or similar) exists but was omitted from startup registration, or if it was never implemented.
  3. Routing grain liveness — if the grain is supposed to be registered dynamically or via a plugin system, verify that the plugin/assembly containing the grain type is loaded into the Orleans runtime before the first message arrives.

Evidence

Fingerprint 782ef3ac96819798
Category Orleans.Messaging
Severity Error
Exception Orleans.Runtime.OrleansException
Namespace memex
Pods memex-portal-deployment-7f65668566-qdk6j, memex-portal-deployment-84855f7679-z4rsp
Occurrences 894
First seen 2026-08-10 01:08:41Z
Last seen 2026-08-10 07:09:21Z
Recent log lines
2026-08-10 07:09:11Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EE73
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:11Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EE70
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:11Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EE71
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:11Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EE6E
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:11Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EE6F
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:20Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EF08
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:20Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EF09
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:21Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EF06
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:21Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EF04
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
2026-08-10 07:09:21Z memex-portal-deployment-84855f7679-z4rsp fail: Orleans.Messaging[100071]
      Failed to address message Request [S127.0.0.1:11111:145328930 sys.client/hosted-127.0.0.1:11111@145328930]->[ routing/default] MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(MeshWeaver.Messaging.IMessageDelivery) #64AFAFBD4C84EF05
      Orleans.Runtime.OrleansException: No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
         at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
         at Orleans.Runtime.Placement.StatelessWorkerDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/StatelessWorkerDirector.cs:line 10
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
      --- End of stack trace from previous location ---
         at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482

Opened automatically from Admin/_LogIncident/782ef3ac96819798. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. rbuergi commented on Aug 10, 2026

    @rbuergi
    Contributor

    Duplicate of #1107 — same defect, filed separately by the log-watcher's first live sweep (the re-filing defect is itself tracked). Keeping #1107 as canonical.

  2. systemorph-com commented on Sep 17, 2026

    @systemorph-com
    Contributor

    Reopened — this fault is happening again.

    Still occurring: 30082 more since the last update (30976 total).

    • Last seen: 2026-09-10 19:08:27Z
    • Pods: memex-portal-deployment-7f65668566-qdk6j, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-84854d8f58-2np7d, memex-portal-deployment-585f6d7b98-z7ktk, memex-portal-deployment-696468c5f4-rjz4p, memex-portal-deployment-9c798d58c-slzct, memex-portal-deployment-59ccff47c5-rjs77, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-59ccff47c5-gsbzq, memex-portal-deployment-59ccff47c5-k8bz8, memex-portal-deployment-59ccff47c5-rmp2l, memex-portal-deployment-846dfb6c98-6mst8, memex-portal-deployment-7bd5586d9c-bv48x, memex-portal-deployment-59ccff47c5-4dh52, memex-portal-deployment-846dfb6c98-4z7l2, memex-portal-deployment-7f9d94b9d4-vjqv4, memex-portal-deployment-846dfb6c98-2jtnf, memex-portal-deployment-7f9d94b9d4-wpqqf, memex-portal-deployment-846dfb6c98-l4g72, memex-portal-deployment-59ccff47c5-58zt4, memex-portal-deployment-7db656d9d9-nbfzk, memex-portal-deployment-67b48bfc55-cxhgq, memex-portal-deployment-5745957455-2xwk4, memex-portal-deployment-5745957455-78w5p, memex-portal-deployment-6b9b97fdb-6szg6, memex-portal-deployment-59ccff47c5-64fg8, memex-portal-deployment-565b67998c-spnqv, memex-portal-deployment-565b67998c-zzjl4, memex-portal-deployment-565b67998c-fw5m7, memex-portal-deployment-565b67998c-wnpvl, memex-portal-deployment-565b67998c-gjfzd, memex-portal-deployment-565b67998c-rbsd7, memex-portal-deployment-565b67998c-24vwd, memex-portal-deployment-78788d4b4b-6nhv4, memex-portal-deployment-75d6b58db5-5b57k, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-866fdb8985-lq6pz, memex-portal-deployment-565b67998c-2kvsk, memex-portal-deployment-866fdb8985-4vd82, memex-portal-deployment-5d6dbfc4c8-5vllq, memex-portal-deployment-5d6dbfc4c8-b79vf, memex-portal-deployment-5d6dbfc4c8-dcl7p, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-84df988ff7-6srrf, memex-portal-deployment-84df988ff7-c7fvv, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-89b8c6b84-5rkdc, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-7d6b864b54-7lqdg, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-64b5fbc96c-df8kr, memex-portal-deployment-7b5c857c97-cgdpw, memex-portal-deployment-6dc7cf965d-zwv4x, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-5586d5c4cb-v9kwv, memex-portal-deployment-7b5c857c97-zlc2b, memex-portal-deployment-687b6b64c9-fwktv, memex-portal-deployment-54cc7585b4-2mbtc, memex-portal-deployment-7c6d6b64f-ldn82, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-64d9857f9d-82sp9, memex-portal-deployment-df7ddfdf4-dtp44, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-df7ddfdf4-9xx48, memex-portal-deployment-df7ddfdf4-xtjhc, memex-portal-deployment-df7ddfdf4-srw4s, memex-portal-deployment-686bf68bdc-4d9s2, memex-portal-deployment-686bf68bdc-8jj6b, memex-portal-deployment-686bf68bdc-95xdt, memex-portal-deployment-686bf68bdc-plj9p, memex-portal-deployment-574b6d7fdf-nqbg4, memex-portal-deployment-574b6d7fdf-j59s6, memex-portal-deployment-686bf68bdc-gfmkh, memex-portal-deployment-79c79b87ff-5h5tz, memex-portal-deployment-79c79b87ff-xbrqf, memex-portal-deployment-6f55d94c6c-cbprp, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-6d58c699f8-2m4hp, memex-portal-deployment-6f55d94c6c-mm2kt, memex-portal-deployment-6d58c699f8-pdwwh, memex-portal-deployment-775748f44d-jn76d, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-775748f44d-5975p, memex-portal-deployment-54b59b5d89-9t4tx, memex-portal-deployment-57b689f955-h6d47, memex-portal-deployment-57b689f955-wrstj, memex-portal-deployment-775748f44d-qsc4f, memex-portal-deployment-5bbfc877dc-r8lwd, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-7969b8cfff-gtmlb, memex-portal-deployment-666b6d5b8f-tkt84, memex-portal-deployment-666b6d5b8f-g874m, memex-portal-deployment-696766b96f-6s2xs, memex-portal-deployment-696766b96f-9h7pt, memex-portal-deployment-696766b96f-mv8z4, memex-portal-deployment-7cfbf545d9-gtlhr, memex-portal-deployment-7cfbf545d9-jk5qc, memex-portal-deployment-696766b96f-bmjsh, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-76cbbdd5f-cdslq, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-x86xl, memex-portal-deployment-76cbbdd5f-6vdbb, memex-portal-deployment-569544576b-fphfv, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-wtcks, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-xqs4p, memex-portal-deployment-867dc45877-hz7wt, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-867dc45877-4gzk6, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-6tzrt, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-5b6c4f89f8-rhmkw, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-5b6c4f89f8-bhm4d, memex-portal-deployment-6f64bf4b56-djkrh, memex-portal-deployment-6f64bf4b56-vvtv6, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-zd594, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-5c885b55c6-chgxl, memex-portal-deployment-d56d6c8db-8lj5g, memex-portal-deployment-d56d6c8db-cq4wl, memex-portal-deployment-d56d6c8db-5mx4p, memex-portal-deployment-5bd5cdf556-xj5tf, memex-portal-deployment-5bd5cdf556-qdj24, memex-portal-deployment-5bd5cdf556-r7ql8, memex-portal-deployment-5954454f55-2ffnc, memex-portal-deployment-5954454f55-hgkjd, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-8556cfc4f9-dwhrx, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-f7d845cd-whszn, memex-portal-deployment-f7d845cd-9n5hf, memex-portal-deployment-6b8cfb8bfc-bnlzs, memex-portal-deployment-6b8cfb8bfc-ttw4m, memex-portal-deployment-858f57dc6-t29qk, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-9d696bb6b-cvncx, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7b67cb5577-cqzp2, memex-portal-deployment-7d8787bcc7-shmm8, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-6dbbf9c5c6-fdnds, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-7544fdd47b-bxtjf, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-7bd66d4fb5-t9l7k, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-858f57dc6-r9fqr, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-858f57dc6-55zcr, memex-portal-deployment-8447f69f67-fjp4v, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-56d4694745-w9cn8, memex-portal-deployment-56d4694745-ss8mq, memex-portal-deployment-f88b7f699-hj6qs, memex-portal-deployment-88c5c8d6c-kn9r5, memex-portal-deployment-88c5c8d6c-dn692, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-7b5f547cb7-58ckg, memex-portal-deployment-7b5f547cb7-4gh5j, memex-portal-deployment-7b5f547cb7-mjv7k, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-knbcm, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-c4d47bf4c-kmcls, memex-portal-deployment-c4d47bf4c-n9k84, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-86779d495b-gkwfl, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-2hl8l, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-6bfbc8ff4b-z47gt, memex-portal-deployment-595d8b9995-9tcnj, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-7d95465fbb-676st, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-5w66k, memex-portal-deployment-76c65fb457-z8cw8, memex-portal-deployment-76c65fb457-2w858, memex-portal-deployment-76c65fb457-jr82z, memex-portal-deployment-76c65fb457-dx4nt, memex-portal-deployment-558d86bb48-xnvjn, memex-portal-deployment-558d86bb48-nxpcc, memex-portal-deployment-558d86bb48-vl44h, memex-portal-deployment-558d86bb48-m4jkh, memex-portal-deployment-558d86bb48-vmmm7, memex-portal-deployment-788cffc666-xbmfs, memex-portal-deployment-788cffc666-swfdl, memex-portal-deployment-788cffc666-46jqr, memex-portal-deployment-5d4bfc98b7-vb64z, memex-portal-deployment-5d4bfc98b7-5r56d, memex-portal-deployment-788cffc666-9hnbx, memex-portal-deployment-5d4bfc98b7-ff447, memex-portal-deployment-788cffc666-kh7sl, memex-portal-deployment-788cffc666-9zhzw, memex-portal-deployment-788cffc666-d8nm7, memex-portal-deployment-64cc8b9b6d-z9w44, memex-portal-deployment-64cc8b9b6d-ptfdj, memex-portal-deployment-64cc8b9b6d-rk5gw, memex-portal-deployment-64cc8b9b6d-q264s, memex-portal-deployment-64cc8b9b6d-hr5hh, memex-portal-deployment-6c954fd5bf-fphtz, memex-portal-deployment-6c954fd5bf-b5q52, memex-portal-deployment-6c954fd5bf-czv78, memex-portal-deployment-6c954fd5bf-7b4xq, memex-portal-deployment-54cc8d6b4d-qsxvp, memex-portal-deployment-54cc8d6b4d-cjnqn, memex-portal-deployment-6c954fd5bf-k6q74, memex-portal-deployment-6c954fd5bf-k525s, memex-portal-deployment-6c954fd5bf-h9lnk, memex-portal-deployment-6c954fd5bf-5tk8b, memex-portal-deployment-6c954fd5bf-kmhpv, memex-portal-deployment-684546b64d-mnqsh, memex-portal-deployment-565c5d6b97-kkkgb, memex-portal-deployment-565c5d6b97-qvv8w, memex-portal-deployment-565c5d6b97-dv8vc, memex-portal-deployment-565c5d6b97-7s6pm, memex-portal-deployment-565c5d6b97-26bkz, memex-portal-deployment-565c5d6b97-wp8g7, memex-portal-deployment-565c5d6b97-zwvjs, memex-portal-deployment-565c5d6b97-lhr4b, memex-portal-deployment-7f6d66995c-dbzj8, memex-portal-deployment-7f6d66995c-tvd9l, memex-portal-deployment-7f6d66995c-r8zmq, memex-portal-deployment-7f6d66995c-jw2j2, memex-portal-deployment-d999f9888-f86ts, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-d999f9888-n6ggc, memex-portal-deployment-748c6d4456-29chg, memex-portal-deployment-748c6d4456-bn2gh, memex-portal-deployment-58b5fbf775-fjbvn, memex-portal-deployment-58b5fbf775-pvbmd, memex-portal-deployment-59ddc494b7-frdz9, memex-portal-deployment-59ddc494b7-r6gqg, memex-portal-deployment-85bd9f5d85-5fcnk, memex-portal-deployment-85bd9f5d85-6th26, memex-portal-deployment-5998589dbd-tzx4w, memex-portal-deployment-5998589dbd-9v8mj, memex-portal-deployment-5678bbc9c8-pl7dq, memex-portal-deployment-5678bbc9c8-4rvjb, memex-portal-deployment-dbc8997cd-w55hb, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-c7f588679-dt584, memex-portal-deployment-6c74bf4c66-88jg8, memex-portal-deployment-6c74bf4c66-5xvlt, memex-portal-deployment-84765b45c8-f4j9m, memex-portal-deployment-84765b45c8-fjsv2, memex-portal-deployment-fbfc4b45-mqzbp, memex-portal-deployment-fbfc4b45-w4vfr, memex-portal-deployment-856d4474c-hr5rd, memex-portal-deployment-856d4474c-4rzkz, memex-portal-deployment-dbc8997cd-mwsm6, memex-portal-deployment-7858d9f8cc-r2v5v, memex-portal-deployment-7858d9f8cc-q4wbs, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-7858d9f8cc-p5swz, memex-portal-deployment-7d8fd6f84b-47cmq, memex-portal-deployment-7d8fd6f84b-mjp87, memex-portal-deployment-7d8fd6f84b-s7sr8, memex-portal-deployment-7d8fd6f84b-slmzp, memex-portal-deployment-7d8fd6f84b-25m2m, memex-portal-deployment-7d8fd6f84b-tq9bw, memex-portal-deployment-7d8fd6f84b-2n9rb, memex-portal-deployment-7d8fd6f84b-zlzjw, memex-portal-deployment-7d8fd6f84b-4xdbt, memex-portal-deployment-7657b88b9-qwcxz, memex-portal-deployment-7657b88b9-8dvfr, memex-portal-deployment-7657b88b9-j6jf8, memex-portal-deployment-7657b88b9-crs2w, memex-portal-deployment-7657b88b9-fc4rq, memex-portal-deployment-7657b88b9-czjwt, memex-portal-deployment-7657b88b9-r7vwt, memex-portal-deployment-7657b88b9-xqt9w, memex-portal-deployment-566949787f-sdx44, memex-portal-deployment-566949787f-tqf99, memex-portal-deployment-69f86967d8-vmj29, memex-portal-deployment-c477dc566-sgkq4, memex-portal-deployment-69f86967d8-9xxb7, memex-portal-deployment-69f86967d8-54p5q, memex-portal-deployment-69f86967d8-bglpq, memex-portal-deployment-69f86967d8-9f8nx, memex-portal-deployment-69f86967d8-kdnlb, memex-portal-deployment-6f76b9c79b-d5cpv, memex-portal-deployment-6f76b9c79b-vnpw5, memex-portal-deployment-6f76b9c79b-4wtcx, memex-portal-deployment-6f76b9c79b-s6xlp, memex-portal-deployment-6f76b9c79b-9spms, memex-portal-deployment-6f76b9c79b-bjsr5, memex-portal-deployment-6f76b9c79b-gx2fc, memex-portal-deployment-865476c896-vxsr4, memex-portal-deployment-865476c896-6bv8p, memex-portal-deployment-865476c896-d64rf, memex-portal-deployment-865476c896-xbbsw, memex-portal-deployment-865476c896-strcr, memex-portal-deployment-865476c896-ssghx, memex-portal-deployment-865476c896-xf6gt, memex-portal-deployment-5f9479f848-57t9h, memex-portal-deployment-5f9479f848-fsvjr, memex-portal-deployment-5f9479f848-qdkms, memex-portal-deployment-5f9479f848-ffrpw, memex-portal-deployment-5f9479f848-8nv7z, memex-portal-deployment-5f9479f848-hmmbn, memex-portal-deployment-5f9479f848-jvt84, memex-portal-deployment-7ccdd994cf-sbrjg, memex-portal-deployment-7ccdd994cf-wwwsj, memex-portal-deployment-768b8dff99-h48bj, memex-portal-deployment-768b8dff99-cn8nn, memex-portal-deployment-5f9479f848-tr9xb, memex-portal-deployment-5f9479f848-j2fn8, memex-portal-deployment-7c6b48957d-xqzdq, memex-portal-deployment-7c6b48957d-c58hl, memex-portal-deployment-5f9479f848-wcbzd, memex-portal-deployment-7c6b48957d-n5vkr, memex-portal-deployment-7c6b48957d-vcrx8, memex-portal-deployment-7c6b48957d-xml97, memex-portal-deployment-7c6b48957d-9zc78, memex-portal-deployment-7c6b48957d-gkzfs, memex-portal-deployment-7c6b48957d-rpkhr, memex-portal-deployment-7c6b48957d-xl99h, memex-portal-deployment-767d8d6684-djzp6, memex-portal-deployment-767d8d6684-9p96r, memex-portal-deployment-7c6b48957d-fllqp, memex-portal-deployment-78cd7b4f8f-npfqs, memex-portal-deployment-78cd7b4f8f-g85gw, memex-portal-deployment-767d8d6684-s6kcf, memex-portal-deployment-767d8d6684-gdl2g, memex-portal-deployment-767d8d6684-vrc78, memex-portal-deployment-767d8d6684-n8pfw, memex-portal-deployment-767d8d6684-svh4z, memex-portal-deployment-6d55467d9d-46jfk, memex-portal-deployment-6d55467d9d-x6xr2, memex-portal-deployment-c4499cbcc-fq9xr, memex-portal-deployment-c4499cbcc-qwtl2, memex-portal-deployment-5d99f5649-t5h8n, memex-portal-deployment-5d99f5649-j995p, memex-portal-deployment-767d8d6684-5s4t8, memex-portal-deployment-cfc9f8448-d4txb, memex-portal-deployment-cfc9f8448-vjlb9, memex-portal-deployment-cfc9f8448-zkfjv, memex-portal-deployment-c759b77bc-88tzq, memex-portal-deployment-c759b77bc-mpzcg, memex-portal-deployment-77d5fcc986-g6n2f, memex-portal-deployment-77d5fcc986-xjm7d, memex-portal-deployment-67c44ccd96-z9g2b, memex-portal-deployment-67c44ccd96-f7dhr, memex-portal-deployment-8d856d8c-nbbkv, memex-portal-deployment-8d856d8c-txmhq, memex-portal-deployment-56699545bf-pk948, memex-portal-deployment-56699545bf-jzbvt, memex-portal-deployment-69cc679d7d-88wds, memex-portal-deployment-554545c8cc-xw7xn, memex-portal-deployment-554545c8cc-m7kw2, memex-portal-deployment-76c67b6b45-5n2zg, memex-portal-deployment-76c67b6b45-v4hdf, memex-portal-deployment-554545c8cc-rb8l6, memex-portal-deployment-66cfbf759-rt9v8, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-66cfbf759-cnwnb, memex-portal-deployment-76677df8f6-pgd6b, memex-portal-deployment-76677df8f6-qwnpl, memex-portal-deployment-66cfbf759-gmgl4, memex-portal-deployment-66cfbf759-v4grm, memex-portal-deployment-66f4b5c9b6-4k6d5, memex-portal-deployment-66f4b5c9b6-mcm2z, memex-portal-deployment-66cfbf759-ks9hl, memex-portal-deployment-799c674567-bszk6, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-799c674567-dg7rc, memex-portal-deployment-799c674567-p5lr8, memex-portal-deployment-658bbfd875-xpf9w, memex-portal-deployment-799c674567-cj5jd, memex-portal-deployment-658bbfd875-mvcml, memex-portal-deployment-79776f9c5f-vxtfc, memex-portal-deployment-79776f9c5f-ztq9j, memex-portal-deployment-799c674567-vj2hq, memex-portal-deployment-7767bf7cd7-rwhxm, memex-portal-deployment-7767bf7cd7-ffrhj, memex-portal-deployment-7767bf7cd7-nx4rw, memex-portal-deployment-8f976f958-j28fg, memex-portal-deployment-8f976f958-p4vvt, memex-portal-deployment-b846c4cc8-v77vj, memex-portal-deployment-b846c4cc8-xvstr, memex-portal-deployment-6458f9d966-t4xcg, memex-portal-deployment-6458f9d966-wdt74, memex-portal-deployment-66b85d6f4f-2cc9q, memex-portal-deployment-66b85d6f4f-v7mht, memex-portal-deployment-856f765d66-47gs2, memex-portal-deployment-856f765d66-l4wjb, memex-portal-deployment-b74d87c5c-5qjxx, memex-portal-deployment-b74d87c5c-n8vqj, memex-portal-deployment-7767bf7cd7-g84qz, memex-portal-deployment-7767bf7cd7-pxnd6, memex-portal-deployment-75597864c8-66ddg, memex-portal-deployment-75597864c8-vllh5, memex-portal-deployment-75597864c8-5wvgr, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-7b747dbdb-sshxj, memex-portal-deployment-7b747dbdb-5526x, memex-portal-deployment-6dd4fbd566-wv9p7, memex-portal-deployment-6dd4fbd566-gqcsm, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-9bf79b774-nvnzz, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-9bf79b774-bwgqp, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-857cc4848c-jdbx6, memex-portal-deployment-857cc4848c-9nrv2, memex-portal-deployment-75d46686fb-5gpz8, memex-portal-deployment-75d46686fb-5nw4m, memex-portal-deployment-9bf79b774-s9sr9, memex-portal-deployment-644ffc96b9-mttj7, memex-portal-deployment-644ffc96b9-58pv4, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-75c9797d76-sg6gd, memex-portal-deployment-75c9797d76-6bs4l, memex-portal-deployment-788469d8b5-v2nz4, memex-portal-deployment-788469d8b5-qz28b, memex-portal-deployment-9bf79b774-29ccp, memex-portal-deployment-dd84cc6cf-mrp8s, memex-portal-deployment-dd84cc6cf-5pxbv, memex-portal-deployment-5b7954fc59-n7vzq, memex-portal-deployment-5b7954fc59-4npbc, memex-portal-deployment-9bf79b774-klpfz, memex-portal-deployment-6c44d9fd7b-fq8tn, memex-portal-deployment-6c44d9fd7b-9pl74, memex-portal-deployment-7f6bf4c7cc-ttpjr, memex-portal-deployment-7f6bf4c7cc-2649s, memex-portal-deployment-5b4db95fdb-jtmds, memex-portal-deployment-5b4db95fdb-4h9nm, memex-portal-deployment-9d6b9745d-xth7z, memex-portal-deployment-9d6b9745d-7zrlf, memex-portal-deployment-9bf79b774-v79fv, memex-portal-deployment-9bf79b774-dst7d, memex-portal-deployment-67dbc7bc9c-cd77d, memex-portal-deployment-67dbc7bc9c-k89w4, memex-portal-deployment-67dbc7bc9c-wcg8k, memex-portal-deployment-67dbc7bc9c-fbkht, memex-portal-deployment-67dbc7bc9c-455hx, memex-portal-deployment-5f8f7cf98d-bjpvh, memex-portal-deployment-5f8f7cf98d-smhqc, memex-portal-deployment-67dbc7bc9c-7dfdf, memex-portal-deployment-64ff8dc844-ldjlz, memex-portal-deployment-64ff8dc844-clrk5, memex-portal-deployment-67dbc7bc9c-sdhf9, memex-portal-deployment-85859db964-q5zcm, memex-portal-deployment-85859db964-5zjqw, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-55547495d7-c8z9r, memex-portal-deployment-55547495d7-v2qqt, memex-portal-deployment-85859db964-qtbnz, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-85859db964-xk5gw, memex-portal-deployment-85859db964-blbdw, memex-portal-deployment-85859db964-rfjgc, memex-portal-deployment-85859db964-gjbwk, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-85859db964-vmf7t, memex-portal-deployment-85859db964-ghsbw, memex-portal-deployment-85859db964-wgtgw, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-85859db964-qmvw5, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-7b964c8947-hthdz, memex-portal-deployment-7b964c8947-pl825, memex-portal-deployment-578b7bc84b-msvh5, memex-portal-deployment-578b7bc84b-7fnp7, memex-portal-deployment-58c88885f8-h5hfs, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-85859db964-b4cfl, memex-portal-deployment-85859db964-hs2z2, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-67b654495f-6xjg9, memex-portal-deployment-7fb5c5fc9b-pqz24, memex-portal-deployment-7fb5c5fc9b-qz466, memex-portal-deployment-85859db964-bldn9, memex-portal-deployment-b964b4c4c-wmrhr, memex-portal-deployment-b964b4c4c-dwvhf, memex-portal-deployment-85859db964-7vvcz, memex-portal-deployment-556bcb747f-wwf54, memex-portal-deployment-556bcb747f-ptp82, memex-portal-deployment-75f57f8644-cv9f5, memex-portal-deployment-75f57f8644-gf57q, memex-portal-deployment-77c854cfcf-rcbtw, memex-portal-deployment-77c854cfcf-gq6fm, memex-portal-deployment-77c854cfcf-clz9l, memex-portal-deployment-77c854cfcf-pd6gq, memex-portal-deployment-85859db964-r2jbp, memex-portal-deployment-85859db964-jkpzl, memex-portal-deployment-85859db964-dv2n2, memex-portal-deployment-77c854cfcf-884zm, memex-portal-deployment-77c854cfcf-ns2r5, memex-portal-deployment-77c854cfcf-99w7f, memex-portal-deployment-85859db964-pggm2, memex-portal-deployment-77c854cfcf-k6m8k, memex-portal-deployment-77c854cfcf-p6mx7, memex-portal-deployment-85859db964-d52m6, memex-portal-deployment-b6d54dcc7-t2b5x, memex-portal-deployment-85859db964-fl8sv, memex-portal-deployment-77c854cfcf-z8v64, memex-portal-deployment-b6d54dcc7-x7lwf, memex-portal-deployment-77c854cfcf-qqz7t, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-cn6sm, memex-portal-deployment-77cfc55bfc-767mq, memex-portal-deployment-69cb4d6486-lj2sb, memex-portal-deployment-69cb4d6486-5cvgd, memex-portal-deployment-7c9696d89b-kdw5p, memex-portal-deployment-7c9696d89b-zskgh, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-zgxdd, memex-portal-deployment-5c8c476d54-55h26, memex-portal-deployment-77cfc55bfc-l9tlh, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-77cfc55bfc-bg4lk, memex-portal-deployment-77cfc55bfc-ptd5l, memex-portal-deployment-6c74846cf8-kw26d, memex-portal-deployment-5d55558945-bh6rq, memex-portal-deployment-77cfc55bfc-92tsr, memex-portal-deployment-54f4c96b6b-8xnhx, memex-portal-deployment-77cfc55bfc-q4f9r, memex-portal-deployment-54f4c96b6b-npxqc, memex-portal-deployment-77cfc55bfc-9tzlj, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-7fd44b7695-xvwnf, memex-portal-deployment-77898c4947-bfnft, memex-portal-deployment-77898c4947-6mffn, memex-portal-deployment-77cfc55bfc-pjgvx, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-77cfc55bfc-4zrfq, memex-portal-deployment-5b9cbf46dd-rvspg, memex-portal-deployment-5b9cbf46dd-pmn4w, memex-portal-deployment-5b9cbf46dd-fxbhd, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-nnp9w, memex-portal-deployment-5b9cbf46dd-gk72x, memex-portal-deployment-5b9cbf46dd-gs6j6, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-5b9cbf46dd-q8v6s, memex-portal-deployment-5b9cbf46dd-xrgsx, memex-portal-deployment-5b9cbf46dd-lgwrk, memex-portal-deployment-5fc89b764-sb7cz, memex-portal-deployment-5fc89b764-dcdcq, memex-portal-deployment-5b9cbf46dd-n67tt, memex-portal-deployment-58c859d4cc-b8pfg, memex-portal-deployment-58c859d4cc-4pqcw, memex-portal-deployment-58c859d4cc-wgcwc, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-58c859d4cc-l2rmp, memex-portal-deployment-58c859d4cc-hlv2q, memex-portal-deployment-58c859d4cc-xsvkj, memex-portal-deployment-58c859d4cc-gqqg7, memex-portal-deployment-58c859d4cc-bml8l, memex-portal-deployment-58c859d4cc-sb9qb, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-79ddd44d77-xzp5d, memex-portal-deployment-f69cd9d56-jczb7, memex-portal-deployment-58c859d4cc-w7m5t, memex-portal-deployment-58c859d4cc-5vfjp, memex-portal-deployment-f69cd9d56-hg9bc, memex-portal-deployment-f69cd9d56-qc5hz, memex-portal-deployment-79ddd44d77-qwdjd, memex-portal-deployment-6d96874b7f-ldw5f, memex-portal-deployment-6d96874b7f-7tvtr, memex-portal-deployment-6d96874b7f-6258m, memex-portal-deployment-58c859d4cc-6266q, memex-portal-deployment-58c859d4cc-7d9pw, memex-portal-deployment-58c859d4cc-fksch, memex-portal-deployment-74894fcb7-blcwh, memex-portal-deployment-7954bc594c-8dx4g, memex-portal-deployment-7954bc594c-kkk4n, memex-portal-deployment-79ddd44d77-s5wdr, memex-portal-deployment-7954bc594c-bgtf2, memex-portal-deployment-7954bc594c-gtcxz, memex-portal-deployment-7954bc594c-94nd8, memex-portal-deployment-7954bc594c-s6sl2, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-kcmxm, memex-portal-deployment-7bff7d8b54-5pczp, memex-portal-deployment-5f455dbc49-pc2cm, memex-portal-deployment-66748969b4-fv4t9, memex-portal-deployment-7bff7d8b54-p5hkc, memex-portal-deployment-64864758dc-q2kn5, memex-portal-deployment-6457bb9544-g6z76, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-w4wnn, memex-portal-deployment-7bff7d8b54-5rrkx, memex-portal-deployment-6457bb9544-hnqcf, memex-portal-deployment-74bfcd8f6b-sksm4, memex-portal-deployment-74bfcd8f6b-gkz48, memex-portal-deployment-74bfcd8f6b-4mgbg, memex-portal-deployment-74bfcd8f6b-lp9zq, memex-portal-deployment-74bfcd8f6b-tv854, memex-portal-deployment-84d966d9bd-nlv6z, memex-portal-deployment-84d966d9bd-5fx86, memex-portal-deployment-74bfcd8f6b-95bc8, memex-portal-deployment-74bfcd8f6b-dftr8, memex-portal-deployment-776b97b9fb-k98fs, memex-portal-deployment-776b97b9fb-rvgmt, memex-portal-deployment-776b97b9fb-9xq2k, memex-portal-deployment-776b97b9fb-bjtxm, memex-portal-deployment-d6bfd5dcb-qk5gf, memex-portal-deployment-d6bfd5dcb-x4hxp, memex-portal-deployment-d6bfd5dcb-glbvw, memex-portal-deployment-d6bfd5dcb-n7ptw, memex-portal-deployment-75f7864cfd-pcxnd, memex-portal-deployment-75f7864cfd-n5ws6, memex-portal-deployment-8b96fdbd4-h7cxj, memex-portal-deployment-d6bfd5dcb-vshjn, memex-portal-deployment-7d5d458cc4-cbztk, memex-portal-deployment-7d5d458cc4-6fq52, memex-portal-deployment-d6bfd5dcb-m7cj5, memex-portal-deployment-8b96fdbd4-5gvlx, memex-portal-deployment-8b96fdbd4-xpc5w, memex-portal-deployment-8b96fdbd4-5xqzc, memex-portal-deployment-6c74f8458-kcrjf, memex-portal-deployment-6c74f8458-875l4, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-6c74f8458-96jq9, memex-portal-deployment-7fd85c4468-gwghs, memex-portal-deployment-7fd85c4468-lg59f, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-jl7xh, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-59955df69b-vllwd, memex-portal-deployment-74c97ccd44-nmjks, memex-portal-deployment-74c97ccd44-6fvqj, memex-portal-deployment-5777bff985-88g9r, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-bf77c847-xsb54, memex-portal-deployment-bf77c847-zw22p, memex-portal-deployment-76c99bdc6-9stvn, memex-portal-deployment-76c99bdc6-g7f89, memex-portal-deployment-bf77c847-6zx7c, memex-portal-deployment-bf77c847-4sd7q, memex-portal-deployment-76c99bdc6-kcb4d, memex-portal-deployment-76c99bdc6-mwnnb, memex-portal-deployment-bf77c847-276kb, memex-portal-deployment-76c99bdc6-8hv7r, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-bf77c847-f286h, memex-portal-deployment-6cdf7f9f-2g8bb, memex-portal-deployment-6cdf7f9f-8dztq, memex-portal-deployment-85b6cfc9c9-p5d7p, memex-portal-deployment-bf77c847-gh7g6, memex-portal-deployment-bf77c847-n78m7, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-bq5tr, memex-portal-deployment-5f49fd47c4-g4ct8, memex-portal-deployment-5f49fd47c4-jnpm4, memex-portal-deployment-6d4564c7b8-9dv6t, memex-portal-deployment-6d4564c7b8-cvdqh, memex-portal-deployment-bf77c847-8g4vz, memex-portal-deployment-b8f65dc46-vsnwh, memex-portal-deployment-b8f65dc46-lxkzg, memex-portal-deployment-6546ff4bc8-cj78b, memex-portal-deployment-6546ff4bc8-9wcwl, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-5bf869698f-6x87d, memex-portal-deployment-bf77c847-r5gcw, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-59d8fd958f-b2z6v, memex-portal-deployment-7456f8f447-6pkwq, memex-portal-deployment-7456f8f447-sljgp, memex-portal-deployment-7456f8f447-lgtrr, memex-portal-deployment-7456f8f447-26bbj, memex-portal-deployment-7456f8f447-ctqhw, memex-portal-deployment-7456f8f447-q9w54, memex-portal-deployment-7456f8f447-pjb9h, memex-portal-deployment-7456f8f447-bs6p6, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-685fb69888-sjl6z, memex-portal-deployment-685fb69888-f8xxd, memex-portal-deployment-84d46fdb95-f9lb5, memex-portal-deployment-84d46fdb95-sd7p4, memex-portal-deployment-84d46fdb95-mvf8b, memex-portal-deployment-7ffbd9b949-nqkvb, memex-portal-deployment-84d46fdb95-82v4v, memex-portal-deployment-7ffbd9b949-h8t4d, memex-portal-deployment-7ffbd9b949-rvqk8
    Recent log lines
    2026-09-10 19:08:27Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: Orleans.Messaging[100071]
          Failed to address message Request [S10.244.2.226:11111:148072105 sys.client/hosted-10.244.2.226:11111@148072105]->[ podhub/mesh/PopyUf3yXUiOlKTTI3xJ4A] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #6C78DD3E254D47B0
          Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
             at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
             at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 22
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
          --- End of stack trace from previous location ---
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
          --- End of stack trace from previous location ---
             at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
    2026-09-10 19:08:27Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: Orleans.Messaging[100071]
          Failed to address message Request [S10.244.2.226:11111:148072105 sys.client/hosted-10.244.2.226:11111@148072105]->[ podhub/mesh/PopyUf3yXUiOlKTTI3xJ4A] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #6C78DD3E254D47BD
          Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
             at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
             at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 22
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
          --- End of stack trace from previous location ---
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
          --- End of stack trace from previous location ---
             at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 240
             at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
    2026-09-10 19:08:27Z memex-portal-deployment-7ffbd9b949-rvqk8 fail: Orleans.Messaging[100071]
          Failed to address message Request [S10.244.2.226:11111:148072105 sys.client/hosted-10.244.2.226:11111@148072105]->[ podhub/mesh/PopyUf3yXUiOlKTTI3xJ4A] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #6C78DD3E254D47BF
          Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
             at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 217
             at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 22
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<>c__DisplayClass19_0.<<GetOrPlaceActivationAsync>g__InnerGetOrPlaceActivationAsync|0>d.MoveNext() in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 535
          --- End of stack trace from previous location ---
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 509
             at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 477
          --- End of stack trace from previous location ---
             at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 482
             at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSourc…[truncated]
    
  3. rbuergi commented on Sep 18, 2026

    @rbuergi
    Contributor

    Re-closing — DUPLICATE of #1107, the canonical, which is closed with a merged fix

    Last seen 2026-09-10 19:08:27Z (the Last seen: in the bot's own reopen comment, 2026-09-17T07:52:39Z) — quiet 7.5 days as of 2026-09-18T06:05Z, while ingest is live (incidents were still being updated 2026-09-18T05:42Z).

    Duplicate of #1107 — "IRoutingGrain grain type not registered on any Orleans silo in the memex cluster", same defect, same No active nodes are compatible with grain routing and interface MeshWeaver.Connection.Orleans.IRoutingGrain. That was stated on this issue at its original close (2026-08-10) and nothing since has changed it. #1107 is the lower number and is the canonical; it is closed as completed, root-caused to a pod shutdown window (not an unregistered grain) and fixed by PR #1152.

    The fix is in current origin/main, verified by reading it rather than by the PR title — git show origin/main:src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs | sed -n '395,420p' shows step 2 of Deliver, the shutdown-window guard:

    // 2. Shutdown window. The host has begun stopping, so the silo is leaving (or has left) the ACTIVE silo set and IRoutingGrain can no longer be PLACED — the dispatch below is guaranteed to throw "No active nodes are compatible with grain routing". Answer the sender HERE instead: the failure is real … but it is TRANSIENT, so it must carry ErrorType.ShuttingDown, never the terminal ErrorType.Failed.

    i.e. the exact OrleansException this incident records is no longer produced — the router answers before dispatching to a grain that cannot be placed. (git merge-base --is-ancestor <1152 merge sha> origin/main reports NOT ancestor, which is expected here: core merges through the merge queue, which rewrites the merge commit. The code, not the sha, is the evidence.)

    The 2026-09-17T07:51–07:53Z reopen was part of a bulk wave whose predicate read an accumulated counter rather than "occurred since the close" (this issue's headline 30082 more since the last update is exactly that accumulated total) — analysed in Reading a Recurrence Reopen.

    Closing a dormant incident is safe by design: a recurrence re-opens the issue automatically with fresh evidence, so the fault cannot be lost by closing it while quiet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions