Repository navigation
RoutingGrain fails directed delivery to a transient pod hub instead of retrying or waiting for activation #5146
Description
What is failing
RoutingGrain (Orleans routing layer) failed a directed delivery to a pod hub and returned a DeliveryFailure to the sender thread. In the captured sample the destination hub path was cache/TmXhZkCpE6r94nulQBzpQ and the sender was a UWDeepfield/IndustryNews/_Thread/… agent thread, i.e. an in-flight agent message was bounced rather than delivered.
Probable cause
Moderate confidence: the directed message targeted a pod hub that was not reachable/registered at delivery time — typically a pod restart or a hub whose activation was not (yet) present, so the routing grain surfaced a DeliveryFailure instead of retrying. The source for RoutingGrain was not found in the readable mesh partitions, so this is inferred from the message shape only, not from code. The destination path cache/<id> also looks like a short-lived/cache-scoped hub, which fits a lifecycle race rather than a persistent routing misconfiguration.
Impact
Minimal: 2 occurrences within ~100 ms on a single pod (memex-portal-deployment-d6dd69cb7-vdkcv), one affected message. No evidence of a systemic outage; the single lost delivery may have interrupted one agent round on that thread.
Where to look
The RoutingGrain directed-delivery path in MeshWeaver.Hosting.Orleans (log category MeshWeaver.Hosting.Orleans.RoutingGrain, log site [ROUTE] Directed delivery to pod hub … failed). Check whether a retry or activation-wait exists for pod hubs that are momentarily unavailable, and whether cache-scoped hub ids (cache/<id>) have a defined lifecycle the router accounts for.
Evidence
| Fingerprint | ea9a0fd2edcfe94b |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-d6dd69cb7-vdkcv |
| Occurrences | 2 |
| First seen | 2026-09-21 11:25:07Z |
| Last seen | 2026-09-21 11:25:07Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 11:25:07Z memex-portal-deployment-d6dd69cb7-vdkcv fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Directed delivery to pod hub cache/TmrXhZkCpE6r94nulQBzpQ failed — surfacing Failed DeliveryFailure to sender UWDeepfield/IndustryNews/_Thread/gather-today-s-reinsurance-news-for-the-bf27
2026-09-21 11:25:07Z memex-portal-deployment-d6dd69cb7-vdkcv fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/ea9a0fd2edcfe94b. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 21, 2026
Still occurring: 2 more since the last update (4 total).
memex-portal-deployment-d6dd69cb7-vdkcvRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 21, 2026
Still occurring: 2 more since the last update (6 total).
memex-portal-deployment-d6dd69cb7-vdkcv,memex-portal-deployment-7fc5f4874b-tl6znRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
Closes as duplicate of #2299.
Reason: same defect —
RoutingGrainsurfaces aDeliveryFailureto the sender for a directed delivery to a pod hub that is merely not activated at that instant, instead of re-resolving the route or waiting for activation. #2299 names exactly that verdict (a stale pod-hub route turned into a sender-visibleDeliveryFailureinstead of being re-resolved). This ticket carries no back-pressure readings at all — no in-flight count, no queue depth — so it is the directed-delivery defect on its own, not the saturation family at #5009/#5018.Evidence migrated to #2299: destination
cache/TmXhZkCpE6r94nulQBzpQ— a short-lived cache-scoped hub, which makes this a lifecycle race rather than a stale route; sender was an agent thread (UWDeepfield/IndustryNews/_Thread/…), so the bounced message interrupted one agent round — a user-visible consequence #2299 did not yet have. 2 occurrences within ~100 ms on podmemex-portal-deployment-d6dd69cb7-vdkcv, namespacememex, 2026-09-21 11:25Z. Fingerprintea9a0fd2edcf.Note for #2299: the same shape appears as the secondary line in #5147 (destination
portal/nodeops-9VkLPkBrRU-rwTJ4WyrfiQ, senderUWDeepfield/IndustryNews/_Thread/gather-today-s-reinsurance-news-for-the-5f7e), closed to #5009 on its head-of-line half.systemorph-com commentedon Sep 22, 2026
Reopened — this fault is happening again.
Still occurring: 2 more since the last update (8 total).
memex-portal-deployment-d6dd69cb7-vdkcv,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674pRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 22, 2026
Closing again as a duplicate of #2299, which is reopened. The maintainer's disposition of 2026-09-21T19:46Z stands; what changed is that #2299 was auto-closed at 04:49Z by the closing keyword in PR #5174's body against that body's own intent, and this bot reopen at 04:14Z is not a recurrence of anything post-fix:
cache/9A4kKKkkdUW2jb1UtTsrfQ, sender familyUWDeepfield/IndustryNews/_Thread/*and window are the same episode RoutingGrain directed delivery targets a deactivated pod-hub activation instead of re-resolving the pod hub #5177 carries under a second fingerprint (4669c3749baf06a8vs thisea9a0fd2edcfe94b— same log site, one with the exception captured and one without);fail:+surfacing Failed DeliveryFailure, which a post-fix(routing): the pod-hub leg classifies a deactivated activation instead of defaulting to terminal (#2299) #5174 image cannot emit for this rejection.The reopen procedure in
Doc/Architecture/LogWatchTriage("A REOPEN is not a recurrence") applies: a fresh fold on a pre-fix image reopens whichever ticket the fingerprint points at. If this reopens again with afail:/Failedsample from a pod whose generation post-dates a roll carrying6b3fda2a48, that is the fix not holding and belongs on #2299 with the pod'sgeneration+commitfromOps/Status.