Skip to content

TryResolveUnifiedPath's router-resident UCR read recurs after redeploy — duplicate of #5937 #5993

Closed as duplicate

Description

@systemorph-com

What is failing

The framework's GetDataResponse answering half of MeshOperations.TryResolveUnifiedPath's one-shot path-resolution read is still posted with the mesh router hub as both sender and target — the identical defect already filed as #5937 (this exact incident shape, Admin/_LogIncident/a54028d6db3a0c43) whose requester half is #5726.

Probable cause — an already-filed defect that has not shipped its fix (high confidence)

The top frame is the same answering lambda, DataExtensions.<HandleGetDataRequest>b__4 (the source line moved from 3339 to 3446 between builds — same method, different build). The requester half (Admin/_LogIncident/767f3a18a347248f, issue #5726) logged a matching occurrence in the same second on the same pod, so this is one execution of the same TryResolveUnifiedPath read via MessageHubExtensions.Observe off the bare DI-injected IMessageHub — the field proven to be the router in production (#4463). The fix prescribed in #5937/#5726 — hop the read onto MeshExtensions.ReadIssuingHub() — has not taken effect in the current deployment.

Impact

One (role, type) report on one portal pod. The ORIGIN site reports once per role+type per hub lifetime, so the underlying call frequency is unknown; TryResolveUnifiedPath is a hot AI path-resolution helper. No user-visible failure — the cost is router-block contention, the class that wedged the portal on 2026-06-11.

Where to look / how to close


Evidence

Fingerprint e37194f50b6abd7c
Category MeshWeaver.Messaging.MessageHub
Severity Error
Top frame MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
Namespace memex
Pods memex-portal-deployment-df798b484-cr6jn
Occurrences 1
First seen 2026-10-02 11:24:49Z
Last seen 2026-10-02 11:24:49Z
Routing not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines
2026-10-02 11:24:49Z memex-portal-deployment-df798b484-cr6jn fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/DDCGD_ql30-qafE8_aWl1Q, target: mesh/DDCGD_ql30-qafE8_aWl1Q). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
         at System.Reactive.AnonymousSafeObserver`1.OnNext
         at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
         at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
         at System.Reactive.Concurrency.Synchronize`1+_.OnNext
         at System.Reactive.Subjects.Fa…[truncated]

Opened automatically from Admin/_LogIncident/e37194f50b6abd7c. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site adea1b71074d7cb9: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Activity

  1. added
    bugSomething isn't working
    sev:MMedium - secondary path broken, or easy workaround
    on Oct 2, 2026
  2. systemorph-com commented on Oct 4, 2026

    @systemorph-com
    ContributorAuthor

    Still occurring: 1 more since the last update (2 total).

    • Last seen: 2026-10-04 00:50:47Z
    • Pods: memex-portal-deployment-df798b484-cr6jn, memex-portal-deployment-7d8d8b5cc6-lwwvq
    Recent log lines
    2026-10-02 11:24:49Z memex-portal-deployment-df798b484-cr6jn fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/DDCGD_ql30-qafE8_aWl1Q, target: mesh/DDCGD_ql30-qafE8_aWl1Q). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    2026-10-04 00:50:47Z memex-portal-deployment-7d8d8b5cc6-lwwvq fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/xSeDOFNZzE2u5lcxKJdZBw, target: mesh/xSeDOFNZzE2u5lcxKJdZBw). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    
  3. systemorph-com commented on Oct 5, 2026

    @systemorph-com
    ContributorAuthor

    Still occurring: 1 more since the last update (3 total).

    • Last seen: 2026-10-05 08:53:12Z
    • Pods: memex-portal-deployment-df798b484-cr6jn, memex-portal-deployment-7d8d8b5cc6-lwwvq, memex-portal-deployment-848f664fd5-jgkbf
    Recent log lines
    2026-10-02 11:24:49Z memex-portal-deployment-df798b484-cr6jn fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/DDCGD_ql30-qafE8_aWl1Q, target: mesh/DDCGD_ql30-qafE8_aWl1Q). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    2026-10-04 00:50:47Z memex-portal-deployment-7d8d8b5cc6-lwwvq fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/xSeDOFNZzE2u5lcxKJdZBw, target: mesh/xSeDOFNZzE2u5lcxKJdZBw). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    2026-10-05 08:53:12Z memex-portal-deployment-848f664fd5-jgkbf fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/66fTDGLBlkWM8bvQjvaKkQ, target: mesh/66fTDGLBlkWM8bvQjvaKkQ). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    
  4. systemorph-com commented on Oct 6, 2026

    @systemorph-com
    ContributorAuthor

    Still occurring: 1 more since the last update (4 total).

    • Last seen: 2026-10-06 22:38:09Z
    • Pods: memex-portal-deployment-df798b484-cr6jn, memex-portal-deployment-7d8d8b5cc6-lwwvq, memex-portal-deployment-848f664fd5-jgkbf, memex-portal-deployment-6bcdbb54fc-g9gh4
    Recent log lines
    2026-10-04 00:50:47Z memex-portal-deployment-7d8d8b5cc6-lwwvq fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/xSeDOFNZzE2u5lcxKJdZBw, target: mesh/xSeDOFNZzE2u5lcxKJdZBw). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    2026-10-05 08:53:12Z memex-portal-deployment-848f664fd5-jgkbf fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/66fTDGLBlkWM8bvQjvaKkQ, target: mesh/66fTDGLBlkWM8bvQjvaKkQ). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    2026-10-06 22:38:09Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: MeshWeaver.Messaging.MessageHub[0]
          ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/m4y4k3SEWk25p9_OEovXnA, target: mesh/m4y4k3SEWk25p9_OEovXnA). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
             at MeshWeaver.Data.DataExtensions+<>c__DisplayClass61_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3446)
             at System.Reactive.AnonymousSafeObserver`1.OnNext
             at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
             at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
             at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
             at System.Reactive.Concurrency.Synchronize`1+_.OnNext
             at System.Reactive.Subjects.Fa…[truncated]
    
  5. rbuergi commented on Oct 7, 2026

    @rbuergi
    Contributor

    Consolidated into #5937: same defect: MeshOperations.TryResolveUnifiedPath issues its one-shot UCR read from the mesh router, and this is the answering half recurring on a later image. Same fix and closing condition: hop the read onto ReadIssuingHub().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsev:MMedium - secondary path broken, or easy workaround

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions