MeshOperations.TryResolveUnifiedPath issues node reads from the router hub (GetDataRequest posted with mesh/{id} as sender)

What is failing

MeshWeaver.AI.MeshOperations issues a one-shot node READ (GetDataRequest) from the DI-injected root IMessageHub — which in the mesh's root container IS the router — during unified-path resolution. The delivery leaves with mesh/{id} as sender, so work-routing traffic is posted by the router itself. This is the ROUTER_TRAFFIC ORIGIN detector doing its job: the same class of defect as the 2026-06-11 prod wedge, where node work on the router's action block starved real routing traffic.

Probable cause (high confidence)

This is a known, documented residue, not a new defect. MeshOperations' hub field was proven to be the router in production (#4463, RecycleCore posting DisposeRequest). #4477 hopped only that one line, leaving sibling exchanges on the identical field posting as mesh/{id} — the architecture doc (Doc/Architecture/RouterTrafficDetection) names three content-collection reads, the UCR read and the script dispatch as still un-hopped. The sample's call site — MeshOperations.TryResolveUnifiedPath (MeshOperations.cs:1255), an Rx Observe subscription with Timeout/Catch wrappers — is one of those sibling reads: a one-shot node read issued directly off the router hub. Neither ratchet guard sees it: GetDataRequest is not a lifecycle message, and the receiver-keyed guard only covers files that have already hopped something.

Impact

Low volume so far (single occurrence, one pod, once-per-role+type dedup makes the origin site self-limiting to ~2 lines/process). The functional risk is the documented wedge: a read executing on the router's action block competes with routing; a burst of these starves SubscribeRequest traffic. Related production precedent: 2026-06-11 portal wedge.

Where to look / fix

MeshWeaver.AI.MeshOperations.TryResolveUnifiedPath (MeshOperations.cs:1255) — hop the read onto hub.ReadIssuingHub() (MeshExtensions, resolves to portal/reads-{meshId}), which is the identity function for every non-router hub. Reference shape: MeshService (hops both IssuingHub + NodeOperationTarget). While there, sweep the remaining sibling exchanges on the same field named in the doc. Background and the ratchet design that should be extended to cover this class: Router Traffic Detection (mesh: Doc/Architecture/RouterTrafficDetection).

Possibly a duplicate of the #4463/#4477/#1140 family — same component, same proven-router field, different call site. If a tracking issue exists for the remaining MeshOperations sibling exchanges, this can be folded into it.

Evidence


Fingerprint	`0ab99a8a5e31aa8c`
Category	`MeshWeaver.Messaging.MessageHub`
Severity	Error
Top frame	`MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)`
Namespace	`memex`
Pods	`memex-portal-deployment-6c967c7c4c-kdff5`
Occurrences	1
First seen	2026-09-17 12:59:25Z
Last seen	2026-09-17 12:59:25Z

Recent log lines

2026-09-17 12:59:25Z memex-portal-deployment-6c967c7c4c-kdff5 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the mesh hub as sender (sender: mesh/q8f5WSUeqE61FTik9UiCSg, target: ClientB/ThomasMueller/_Thread/please-add-thomas-muellers-phone-nnumber-6d5a/97c5742d). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass31_0.<TryResolveUnifiedPath>b__0 (MeshOperations.cs:1255)
         at System.Reactive.Linq.QueryLanguage+CreateWithActionDisposable`1.SubscribeCore
         at System.Reactive.ObservableBase`1.Subscribe
         at System.Reactive.Linq.ObservableImpl.Timeout`1+Relative+_.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.Catch`2.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Sink`2.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.Catch`2.Run
         at System.Reactive.Producer`2.SubscribeRaw

_{Opened automatically from Admin/_LogIncident/0ab99a8a5e31aa8c. Recurrences are folded into this issue rather than opening new ones.}

Already fixed on main — this sighting is from an image that predates the fix.

The UCR read was hopped onto ReadIssuingHub() by #4487 (bb4ec4038d, "issue the agent surface's
reads and the editor's write off the router", merged 2026-09-16), whose body names it explicitly:
"MeshOperations: the UCR read, three content-collection reads and the script dispatch →
ReadIssuingHub()". On main today TryResolveUnifiedPath reads

var issuingHub = address.Equals(hub.Address) ? hub : hub.ReadIssuingHub();
…
innerSubscription = issuingHub.Observe(new GetDataRequest(reference), o => o.WithTarget(address))

— with the self-address exclusion the review round added, so a UCR that resolves back to this very
hub stays local instead of turning into a routed request the router has to execute.

Why it still fired on 2026-09-17 12:59:25Z. Neither portal is running an image that contains
it (measured just now against /api/version, ancestry checked with git merge-base --is-ancestor):

portal	running commit	dated	contains `bb4ec4038d` (#4487)?
memex.systemorph.com	`afde4eabe0740ff658f3dae8a3073c33a2b3ea02`	2026-09-15 21:50Z	no
memex.meshweaver.cloud	`c84c6c05503228860df03c4a8b596e497e6d218c`	2026-09-12 09:30Z	no

So there is nothing to change in code. Closing; the recurrence bot will stop filing it once the
portals roll onto a current image.

The three siblings from the same burst are NOT in this category — #4614 / #4615 / #4617 are one
live defect (the router as a stream SUBSCRIBER) and are being fixed; see that PR for why
ReadIssuingHub() is the wrong remedy there.

🚩 Expect this to recur until the portals roll, and read those recurrences as delivery lag rather than as a regression. The fix (#4487, bb4ec4038d) is merged, but memex.systemorph.com runs afde4ea (2026-09-15 21:50Z) and memex.meshweaver.cloud runs c84c6c0 (2026-09-12 09:30Z) — neither contains it. Every occurrence reported from those images is pre-fix code doing what it did before. Staying closed is deliberate: the code question is answered, and the outstanding item is operational, not a defect to track here.

Duplicate of #5620 — same log site MeshWeaver.Messaging.MessageHub: ROUTER_TRAFFIC[ ORIGIN]: {type} … with the mesh hub as sender/target; consolidated.

Read before closing: This issue's own call site (MeshOperations.TryResolveUnifiedPath:1255) was fixed by #4487 (bb4ec40) and has not appeared since 2026-09-17. The 2026-09-23 reopen is a DIFFERENT call site — MessageHubExtensions.Observe ← ContentIndexingObserver.RegisterCollection (ContentIndexingObserver.cs:179) ← IndexFileActivity ← ContentIndexingActivity.Run — at 20:56:19Z on 76cf776847-9p7z4, i.e. the exchange #5620 reports.

The fix, verification and any recurrence are tracked on #5620.

The 2026-09-23 20:56:19Z recurrence folded here is a DIFFERENT call site (ContentIndexingObserver.RegisterCollection, MeshWeaver.Plugins), not TryResolveUnifiedPath — tracked and fixed on #5620 (Refs Systemorph/MeshWeaver.Plugins#2355, merged).

(re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-09-23)

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-23 20:56:19Z
Pods: memex-portal-deployment-76cf776847-9p7z4, memex-portal-deployment-6c967c7c4c-kdff5

Recent log lines

2026-09-17 12:59:25Z memex-portal-deployment-6c967c7c4c-kdff5 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the mesh hub as sender (sender: mesh/q8f5WSUeqE61FTik9UiCSg, target: ClientB/ThomasMueller/_Thread/please-add-thomas-muellers-phone-nnumber-6d5a/97c5742d). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass31_0.<TryResolveUnifiedPath>b__0 (MeshOperations.cs:1255)
         at System.Reactive.Linq.QueryLanguage+CreateWithActionDisposable`1.SubscribeCore
         at System.Reactive.ObservableBase`1.Subscribe
         at System.Reactive.Linq.ObservableImpl.Timeout`1+Relative+_.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.Catch`2.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Sink`2.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.Catch`2.Run
         at System.Reactive.Producer`2.SubscribeRaw
2026-09-23 20:56:19Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the mesh hub as sender (sender: mesh/EfNTPfFIy0WLbHUEEmk3jg, target: Posts). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)
         at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingObserver.RegisterCollection (ContentIndexingObserver.cs:179)
         at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingObserver+<>c__DisplayClass10_0.<IndexFileActivity>b__0 (ContentIndexingObserver.cs:226)
         at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingActivity+<>c__DisplayClass0_1.<Run>g__Run|1 (ContentIndexingActivity.cs:197)
         at MeshWeaver…[truncated]

_{Re-addressed by the current identity function: this incident inherited 0ab99a8a5e31aa8c (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Navigation Menu

Repository navigation

MeshOperations.TryResolveUnifiedPath issues node reads from the router hub (GetDataRequest posted with mesh/{id} as sender) #4616

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MeshOperations.TryResolveUnifiedPath issues node reads from the router hub (GetDataRequest posted with mesh/{id} as sender) #4616

Description

Evidence

Activity

rbuergi commented on Sep 17, 2026

rbuergi commented on Sep 17, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions