JsonSynchronizationStream posts external-client SubscribeRequest from the mesh router hub instead of a ReadIssuingHub

What is failing

The JSON synchronization stream's external client posts its SubscribeRequest with the mesh router hub as the sender address (mesh/...) instead of a hub of its own. The router hub exists only to route; when it is made an origin of a work delivery, it spends its turn loop doing that work instead of routing, and under load the installation stops answering. Nothing is blocked today — the guard is a report only — but the message flow keeps the router on the delivery path for the whole lifetime of the sync subscription.

Probable cause

Confidence: high. This is the sender-side ROUTER_TRAFFIC ORIGIN report introduced 2026-09-13 (see A misrouted-message report now names the code that sent it); its call stack names the exact offender, which the receiver-side report could never do. The chain is:

MeshOperations.RenderResolvedArea (MeshOperations.cs:640) resolves a workspace read
Workspace.GetExternalClientSynchronizationStream (Workspace.cs:889)
JsonSynchronizationStream.CreateExternalClient (JsonSynchronizationStream.cs:526)
whose lambda at JsonSynchronizationStream.cs:522 subscribes a scoped observable through ImpersonationScopeExtensions.SubscribeScopedObservable (ImpersonationScopeExtensions.cs:210) — and the hub captured for the subscribe is the mesh router rather than a read-issuing hub.

The subscription observable is built while the ambient hub is the router; the code never hops off with MeshExtensions.ReadIssuingHub() before posting. Per the guard's own guidance, a read must be posted from a ReadIssuingHub(), not from the router.

Impact

Low today, systemic if left alone: one occurrence on one pod (memex portal), the first time this new sender-side guard caught this call site. The misaddressed message is still delivered and works, but every external-client sync subscription opened from RenderResolvedArea rides the router as an endpoint, adding work to the one hub whose latency is the whole installation's latency. The related receiver-side fingerprint (incident Admin/_LogIncident/b6dc7a15e653fc41, 41k+ occurrences since 2026-08-10, filed as #1140, previously #1121/#1136) shows this family of misaddressing is chronic fleet-wide; this ticket is the first one with a named, fixable call site in this path.

Where to look

MeshWeaver.Data.Serialization.JsonSynchronizationStream, the CreateExternalClient lambda at JsonSynchronizationStream.cs:522 — post the SubscribeRequest via MeshExtensions.ReadIssuingHub() instead of the ambient (router) hub.
Cross-check the sibling report MessageHub: ROUTER_TRAFFIC ORIGIN: SubscribeAck … (incident Admin/_LogIncident/fbbc7901916c5265, same second, same pod) — it is the ack leg of this same subscription and should fall with the same fix.
Related: Portal hubs send messages addressed from/to the mesh router hub instead of using their own hub identity #1140 (receiver-side report of the same router-traffic family) — this ticket is not a duplicate; it pins a specific sender the earlier ticket could only guess at.
Background: Message-Based Communication for the hub topology, and the 2026-09-13 WhatsNew note above for how this report identifies the sender.

Evidence


Fingerprint	`5397f58e61285a64`
Category	`MeshWeaver.Messaging.MessageHub`
Severity	Error
Top frame	`MeshWeaver.Data.Serialization.JsonSynchronizationStream+<>c__DisplayClass15_0`2.b__9 (JsonSynchronizationStream.cs:522)`
Namespace	`memex`
Pods	`memex-portal-deployment-6c967c7c4c-kdff5`
Occurrences	1
First seen	2026-09-17 12:58:47Z
Last seen	2026-09-17 12:58:47Z

Recent log lines

2026-09-17 12:58:47Z memex-portal-deployment-6c967c7c4c-kdff5 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: SubscribeRequest was POSTED with the mesh hub as sender (sender: mesh/q8f5WSUeqE61FTik9UiCSg, target: ClientB/CompanyProfile). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream+<>c__DisplayClass15_0`2.<CreateExternalClient>b__9 (JsonSynchronizationStream.cs:522)
         at MeshWeaver.Messaging.ImpersonationScopeExtensions+SubscribeScopedObservable`1.Subscribe (ImpersonationScopeExtensions.cs:210)
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream.CreateExternalClient (JsonSynchronizationStream.cs:526)
         at System.Lazy`1.ViaFactory
         at System.Lazy`1.CreateValue
         at MeshWeaver.Data.Workspace.GetExternalClientSynchronizationStream (Workspace.cs:889)
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass18_0.<RenderResolvedArea>b__0 (MeshOperations.cs:640)
         at System.Reactive.Linq.ObservableImpl.Defer`1+_.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_.OnNext
         at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
         at System.Reactive.Linq.ObservableImpl.ReturnImmediate`1.Run

_{Opened automatically from Admin/_LogIncident/5397f58e61285a64. Recurrences are folded into this issue rather than opening new ones.}

fix(#6047): the heartbeat's StreamEndedEvent is issued from the strea…

(re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-09-18)

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-18 07:25:23Z
Pods: memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-ztq58

Recent log lines

2026-09-17 12:58:47Z memex-portal-deployment-6c967c7c4c-kdff5 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: SubscribeRequest was POSTED with the mesh hub as sender (sender: mesh/q8f5WSUeqE61FTik9UiCSg, target: ClientB/CompanyProfile). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream+<>c__DisplayClass15_0`2.<CreateExternalClient>b__9 (JsonSynchronizationStream.cs:522)
         at MeshWeaver.Messaging.ImpersonationScopeExtensions+SubscribeScopedObservable`1.Subscribe (ImpersonationScopeExtensions.cs:210)
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream.CreateExternalClient (JsonSynchronizationStream.cs:526)
         at System.Lazy`1.ViaFactory
         at System.Lazy`1.CreateValue
         at MeshWeaver.Data.Workspace.GetExternalClientSynchronizationStream (Workspace.cs:889)
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass18_0.<RenderResolvedArea>b__0 (MeshOperations.cs:640)
         at System.Reactive.Linq.ObservableImpl.Defer`1+_.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_.OnNext
         at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
         at System.Reactive.Linq.ObservableImpl.ReturnImmediate`1.Run
2026-09-18 07:25:23Z memex-portal-deployment-bbf878dfb-ztq58 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: SubscribeRequest was POSTED with the mesh hub as sender (sender: mesh/qKt0A6hohkiBDFRb0UWZ_A, target: rsalzmann/Feedback/fdb-20260918-0723). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream+<>c__DisplayClass15_0`2.<CreateExternalClient>b__9 (JsonSynchronizationStream.cs:522)
         at MeshWeaver.Messaging.ImpersonationScopeExtensions+SubscribeScopedObservable`1.Subscribe (ImpersonationScopeExtensions.cs:210)
         at MeshWeaver.Data.Serialization.JsonSynchronizationStream.CreateExternalClient (JsonSynchronizationStream.cs:526)
         at System.Lazy`1.ViaFactory
         at System.Lazy`1.CreateValue
         at MeshWeaver.Data.Workspace.GetExternalClientSynchronizationStream (Workspace.cs:889)
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass21_0.<RenderResolvedArea>b__0 (MeshOperations.cs:674)
         at System.Reactive.Linq.ObservableImpl.Defer`1+_.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_.OnNext
         at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
         at System.Reactive.Linq.ObservableImpl.ReturnImmediate`1.Run

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-09-18)

Already fixed by #4622 — this is a DEPLOYMENT matter, not a code one

This is the SubscribeRequest (sender) leg of the single defect #4622 fixed. #4615 (the
SubscribeAck leg), #4617 (the lifecycle-event leg) and #4697 (the DataChangedEvent leg) are the
same subscription seen from the other ends — all four fingerprints fired in the same second,
2026-09-17 12:58:47Z, on pod memex-portal-deployment-6c967c7c4c-kdff5, off one
ClientB/CompanyProfile render. This issue was closed by that merge at 17:30:23Z and the
recurrence bot reopened it at 2026-09-18T07:26:26Z.

The fix is on `main`


`461dce7ca7` — "fix(mesh): a layout-area SUBSCRIPTION opens off the router (#4614/#4615/#4617)"	merged as `24f1499a87`, 2026-09-17T17:30:21Z
`git merge-base --is-ancestor 461dce7ca7 origin/main`	true
`src/MeshWeaver.Mesh.Operations/MeshOperations.cs:689`	`hub.StreamSubscribingHub().GetWorkspace()` — the comment above it names this issue by number
runtime pin	`RouterTrafficOnNodeCreateFromTheRootHubTest.ARenderAreaIssuedFromTheRootMeshHub_NeverSubscribesAsTheRouter` — reverting the fix reproduces `IsOrigin=True SubscribeRequest Role=sender Sender=mesh/…`, i.e. this exact record; with it, zero

MeshOperations.RenderResolvedArea — the frame at the bottom of this issue's own call stack
(MeshOperations.cs:640) — opened the stream on hub.GetWorkspace(), and IWorkspace is
AddScoped, so the workspace belonged to the router. That is the one address in this family a
caller chooses.

And it is running nowhere

memex.systemorph.com  /api/version -> fc8cd583  (= merge of #4621, 2026-09-17 16:41:55Z)   4 samples, identical
memex.meshweaver.cloud /api/version -> c84c6c05 (2026-09-12, 1164 commits behind main)
git merge-base --is-ancestor 461dce7ca7 fc8cd583  ->  false   (48 minutes short of the fix)
git merge-base --is-ancestor 461dce7ca7 c84c6c05  ->  false

🚨 The pod that filed the 2026-09-18 07:25:23Z recurrence started at 2026-09-17T19:42Z — after the
merge. The discriminator is the image's commit, never the pod's start time.

Disposition

Closing as fixed by #4622. The remedy is a roll onto an image built from a commit containing
461dce7ca7 — no code change.

Until that roll lands this fingerprint keeps firing and the recurrence bot will reopen this issue
again. That is expected and is NOT evidence that #4622 regressed — please do not write a second fix
for it. In particular, do not apply the remedy the ORIGIN log line prints (ReadIssuingHub()):
#4622 measured that as a regression for a subscription — portal/reads-{meshId} registers no
handlers, so it carries no RouteStreamMessage route and no sync/{streamId} sub-hub, and the
owner's fan-out would arrive nowhere, stopping data sync silently.

Full write-up: #4697 and Doc/Architecture/RouterTrafficDetection.

_{Triaged 2026-09-18 by Claude Opus 5 — read-only.}

Navigation Menu

Repository navigation

JsonSynchronizationStream posts external-client SubscribeRequest from the mesh router hub instead of a ReadIssuingHub #4614

What is failing

Probable cause

Impact

Where to look

Evidence

Already fixed by #4622 — this is a DEPLOYMENT matter, not a code one

The fix is on `main`

And it is running nowhere

Disposition

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

JsonSynchronizationStream posts external-client SubscribeRequest from the mesh router hub instead of a ReadIssuingHub #4614

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

rbuergi commented on Oct 11, 2026

rbuergi commented on Oct 11, 2026

Already fixed by #4622 — this is a DEPLOYMENT matter, not a code one

The fix is on main

And it is running nowhere

Disposition

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The fix is on `main`