Repository navigation
MeshOperations.TryResolveUnifiedPath runs its one-shot UCR read on the router — GetDataRequest posted from and addressed at the mesh hub #5937
Description
What is failing
MeshOperations.TryResolveUnifiedPath issues its one-shot read (a GetDataRequest on the MeshOperations facade — unified-path resolution, used by the MCP verbs and page previews) through the bare DI-injected IMessageHub, which in the root container is the mesh hub. The request is posted with the router as both sender and target, so the read executes on the router's own action block and its response returns to the router — the same mechanism as the 2026-06-11 portal wedge, where work on the router's block starved real SubscribeRequest traffic.
Probable cause (high confidence on the shape, from the frames and the architecture record; the source is not in this mesh, so the call site itself needs one look)
This incident is the response half of the pair: DataExtensions.<HandleGetDataRequest>b__4 (DataExtensions.cs:3339) is the framework answering via ResponseFor(delivery) — the innocent answering half, since a reply goes where the request came from. The requester half was fingerprinted in the same second on the same pod (sibling incident Admin/_LogIncident/767f3a18a347248f) and names the offender: MeshOperations.TryResolveUnifiedPath (MeshOperations.cs:1342) posting via hub.Observe off the bare hub field. That field is proven to be the router in production (#4463: RecycleCore's DisposeRequest off the identical field). Fixing the requester — hop the read onto MeshExtensions.ReadIssuingHub() (the seam for a bounded one-shot READ) and address it at the owning node rather than mesh/{id} — silences both reports at once; both seams are the identity function for a non-router hub, so this is a no-op everywhere else.
Impact
One (role, type) report on one portal pod. Note the origin site reports once per role+type per hub lifetime, so this is one call-site class, not one event — the path runs on every render that resolves a unified path. Nothing user-visible is broken today; the cost is router-block contention, which is silent until a burst wedges the portal.
Where to look
MeshWeaver.AI.MeshOperations.TryResolveUnifiedPath—MeshOperations.cs:1342— thehub.Observe(...)posting theGetDataRequest.- Router Traffic Detection — the seam table and the role-dependent remedy;
ReadIssuingHub()is the right seam for this delivery, and hopping a subscription onto it would be the silent-loss failure mode, so the seam choice here matters. - Worth checking why
RouterAsRouterCapableReceiverRatchetGuarddid not flag this file:MeshOperations.cshas declared itshubrouter-capable since fix(mesh): issue the recycle teardown off the router, and derive the lifecycle guard's denominator (#4463) #4477 hoppedRecycleCoreon the same field, so a bare targeted post from it should be in that guard's denominator — unless this post's target spelling falls into the structural self-directed exclusion.
Related
- Sibling incident
Admin/_LogIncident/767f3a18a347248f(theGetDataRequestrequester half) — same defect, one ticket can close both. - RecycleCore posts DisposeRequest from the root mesh hub instead of the NodeOperationIssuingHub seam #4463 (same field,
DisposeRequest— the production proof the field is the router), fix(mesh): issue the recycle teardown off the router, and derive the lifecycle guard's denominator (#4463) #4477 (the hop on the sibling line). Not a duplicate of any filed issue.
Evidence
| Fingerprint | a54028d6db3a0c43 |
| Category | MeshWeaver.Messaging.MessageHub |
| Severity | Error |
| Top frame | MeshWeaver.Data.DataExtensions+<>c__DisplayClass57_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3339) |
| Namespace | memex |
| Pods | memex-portal-deployment-65b5c866cd-6bkwh |
| Occurrences | 1 |
| First seen | 2026-10-01 00:20:11Z |
| Last seen | 2026-10-01 00:20:11Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-01 00:20:11Z memex-portal-deployment-65b5c866cd-6bkwh fail: MeshWeaver.Messaging.MessageHub[0]
ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target (sender: mesh/iPPS4W--bUmoMZQ9pd0ZTQ, target: mesh/iPPS4W--bUmoMZQ9pd0ZTQ). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
at MeshWeaver.Data.DataExtensions+<>c__DisplayClass57_0.<HandleGetDataRequest>b__4 (DataExtensions.cs:3339)
at System.Reactive.AnonymousSafeObserver`1.OnNext
at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
at System.Reactive.Linq.ObservableImpl.SelectMany`2+ObservableSelector+_+InnerObserver.OnNext
at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
at System.Reactive.Concurrency.Synchronize`1+_.OnNext
at System.Reactive.Subjects.Fa…[truncated]
Opened automatically from Admin/_LogIncident/a54028d6db3a0c43. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site da430b76f61a3bdf: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
rbuergi commentedon Oct 7, 2026
Consolidation: #5993 (and its byte-identical twin #5992) folded in here. #5993 was this same response-half incident,
DataExtensions.<HandleGetDataRequest>b__4answeringTryResolveUnifiedPath's router-resident read, recurring after a redeploy. Its source line moved 3339 to 3446, same method. Sightings carried over (ROUTER_TRAFFIC ORIGIN: GetDataResponse was POSTED with the mesh hub as sender AND target):memex-portal-deployment-df798b484-cr6jn…-7d8d8b5cc6-lwwvq…-848f664fd5-jgkbf…-6bcdbb54fc-g9gh4That is 4 occurrences on 4 successive images. The requester half keeps folding into #5726, whose original indexing call site was fixed by Plugins#2355. The prescribed fix, hopping the read onto
MeshExtensions.ReadIssuingHub(), had not taken effect as of the last sighting. Recurrence comments for that fingerprint will keep landing on the closed #5993.rbuergi commentedon Oct 8, 2026
Root cause found and reproduced. The fix is in #6293.
The read was not leaving the router because a seam was missing.
TryResolveUnifiedPathalready issued node-addressed reads onReadIssuingHub(). The problem was the target. A unified path with no address part (content/x.md,schema/,data/,layoutAreas/, e.g. from an agent chat with no context) fell back tohub.Address. For the agent surface that is the root mesh hub, so the read was a self-read on the router. The self-directed exclusion kept it outside bothsrc/ratchets.Reproduction:
new MeshOperations(Mesh).Get("schema/")from the root hub (RouterTrafficOnNodeCreateFromTheRootHubTest) captures both production lines down to the frame:The router answered
content/…with "collection not found",data/with null andlayoutAreas/with[]. It owns nothing a unified path can name.Fix: a read whose resolved target is a mesh-typed address is now answered with no delivery:
Error: '…' names no node — a unified path is '{nodePath}/…'. A five-path theory pins it. With the check disabled it fails 5/5 with the production pair. A companion test keeps node-addressed reads working and off the router.This stays open until the fix is verified on the deployed portal, i.e. the fingerprint stops recurring on an image that carries #6293.
docs(router): #5937 read used the seam conditionally, the bare branch…
Merge pull request #6293 from Systemorph/fix/one-shot-reads-off-router
rbuergi commentedon Oct 9, 2026
Reading, 2026-10-09: fixed and deployed, but the zero does not discriminate yet.
15286023d8) is in memex.systemorph.com's core8b791643f1. It has been live since ci.10249, around 2026-10-08 15:00Z.Ops/Actions/logs-sevM-a4315-20261009-q5(pod…-6bcdbb54fc-g9gh4) lands 2ROUTER_TRAFFIClines from the 10-06 sighting.Ops/Actions/logs-sevM-a4315-20261009-q2covers all memex pods, 2026-10-07 14:32Z → 2026-10-09 14:52Z, not truncated, and returns 0ROUTER_TRAFFIClines.Close condition: about 4 more days of zero on memex, or a roll of memex-cloud (still on core
57a6e5fd15, without #6293) followed by the same reading there.rbuergi commentedon Oct 9, 2026
memex-cloud now carries #6293 too. The zero still does not discriminate; re-check 2026-10-13.
Roll context (read 2026-10-09 ~20:00Z). memex.meshweaver.cloud (memex-cloud) is off
3.0.0-ci.10148. Its last old-generation pod (6b756548d4-gz6rn) shut down at 18:18:09Z. Since then it has run six generations, all on images that carry the fixes:8fc6bfb94(booted ~18:00Z),9fdfd5,5f54df,c999c9,c6b6b7(ci.10340) and884964bb7(ci.10344, booted 19:47–19:50Z)./api/versionreads coreb5b640f1.Ops/Status/memex-cloud(SampleOps/Actions/sample-memex-cloud-20261009-2000-sevm) shows 3/3 ready on ci.10344. The rolls produced 35Application is shutting downlines in 17:55–20:00Z (Ops/Actions/sevm-20261009-2000-cloud-lifecycle, 83 lines, not truncated).15286023d8(#6293) is an ancestor ofb5b640f1.Ops/Actions/sevm-20261009-2005-cloud-ui-54h(10-07 13:02Z → 10-09 20:02Z, filter includesROUTER_TRAFFIC ORIGIN: GetData) reads 0 on memex-cloud. About 53 h of that window predate the roll, and they also read 0. The last memex-cloud sighting was 10-06 22:38Z, so this is the same weak zero as memex's.Close condition (unchanged): zero on both portals through 2026-10-13, about 4 days at the pre-fix rate of one per day. Query
ROUTER_TRAFFIC ORIGIN: GetDataon namespacesmemexandmemex-cloud.rbuergi commentedon Oct 11, 2026
Closing: no recurrence on either portal since the fix went live, over more pod generations than the fault ever skipped before.
e438ab89e7(2026-10-11 ~11:00Z); fix(router): one-shot reads never execute on the router (#5937, #5958) #6293 (15286023d8) is an ancestor. It has been live on memex since about 2026-10-08 15:00Z and on memex-cloud since 2026-10-09 18:18Z.Admin/_LogIncident/767f3a18a347248f(request half) ande37194f50b6abd7c/a54028d6db3a0c43(response half), were last touched 2026-10-06 22:51Z. Listing everyROUTER_TRAFFICincident by last modification shows nothing of theGetDatashape after that.6a3f67ac241e4f35(aROUTER_TRAFFIC ORIGINline of another shape, ROUTER_TRAFFIC ORIGIN detector misreports the router's own delivery-failure NACK when the failed delivery originated from the mesh hub #5713) on 2026-10-10 16:12Z, and about 200 other incidents since 2026-10-08 12:00Z.ROUTER_TRAFFICon memex: 0 (Ops/Actions/wA-20261011-memex-router-24h). On memex-cloud: 1, the ROUTER_TRAFFIC ORIGIN detector misreports the router's own delivery-failure NACK when the failed delivery originated from the mesh hub #5713 NACK and nothing else (Ops/Actions/wA-20261011-cloud-router-24h).Why this zero discriminates now. The guard reports once per role and type per hub lifetime, so the unit is the pod, not the day. Before the fix the pair appeared on every image in turn: 10-01, 10-02, 10-04, 10-05, 10-06, five sightings on five successive images. Since the fix both portals have rolled repeatedly (memex-cloud alone went through six generations on 10-09) and none of those new router hubs reported it.
RouterTrafficOnNodeCreateFromTheRootHubTestkeeps the reproduction (new MeshOperations(Mesh).Get("schema/")from the root hub) and fails 5/5 with the check disabled.A wider
Logswindow (72 h) was refused by Loki with 429 during this sweep, so the Loki half covers 24 h only; the incident feed is the instrument for the days before.