Repository navigation
Content indexing issues GetDataRequest through the mesh hub instead of ReadIssuingHub — indexed document silently misses the search index #5726
Description
What is failing: The content indexing pipeline in MeshWeaver.ContentCollections.Indexing (ContentIndexingObserver.RegisterCollection, via MessageHubExtensions.Observe at MessageHubExtensions.cs:118) issues its one-shot GetDataRequest for the collection being indexed through the mesh hub itself. The mesh hub is the router and must never be the sending end of a work delivery — a bounded read has to be posted through MeshExtensions.ReadIssuingHub().
Probable cause: High confidence. The log's own diagnostic text lays out the contract and the stack trace pins the offender: ContentIndexingObserver.RegisterCollection (ContentIndexingObserver.cs:179) constructs a GetDataRequest targeting the collection namespace (Hardware) and posts it with the router as sender instead of hopping it onto the read-issuing seam. The concrete instance was indexing a PDF in the Hardware namespace. Because such deliveries fail silently, the affected file is likely silently missing from the content search index.
Impact: Single occurrence, single pod, one document. A document's search index entry is silently absent — a secondary-path defect with a trivial workaround (re-index), but the silent-failure mode makes it worth fixing properly.
Where to look: ContentIndexingObserver.RegisterCollection (ContentIndexingObserver.cs:179) and the Observe call site (ContentIndexingObserver.cs:226) — replace the hub used to post the GetDataRequest with MeshExtensions.ReadIssuingHub() (this is a read, not a node lifecycle write or stream subscription, so the other two seams do not apply).
No existing incident or issue found for this call site — this appears to be a first report.
Evidence
| Fingerprint | 29d15c79f382bcd8 |
| Category | MeshWeaver.Messaging.MessageHub |
| Severity | Error |
| Top frame | MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118) |
| Namespace | memex |
| Pods | memex-portal-deployment-56fbdcd48f-sz5st |
| Occurrences | 1 |
| First seen | 2026-09-24 09:25:06Z |
| Last seen | 2026-09-24 09:25:06Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-24 09:25:06Z memex-portal-deployment-56fbdcd48f-sz5st fail: MeshWeaver.Messaging.MessageHub[0]
ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the mesh hub as sender (sender: mesh/bCaJQ_o0BkCLQOndmSjN6A, target: Hardware). The mesh hub is the ROUTER and must not be an end of a work delivery. If the role above includes 'sender', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#4614). If the role above includes 'target', ask where that target CAME FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#4697). 'sender AND target' means both apply. Reported once per role+type for this hub. Call site:
at MeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)
at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingObserver.RegisterCollection (ContentIndexingObserver.cs:179)
at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingObserver+<>c__DisplayClass10_0.<IndexFileActivity>b__0 (ContentIndexingObserver.cs:226)
at MeshWeaver.ContentCollections.Indexing.Graph.ContentIndexingActivity+<>c__DisplayClass0_1.<Run>g__Run|1 (ContentIndexingActivity.cs:197)
at MeshWea…[truncated]
Re-addressed by the current identity function: this incident inherited 0ab99a8a5e31aa8c (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/29d15c79f382bcd8. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 40f55b17f23fa9b3: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph-com commentedon Sep 26, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-56fbdcd48f-sz5st,memex-portal-deployment-56fbdcd48f-cbms9Recent log lines
Re-addressed by the current identity function: this incident inherited
0ab99a8a5e31aa8c(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 26, 2026
Fixed in Systemorph/MeshWeaver.Plugins#2355 (merged 2026-09-24 12:16Z,
c9e7c4b77). That PR is "fix(indexing): issue the collection-config read off the router".RegisterCollectionno longer issues itsGetDataRequestfrom the mesh router.CollectionConfigReadIssuedOffTheRouterTestpins the change, and it reproduces the stack quoted in this issue.Both occurrences ran on images without the fix.
56fbdcd48f-sz5st, and 09-25 13:24:30Z on56fbdcd48f-cbms9.The fix is now deployed on both portals.
3.0.0-ci.9332since 04:06Z today. Its Plugins half is1470fbf(CD run 36098618978,HEAD is now at 1470fbf).3.0.0-ci.9349, Plugins5e8b85f.git merge-base --is-ancestor c9e7c4b77returns 0 for both.#5724 was the answering half of the same two deliveries (same pod, same second). It is closed as a duplicate of this issue.
Closing as completed. The recurrence bot reopens this issue if the stack fires on
5dbb7df97bor on any later ReplicaSet.Not established. The fix has been live for about 2 h on memex. At the observed rate of one hit per day, a quiet window that short says nothing on its own. The closure rests on the ancestry above and on the PR's negative-control test, not on a clean window.
systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
40f55b17f23fa9b3(MeshWeaver.Messaging.MessageHub), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.767f3a18a347248f40f55b17f23fa9b3MeshWeaver.Messaging.MessageHubMeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the {id} hub as sender AND target (sender: {path}, target: {path}). The {id} hub is the ROUTER and must not be an end of a work delivery. If the role above includes '{value}', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#{n}). If the role above includes '{value}', ask where that target {id} FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#{n}). '{value}' means both apply. Reported once per role+type for this hub. Call site:memexmemex-portal-deployment-65b5c866cd-6bkwhAdmin/_LogIncident/767f3a18a347248fWhat triage wrote about this fingerprint
What is failing
MeshWeaver.AI.MeshOperations.TryResolveUnifiedPath(MeshOperations.cs:1342) posts the one-shotGetDataRequestit uses to resolve a unified path from the DI-injected rootIMessageHub— which is the router — and in this occurrence the delivery was addressed at the router as well (sender and target bothmesh/{id}). Work-read traffic executing on the router's action block competes with routing; this is the same defect class as the 2026-06-11 production wedge, where node work on the router starved realSubscribeRequesttraffic.Probable cause
High confidence on the sender half — this is a known, documented residue, not a new defect. #4463 proved in production that
MeshOperations'hubfield is the router; #4477 hopped only theDisposeRequestline, leaving sibling exchanges on the identical field posting asmesh/{id}. This exact call site was filed as #4616 (then at MeshOperations.cs:1255) as one of those siblings — the "UCR read" thatDoc/Architecture/RouterTrafficDetectionstill lists un-hopped. Neither ratchet guard sees it:GetDataRequestis not a lifecycle message, and the receiver-keyed guard only covers files that have already declared a seam. The frame has moved 1255→1342, so the code around it has changed without the read being hopped.The target half is new relative to #4616's sample (where the target was an ordinary node path): here the read was addressed at the mesh hub itself. Moderate confidence, evidence points at the unified path being resolved coming out as — or falling back to — the mesh root, so the read never left the router in either direction. That derivation needs a look at how
TryResolveUnifiedPathcomputes the target address for a root-level path; if the call site genuinely chosemesh/{id}as the destination, the read should be addressed at the owning node (MeshExtensions.NodeOperationTarget()), not at the router.Impact
Single occurrence, single pod; the origin site's once-per-role+type dedup caps it at ~2 lines per process, so log volume is self-limiting. No current user-visible failure is evidenced — the risk is the documented starvation/wedge class if this read (or a burst of it) executes on the router's block. Secondary path (unified-path resolution inside AI/mesh operations) with a well-understood, low-risk fix.
Where to look
MeshWeaver.AI.MeshOperations.TryResolveUnifiedPath(MeshOperations.cs:1342) — hop the read ontohub.ReadIssuingHub()(this is a bounded one-shot READ: do not useNodeOperationIssuingHub()orStreamSubscribingHub(), the three seams are not interchangeable and the wrong one fails silently), and check the target derivation for paths that resolve to the mesh root. Reference shape:MeshService(hops bothIssuingHub+NodeOperationTarget). Background and the ratchet design that should be extended to cover this class: Router Traffic Detection (mesh:Doc/Architecture/RouterTrafficDetection).Likely tracked by #4616 — same call site and same proven-router field for the sender role; a human can fold this into #4616 if it is still open. This occurrence adds the
targetrole (read addressed at the router), which #4616's evidence did not show, so the call site is worth a look even if #4616 is closed.Reopened — this fault is happening again.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-65b5c866cd-6bkwhRecent log lines
systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
40f55b17f23fa9b3(MeshWeaver.Messaging.MessageHub), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.767f3a18a347248f40f55b17f23fa9b3MeshWeaver.Messaging.MessageHubMeshWeaver.Messaging.MessageHubExtensions.Observe (MessageHubExtensions.cs:118)ROUTER_TRAFFIC ORIGIN: GetDataRequest was POSTED with the {id} hub as sender AND target (sender: {path}, target: {path}). The {id} hub is the ROUTER and must not be an end of a work delivery. If the role above includes '{value}', this post LEFT the router: hop it onto the seam that matches what this delivery IS — MeshExtensions.NodeOperationIssuingHub() for a node LIFECYCLE write, MeshExtensions.ReadIssuingHub() for a bounded one-shot READ, MeshExtensions.StreamSubscribingHub() for a remote stream SUBSCRIPTION. The three are NOT interchangeable and the wrong one fails SILENTLY: ReadIssuingHub() registers no handlers by design, so a subscription hopped onto it stops reporting AND stops receiving data (#{n}). If the role above includes '{value}', ask where that target {id} FROM, because the two cases have opposite fixes: if this call site CHOSE the router as the destination, the call site is the offender — address the owning node instead (MeshExtensions.NodeOperationTarget()); if the target was read off an incoming request or subscription (request.Subscriber, ResponseFor(delivery) — SubscribeAck, DataChangedEvent, StreamErrorEvent and StreamEndedEvent all are), then this call site is the innocent answering half and the hub to move is the one that SUBSCRIBED or REQUESTED (#{n}). '{value}' means both apply. Reported once per role+type for this hub. Call site:memexmemex-portal-deployment-65b5c866cd-6bkwhAdmin/_LogIncident/767f3a18a347248fWhat triage wrote about this fingerprint
What is failing
MeshWeaver.AI.MeshOperations.TryResolveUnifiedPath(MeshOperations.cs:1342, viaMessageHubExtensions.Observewith a RxTimeout) issues a one-shotGetDataRequest— a bounded path-resolution read — with the root mesh hub as both sender and target. The read therefore leaves from and executes on the router's action block, the block that also carries message routing for the whole process.Probable cause
MeshOperationsholds the DI-injectedIMessageHub, which in the mesh's root container is the router — proven in production on this same field by #4463. The sender half is the familiar defect: this call site posts off the bare field instead of a seam (MeshExtensions.ReadIssuingHub()for a bounded read). The target half is genuinely ambiguous and must be answered at the call site: eitherTryResolveUnifiedPathchose the mesh hub as destination (e.g. as the resolution authority for root-scoped unified paths), or the observe helper defaulted the target to the posting hub's own address. Confidence is high on the sender half (the frame plus the field's #4463 lineage) and low on the target half — the evidence does not separate the two cases, which have different fixes, and #4697 is the recorded case of an auto-filed ticket over-generalising target-role advice.Note this exact shape — a post naming the receiver's own address — is structurally outside both
src/ratchets' denominators (the self-directed exclusion), so neither guard could ever catch it; the runtime ORIGIN line is the only instrument for it, consistent with this being a fresh sighting rather than a guard regression.Impact
No user-visible failure yet — the read keeps working, and the detector never blocks. The ORIGIN site dedups per (role, message type) per hub lifetime, so the true call frequency behind this line is unknown, and
TryResolveUnifiedPathis a hot helper for AI path resolution. The class is the one that wedged the portal in production on 2026-06-11 (work on the router's block starving realSubscribeRequestrouting), though a single one-shot read is far lighter than that CRUD burst. Secondary-path defect with an easy fix → sev:M.Where to look
MeshWeaver.AI.MeshOperations, the<TryResolveUnifiedPath>b__0lambda at MeshOperations.cs:1342;MessageHubExtensions.Observe(MessageHubExtensions.cs:118).MeshExtensions.ReadIssuingHub(); for the target, answer the doc's question at the call site ("did this call site choose the address or echo it?") before touching anything — if the address is chosen, the fix is here; if the mesh hub is genuinely the resolution authority for root-scoped paths, the execution target needs its own design.DisposeRequest— the proof the field is the router), Portal hubs send messages addressed from/to the mesh router hub instead of using their own hub identity #1140 (the sibling-exchange family on this field, of which the UCR read may be this site's parent), JsonSynchronizationStream posts DataChangedEvent with the mesh hub as target during stream initialization #4697 (cautionary precedent for target-role advice).Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-65b5c866cd-6bkwhRecent log lines
systemorph-com commentedon Oct 2, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-df798b484-cr6jnRecent log lines
systemorph-com commentedon Oct 4, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-7d8d8b5cc6-lwwvqRecent log lines
systemorph-com commentedon Oct 5, 2026
Still occurring: 1 more since the last update (4 total).
memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-7d8d8b5cc6-lwwvq,memex-portal-deployment-848f664fd5-jgkbfRecent log lines
systemorph-com commentedon Oct 6, 2026
Still occurring: 1 more since the last update (5 total).
memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-df798b484-cr6jn,memex-portal-deployment-7d8d8b5cc6-lwwvq,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-6bcdbb54fc-g9gh4Recent log lines
rbuergi commentedon Oct 8, 2026
Closing as a duplicate of #5937.
This issue's own call site,
ContentIndexingObserver.RegisterCollection, was fixed by Plugins#2355 and deployed (see the 09-26 closure above). What reopened it is the folded fingerprint767f3a18a347248f:GetDataRequest … sender AND targetatMeshOperations.TryResolveUnifiedPath (MeshOperations.cs:1342). Its five samples (2026-10-01..06) all carry that frame and none carries the indexing frame. That is the requester half of #5937, which tracks the answering half from the same exchange.Root cause and fix are in #5937 and #6293: an address-less unified path was being read from the router itself. The router is now never the target of that read. Regression test:
AnAddressLessUnifiedPathReadFromTheRootMeshHub_NeverPutsTheRouterOnEitherEnd. With the fix disabled it reproduces this issue's line.If
29d15c79f382bcd8(the indexing stack) ever recurs on an image containing Plugins#2355, that is a different defect and should reopen this issue.