Skip to content

RecycleCore posts DisposeRequest from the root mesh hub instead of the NodeOperationIssuingHub seam #4463

Closed

Description

@systemorph-com

What is failing

MeshOperations.RecycleCore posts its DisposeRequest directly on the root mesh hub (the DI-injected IMessageHub, which IS the router), making the router an end of a work delivery. This is the defect class that puts node-hub lifecycle work on the router's action block and competes with routing — the same shape as the 2026-06-11 prod portal wedge and issues #1113/#1121/#1136/#1140.

Probable cause (high confidence)

The RecycleCore lambda at MeshOperations.cs:4091 builds and posts the DisposeRequest without first hopping onto MeshExtensions.NodeOperationIssuingHub(). The origin-site stack names it unambiguously, and this is exactly the caller class the architecture doc calls out (a mesh-singleton operations service holding the root hub). The ratchet guard (RouterAsNodeOperationOriginRatchetGuard) only keys on node-CRUD request types — DisposeRequest is not among them — so this site was invisible to the guard and only the origin detector caught it. The recycle path was recently reworked to defer behind running installs (see Doc/WhatsNew/2026-09-13-recycling-a-node-waits-for-an-install-writing-under-it), so the un-hopped post may well have come in with that change.

Impact

One occurrence on one pod so far — low because the origin site reports only once per (role, type) per hub lifetime. It is not routine noise: every recycle issued through the operations surface posts this way, and the underlying risk is router starvation under a burst of recycles. The fix is a no-op for correctness (the seam is the identity function for any non-mesh hub).

Where to look

  • MeshWeaver.AI.MeshOperations, RecycleCore — the lambda at MeshOperations.cs:4091 that posts the DisposeRequest.
  • Fix: issue the DisposeRequest via hub.NodeOperationIssuingHub() before posting. MeshService is the reference shape (hops both issuing hub and target).
  • Rulebook: Doc/Architecture/RouterTrafficDetection.
  • Consider extending the ratchet to cover DisposeRequest so this site cannot regress.

Evidence

Fingerprint fea87cd10adfd05b
Category MeshWeaver.Messaging.MessageHub
Severity Error
Top frame MeshWeaver.AI.MeshOperations+<>c__DisplayClass95_0.<RecycleCore>b__3 (MeshOperations.cs:4091)
Namespace memex
Pods memex-portal-deployment-7cb6684584-ztqz8
Occurrences 1
First seen 2026-09-16 01:10:13Z
Last seen 2026-09-16 01:10:13Z
Recent log lines
2026-09-16 01:10:13Z memex-portal-deployment-7cb6684584-ztqz8 fail: MeshWeaver.Messaging.MessageHub[0]
      ROUTER_TRAFFIC ORIGIN: DisposeRequest was POSTED with the mesh hub as sender (sender: mesh/Mw6ZXCJcUEmJTXAXoeuGxw, target: Hosting/Triage/ci-failure/systemorph-meshweaver-35041090763). The mesh hub is the ROUTER and must not be an end of a work delivery — hop off it with MeshExtensions.NodeOperationIssuingHub() (writes) or MeshExtensions.ReadIssuingHub() (reads) before posting. Reported once per role+type for this hub. Call site:
         at MeshWeaver.AI.MeshOperations+<>c__DisplayClass95_0.<RecycleCore>b__3 (MeshOperations.cs:4091)
         at System.Reactive.Linq.ObservableImpl.Select`2+Selector+_.OnNext
         at System.Reactive.Linq.ObservableImpl.Timeout`1+Relative+_.OnNext
         at System.Reactive.Linq.ObservableImpl.Take`1+Count+_.OnNext
         at System.Reactive.SafeObserver`1+WrappingSafeObserver.OnNext
         at MeshWeaver.Messaging.AccessContextCaptureExtensions+RestoringObserver`1.OnNext (AccessContextCaptureExtensions.cs:210)
         at System.Reactive.Subjects.FastImmediateObserver`1.EnsureActive
         at System.Reactive.Subjects.ReplaySubject`1+ReplayBase.Subscribe
         at System.ObservableExtensions.SubscribeSafe
         at System.Reactive.Linq.ObservableImpl.Select`2+Selector.Run
         at System.Reactive.Producer`2.SubscribeRaw
         at System.ObservableExtensions.SubscribeSafe

Opened automatically from Admin/_LogIncident/fea87cd10adfd05b. Recurrences are folded into this issue rather than opening new ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions