Repository navigation
MeshNodeHubFactory binds Feedback plugin instances against the Store/Plugin root, leaving instances with no NodeType and an error overlay #5264
Description
What is failing
MeshNodeHubFactory.EnrichWithNodeType resolves the type for node instances under the Feedback space against the path Feedback — but that path is occupied by the space's Store/Plugin root, not a NodeType declaration. Instances of the Feedback NodeType (e.g. rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub) therefore have no type to bind to and are served the error overlay, i.e. those instances are broken in the portal. The factory's fallback recovery (As<NodeTypeDefinition>) also throws JsonException on the plugin content, so it logs a fail rather than recovering.
Probable cause
Confidence: high that this is the same classification defect already filed as #2231 (incident Admin/_LogIncident/b2ba8598f166822f, which this incident supersedes and whose reporter fingerprint folded into this one): the hub factory's dispatch treats the Feedback Store/Plugin root as a NodeType-hub candidate even though the actual NodeType declaration lives at Feedback/Feedback — the name collision between plugin root and child type is what confuses resolution. This burst adds the previously-missing evidence of user impact: a concrete instance gets the error overlay instead of binding its type, and the As<NodeTypeDefinition> recovery path throws JsonException on PluginContent. The MeshNodeHubFactory source is not in the mesh, so the exact method could not be read; start where EnrichWithNodeType resolves the type path and gate it on the resolved node actually being a NodeType declaration before binding.
Impact
Three occurrences on one pod within 45 seconds, at the moment a user's Feedback instance was opened. Not a broad outage, but each occurrence is a real instance rendered unusable (error overlay). Low volume so far; #2231 shows the same site has been logging related failures since late August across many pods, so the underlying defect is not self-limiting.
Where to look
MeshWeaver.Graph.Configuration.MeshNodeHubFactory — EnrichWithNodeType and its type-path resolution. Live example: plugin root at Feedback (Store/Plugin, content PluginContent) versus the actual declaration at Feedback/Feedback (NodeType).
Likely a duplicate of #2231 — if that issue's fix already covers type binding (not only the unchecked cast), this can be closed against it; the new evidence here is the user-visible error-overlay impact and the JsonException in the recovery path.
Evidence
| Fingerprint | a6971f09d7d241f9 |
| Category | MeshWeaver.Graph.Configuration.MeshNodeHubFactory |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-5c99d79997-7qdb4 |
| Occurrences | 3 |
| First seen | 2026-09-22 17:48:06Z |
| Last seen | 2026-09-22 17:48:51Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Graph.Configuration.MeshNodeHubFactory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 17:48:06Z memex-portal-deployment-5c99d79997-7qdb4 fail: MeshWeaver.Graph.Configuration.MeshNodeHubFactory[0]
EnrichWithNodeType: path 'Feedback' is occupied by a node that is not a NodeType declaration ('Feedback' is a 'Store/Plugin' node) — instance 'rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub' has no type to bind to; applying error overlay
2026-09-22 17:48:06Z memex-portal-deployment-5c99d79997-7qdb4 fail: MeshWeaver.Graph.Configuration.MeshNodeHubFactory[0]
As<NodeTypeDefinition> for Feedback could not recover value: JsonException. Content and exception details withheld.
2026-09-22 17:48:51Z memex-portal-deployment-5c99d79997-7qdb4 fail: MeshWeaver.Graph.Configuration.MeshNodeHubFactory[0]
As<NodeTypeDefinition> for Feedback could not recover value: JsonException. Content and exception details withheld.
Re-addressed by the current identity function: this incident inherited b2ba8598f166822f (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/a6971f09d7d241f9. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 22, 2026
Read against
main(ad0d528876): two halves, one fixed on a PR, one an owner actThis is not a factory resolution bug, and the "no fallback" finding of #5008 still holds.
MeshNodeHubFactoryasks about the ONE path the instance names;rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubnames the bareFeedback, which the Store plugin root occupies. Refusing it with an overlay that names both sides is the intended answer (Doc/Architecture/DanglingNodeTypes→ Hole C). The first log line of this burst is that refusal, working.The second and third lines are a real defect still on
main, fixed in #5308. #5008 took the logger off every predicate on the occupied-path route, but missedArmOverlaySelfHeal: both refusals arm it (null gate) over the occupant's stream, and itsEvaluateconverted every emission with the logger. So each emission of the plugin root loggedAs<NodeTypeDefinition> for Feedback could not recover value: JsonExceptionatErrorfor the watcher's life — the 17:48:06Z and 17:48:51Z lines, and the fingerprint that kept #2231 reopening. Control in #5308: 3 Error lines on the unfixed watcher, 0 with the fix; the watcher still recycles once a real declaration lands.Owner act — repairing the instance (not done; I cannot read the control instance, which answers
503from here right now): a full-nodeupdateofrbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubwithnodeType: Feedback/Feedback(the repair row in DanglingNodeTypes → The repair path;patchrefusesnodeType). Then recycle that address.What I did NOT establish
c55301f3(pre-Feedback NodeType resolution lands on the plugin node's PluginContent when the definition is absent from the tenant partition #5008) at 12:05Z per A new pod waits forever on a partition deleted long ago — the readiness gate holds on a NotFound it treats as transient #5073's thread, and the incident's podmemex-portal-deployment-5c99d79997-7qdb4has an image I could not read. If it was created after Feedback NodeType resolution lands on the plugin node's PluginContent when the definition is absent from the tenant partition #5008 was live, a write boundary still admits the bare path and that is a separate defect — the check is the instance'screatedtime against the pod's image.Feedbackinstances on memex.meshweaver.cloud nameFeedback/Feedback; the one bare one there isrbuergi/Feedback/20260920-1207-…(read today,nodeType: Feedback, v3).Not closing:
sev:Hcloses on post-roll verification (no newAs<NodeTypeDefinition> for Feedbackline after a roll carrying #5308, plus the instance repaired).rbuergi commentedon Sep 23, 2026
Status 2026-09-23 ~04:35Z — #5308 is live on
memex-cloudonly; this incident's namespace (memex) does not run it yet.memex.meshweaver.cloud→39f49494eb;45af88a2a8(Merge fix(nodetype): the collision overlay's self-heal watcher stops re-emitting the #2231 fingerprint #5308) is an ancestor.memex.systemorph.com→3da3423c4b; fix(nodetype): the collision overlay's self-heal watcher stops re-emitting the #2231 fingerprint #5308 is NOT an ancestor. The incident (a6971f09d7d241f9) was on namespacememex, i.e. this instance.Both remaining steps are unchanged and both need the control instance, which answers 503 now: (1) after
memexrolls onto a build carrying45af88a2a8, no newAs<NodeTypeDefinition> for Feedback could not recover valueline; (2) the owner's full-nodeupdateofrbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubtonodeType: Feedback/Feedback, then a recycle of that address. Staying open.rbuergi commentedon Sep 23, 2026
Status 2026-09-23: #5308 is live on memex-cloud but not on memex, and every recurrence comes from memex
/api/versionnow reads:39f49494eb, which contains fix(nodetype): the collision overlay's self-heal watcher stops re-emitting the #2231 fingerprint #5308 (45af88a2a8, merged 2026-09-22 19:52Z). Ancestry checked locally withgit merge-base --is-ancestor.3da3423c4b, which does not contain it.The newest samples of this fingerprint (on #5008's thread, last seen 2026-09-22 19:08:57Z,
As<NodeTypeDefinition> for Feedback: value is PluginContent) all come from memex pods (memex-portal-deployment-5c99d79997-*). So every sample predates the fix on the portal that emitted it.What closes this (unchanged). memex rolls to a build that contains
45af88a2a8, and after that there is no newAs<NodeTypeDefinition> for FeedbackError line. The owner act from the previous comment is also still open: a full-nodeupdateofrbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubwithnodeType: Feedback/Feedback, then recycle that address. Both need the control instance, whose MCP answers 503 this session. Not closing:sev:H.rbuergi commentedon Sep 23, 2026
Status 2026-09-23 ~18:20Z: #5308 now runs on
memex, and nothing new has been folded in the window the pipeline coversmemex(the incident's namespace) now runs3.0.0-ci.9218, core39f49494, which contains45af88a2a8(fix(nodetype): the collision overlay's self-heal watcher stops re-emitting the #2231 fingerprint #5308). Source:Ops/Status/memexat 16:34Z. Both 9218 replicas booted at 05:15Z and 05:24Z.namespace:Admin/_LogIncident name:*Feedback*on the control instance: the twoAs<NodeTypeDefinition> for Feedbackincidents (2b608cd1880c7d52,b2ba8598f166822f) were last modified at 2026-09-22 21:30Z and 17:49Z, anda6971f09d7d241f9at 2026-09-22 17:49Z. None has been modified since the roll.48550a13a2f4395bwas created at 18:14:44Z withlastSeen15:15:11Z. So the pipeline has read at least 05:24Z to ~15:15Z on the fixed image.🚨 This absence is weak evidence. The line only fires when a Feedback instance with the bare type is opened, so a quiet window says nothing unless someone opened one.
Separate, and not this defect:
/healthon the 9218 replica reports underLIVE RECORD CENSUSthat 2 NodeType records inFeedback/…were re-keyed tos9e58afeafter it booted. That is the 9260 pod of the roll that is currently in progress and not converging (1/2 ready). Until the fleet is on one image, Feedback pages on the 9218 replica render area-not-found. That is a roll artefact, not the overlay described here.Still open, and still the owner's to do (unchanged): a full-node
updateofrbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubtonodeType: Feedback/Feedback, then recycle that address. My session on the control instance's MCP expired before I could read the instance's currentnodeType, so I have not confirmed whether it has already been repaired.Not closing: closing needs the instance repaired plus a Feedback instance opened on a post-#5308 replica without the
As<NodeTypeDefinition>Error line.rbuergi commentedon Sep 24, 2026
Post-roll reading 2026-09-24 ~06:52Z: the log half is verified clean, and the owner repair is still open
Image. memex serves
3.0.0-ci.9218(core39f49494eb, which contains #530845af88a2a8) on56fbdcd48f-4dpcnand-sz5st, perOps/Status/memex06:41:59Z. It has done so since 2026-09-23 05:15/05:24Z.Readings,
Logsthrough memex-cloud's operator:Ops/logs-memex-20260924-verify-feedback-nt,{namespace="memex"} |~ "(?i)As<NodeTypeDefinition> for Feedback|value is PluginContent", 1440 min: 0 lines. The same query on memex-cloud over 900 min also returned 0 lines.Ops/logs-memex-20260924-verify-meshnodehubfactory,|~ "(?i)MeshNodeHubFactory", same window: ≥200 lines (cut). This fingerprint's category was ingested throughout.So criterion (1), no new
As<NodeTypeDefinition> for Feedbackline after the roll, holds over 24 h. #5008 is closed on that reading.Still open for criterion (2), the owner act. Do a full-node
updateofrbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubtonodeType: Feedback/Feedback, then recycle that address. It needs the control instance. I did not read or change it, so I have not confirmed whether it was already repaired.rbuergi commentedon Sep 25, 2026
Triage re-read 2026-09-25 ~15:30Z. Running builds (
/api/version): memex.systemorph.com =39f49494eb(ci.9218), memex.meshweaver.cloud =140c5fb7ae(ci.9291). Ancestry was checked withgit merge-base --is-ancestoragainstorigin/main. Both portals are still held at the V58 migration: the ci.9332 pods crash-loop onDbVersionGate db_version=57 < expected 58. Nothing will roll until a governed Reconcile ofDeployments/memexruns the migration Job.Status: the log half is verified (24 h clean, see the 09-24 comment). The owner act is still open. I tried to read
rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubon the control instance. The read answeredUnavailable … reached no verdict within 10s, so I could not tell whether it has already been repaired. The remaining act belongs to the owner: a full-nodeupdatetonodeType: Feedback/Feedback, then recycle that address. No code action.rbuergi commentedon Sep 26, 2026
Production verification and record repair 2026-09-26 16:47Z: resolved
The code-side repair (#5308,
45af88a2a8) is an ancestor of the currently serving memex core commit45e7f94e4ec36417ebcbda6125f7061c0a803f10. Earlier post-roll coverage had already verified that its watcher no longer emits the conversion-error loop.I completed the remaining owner repair for the affected record:
rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hubin full and corrected only itsnodeType, from the colliding plugin-root pathFeedbackto the actual declarationFeedback/Feedback.nodeType: Feedback/Feedbackand materializesFeedbackcontent.Detailarea successfully. A fresh, untruncated one-minute Logs window after that activation read 0 lines for both collision/error signatures (As<NodeTypeDefinition> for Feedbackandpath 'Feedback' is occupied …), with 0 landing failures.The single matching entry in the preceding 20-minute window was timestamped 16:45:58Z, before the repair at 16:46:43Z. The malformed record and its recurring overlay/error condition are resolved in production.