A deploy makes custom-NodeType per-node hubs answer Not found instead of waiting — in-flight writes are silently discarded #667

Repo: Systemorph/MeshWeaver (per-node hub activation / NodeType assembly reload)
Environment: https://memex.meshweaver.cloud, observed 2026-07-27 07:40–08:20 UTC
Severity: every rolling deploy is a window in which reads lie and in-flight writes are lost

Summary

While a NodeType's assembly is being reloaded — which happens on every deploy, because each NodeType recompiles against the new framework build the first time it is touched — reads of nodes of that type return Not found rather than waiting for activation or surfacing the activation failure. A transient recycle is therefore indistinguishable from genuine absence.

That false negative is not cosmetic. Existence checks built on it take the wrong branch, and any write that takes the CreateOrUpdateNode update path fails with no initial state arrived … within 30s, discarding the caller's whole operation. A user lost a course install to exactly this on 2026-07-27.

Timeline (all UTC, 2026-07-27)

The user's install ran inside the deploy window. Everything below is a consequence of nodes being read while their NodeType assembly was being replaced.

Evidence — all measured directly

• A read flipped to Not found across a recompile. get @Edu/Exercise returned the full node; the identical path with the identical identity returned Not found roughly two minutes later, spanning the 08:16:23 recompile. Nothing was deleted in between.
• Custom-NodeType nodes ghost; core-type nodes do not. get @AgenticEngineering/MyExercises (nodeType: Edu/CourseInvite) → Not found, while the same node is returned by the AgenticEngineering/* child listing, and every sibling Markdown node in the same partition under the same gating reads fine.
• Area renders time out. get @AgenticEngineering/area/Subscribe → Delivery to 'AgenticEngineering' failed: No response received in hub cache/vqYmlVnVS0KqrWZufo2T_A within 00:01:00 for request SubscribeRequest → target AgenticEngineering. get @AgenticEngineering/area/Install → timed out.
• The query index and the authoritative read disagree. sglauser/AgenticEngineering is present in the index (version 13, lastModified 2026-07-27 07:49:00) but get @sglauser/AgenticEngineering returns Not found.
• compilationStatus is not reliably terminal. Store/Install sat at Compiling from 08:18:49 with its last success at 07:53:12. On a local dev portal, Latex/Document has sat at Pending since 2026-07-21 while serving normally. A caller cannot use this field to tell whether a type is usable.

Why this matters beyond flaky reads

Store/Plugin/Source/Localizer already carries a defensive workaround for exactly this failure mode:

/// The node type of a viewer's localized-course home root — a CORE type (Markdown) on
/// purpose: its hub is always ready, so the one node the viewer navigates to can never ghost.
public const string HomeRootNodeType = "Markdown";

Pinning one node to a core type is not a fix — it only protects the node whose type the author controls. The install still walked into the same wall on an Edu/Exercise node it does not own:

Update aborted: no initial state arrived for
'sglauser/AgenticEngineering/Introduction/Exercise/CharacterizeThenTame' within 30s.

and lost the entire run (see the companion Store/Plugin issue). A hub reporting Not found for a node that exists also silently corrupts every existence check built on it — EnsureHomeRoot skipping a create it should have made, MissingAfterInstall mis-verifying, Entitlements.IsEntitled reading a transient false and bouncing a paying user to the paywall.

Ruled out — please do not re-derive this

The differing compiledFrameworkVersion values across NodeTypes look like two portal builds serving at once. They are not. compiledFrameworkVersion records the framework build at the moment each NodeType was last compiled, so a fleet on a single build legitimately shows a spread of values.

Verified on a single-instance local portal (https://memex.localhost:8443), one process, one build:

Same-build compiles share a hash; the type last compiled three days earlier carries the previous one. The prod split (Store/Install 1e69fc09… at 07:53:12 vs Store/Plugin 26336dc0… at 08:01:06) is explained entirely by the CD run that succeeded at 07:58:08 between them. There is no mixed fleet and no recompile ping-pong; recompiling after a deploy is the designed behaviour. The bug is what happens to readers while it does.

Suggested fix

1. A per-node hub mid-activation must never resolve a read to Not found. Reporting absence for a node that exists is what turns a routine deploy into a data bug. The read should await initial state (bounded) or surface the activation failure explicitly — a false negative is the one answer it must not give.
2. Make assembly reload non-destructive to in-flight callers. A recompile triggered by a deploy should not tear down activations that are servicing requests; drain, or serve the previous assembly until the new one is ready.
3. Do not raise the 30 s guard in CreateOrUpdateNode. The guard is correctly reporting that the hub never came up; increasing it only makes the failure slower and widens the window in which a caller's work is lost.
4. Consider whether compilationStatus should be a reliable readiness signal — today Compiling and Pending both persist long after the type is (or is not) usable.

How to reproduce

All steps are read-only.

A. The ghosting (needs a deploy or a forced recompile in flight)

1. Pick a node of a custom NodeType, e.g. AgenticEngineering/MyExercises (Edu/CourseInvite).
2. get @{that path} → returns the node.
3. Trigger a recompile of its NodeType (a deploy does this automatically; otherwise force a release on the NodeType node).
4. Re-run the identical get while compilationStatus is Compiling.
   Expected: the node, or an explicit activation error.
   Actual: Not found.

B. Custom types ghost, core types do not

5. get @AgenticEngineering/* — the listing includes AgenticEngineering/MyExercises (Edu/CourseInvite).
6. get @AgenticEngineering/MyExercises → Not found.
7. get @AgenticEngineering/Introduction (Markdown, same partition, same gating) → returns fine.

C. Area renders on an affected hub

8. get @AgenticEngineering/area/Subscribe
   Actual: No response received in hub cache/… within 00:01:00 for request SubscribeRequest.
9. get @AgenticEngineering/area/Install → times out.

D. Index vs authoritative disagreement

10. search path:{viewer}/AgenticEngineering → returns the node (version 13).
    get @{viewer}/AgenticEngineering → Not found.

E. The downstream data loss

11. During the same window, trigger a CreateOrUpdateNode that takes the update path against a custom-NodeType node — e.g. re-run a Store/Plugin install (/{plugin}/Install → Get) over an already-copied Edu/Exercise subtree.
    Actual: Update aborted: no initial state arrived for '…/Exercise/CharacterizeThenTame' within 30s, and the caller's whole operation is discarded.

F. Confirm the framework-hash spread is a red herring (see "Ruled out")

12. On any single-instance portal, get several NodeTypes and compare compiledFrameworkVersion against lastCompileSucceededAt. Types compiled against the same build share a hash; types last compiled before the most recent deploy carry the older one.
13. Correlate a prod hash change with gh run list --repo Systemorph/MeshWeaver — a successful Continuous Delivery (main) run sits between the two compile times.