Skip to content

A static node must not shadow a durable row at the same path — make the collision loud instead of a 30s install hang #1209

Closed

Description

@rbuergi

What happened

Diagnosed 2026-08-11 while investigating a deterministic install: TimeoutException on the Agent and Skill plugin packages (plugins gate, platform image from core 22e4a0c9a).

A host that calls bare .AddAI() (no serveFromPartition) registers BuiltInAgentProvider/BuiltInSkillProvider as static node providers serving paths Agent and Skill. If that same mesh then installs the durable Agent/Skill plugin packages — which write real rows at exactly those paths — the two collide:

  1. the durable root create is refused NodeAlreadyExists → applied as an update, activating the hub on the STATIC node;
  2. MeshDataSource.WithMeshNodes() takes its static branch (FindServedStaticNode) and seeds the per-node hub with AgentNodeType.CreateMeshNode() — NodeType null, no persistence backing;
  3. the hub emits one Full snapshot at v0 and never again, so anything waiting on the node's stream (here PackageInstaller.RootRetypeReconciled, PackageInstaller.cs:1762-1775) waits out its 30 s timeout.

Core already KNOWS this hazard and documents it in two places — src/MeshWeaver.AI/AgentNodeType.cs ("the static type-def and the durable row BOTH claim @agent, and the static one wins the serve seams … the per-node hub is seeded from a node that is by design NEVER PERSISTED", #902) and src/MeshWeaver.AI/AiContentSources.cs ("Drop Agent and the in-memory provider comes back and SHADOWS the DB nodes the Agent plugin installs"). The cure is per-host configuration (serveFromPartition), which the portal gets right via Helm and which the gate harness did not.

NodeTypeRebindWatcher (added in 72df41936) did not cause this — it stopped masking it: it recycles the hub on a NodeType change, and for Agent it fires twice, the second landing inside the final retype's write window.

Why a config-only cure is not enough

The failure mode is silent and total: the partition root becomes unreadable / un-creatable / un-writable, surfacing only as a 30-second timeout somewhere downstream, with nothing naming the collision. Every host must get serveFromPartition right, forever, with no diagnostic if it doesn't. Known bare-.AddAI() callers today include memex/Memex.Client/MauiProgram.cs:108 and any monolith/deployment whose Features:StaticRepoSync:Partitions is empty (the default — only Helm sets the list).

Proposal

Make the collision loud and early rather than a downstream hang. Options, roughly in order of preference:

  1. MeshDataSource.WithMeshNodes refuses to serve a static node at a path that has a durable row (durable wins), and logs it at Warning naming both claimants.
  2. A startup assertion: a registered IStaticNodeProvider path that also resolves to a persisted partition root fails fast with a message naming the serveFromPartition fix.
  3. At minimum, PackageInstaller's post-install stream waits should name the suspected cause on timeout ("path X is served by a static provider — its hub is not persistence-backed") instead of a bare TimeoutException.

Immediate follow-up (separate, already in flight): the gate harness's own .AddAI() call is fixed so the plugins gate stops failing on Agent/Skill.

Diagnosis evidence: control run against the previous image digest passes; renaming the module root Agent→Agentz (identical content) passes; fixing the harness's AddAI makes all 20 modules pass.

🤖 Generated with Claude Code

Activity

  1. added a commit that references this issue on Aug 12, 2026
  2. rbuergi commented on Aug 27, 2026

    @rbuergi
    ContributorAuthor

    The serve half of this collision is now closed too — see #2517 and its fix in #2534.

    This issue fixed the INSTALL symptom (the deterministic 30 s install: TimeoutException) by making the collision loud: DescribeStaticServeCollision names the contested path, the static claimant, and the serveFromPartition cure. What it left open is the case where nothing is installing — a page simply being served at a contested path.

    That surfaced live on 2026-08-27: curl https://memex.meshweaver.cloud/Agent returned og:title="Agent" with no og:description and no JSON-LD, i.e. the built-in NodeType definition rather than the authored Store package; /Skill the same; and Skill had stopped appearing as a Store/Plugin row at all. The mechanism is exactly the one described here — "the static one wins the serve seams … the per-node hub is seeded from a node that is by design NEVER PERSISTED" — reached from the serve side instead of the install side.

    The trigger was that this issue's cure is per-host configuration, and the configuration stopped arriving. MeshBuilder.Configuration was null on every deployed portal (the portal composes its own module union and calls InstallAssemblies directly, bypassing the one caller that handed configuration over), so serveFromPartition read as absent and every AI partition fell back to in-memory serving. This issue's own "Why a config-only cure is not enough" section called that shape exactly: "Every host must get serveFromPartition right, forever, with no diagnostic if it doesn't."

    #2534 removes the "forever" part: MeshHostApplicationBuilder takes the deployment's configuration at construction, so a host binding a mesh to an IHostApplicationBuilder cannot fail to supply it, and a null is now reported on stderr instead of being silently read as a deliberate in-memory choice.

    Worth recording for anyone who lands here next: the blast radius is the set of built-in NodeTypes whose name matches a package root — Agent, Skill, Harness, Model, Provider, five, all gated on the same switch. Nothing else in the 80-package catalog collides.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions