A draining silo keeps running the NodeType adoption sweep and clears shared coordinates for the whole 30-min grace period (Underwriting nav ≥120 s per roll) #3129

A draining silo keeps running the NodeType adoption sweep and clobbers shared coordinates for the whole grace period

Reported as: memex.systemorph.com, navigating Underwriting/ → a submission "extremely slow" (2026-09-02 ~15:00Z). Measured 14:57–15:10Z in Loki; the cost is NOT in the Underwriting content.

What happened

The 14:31Z rollout (RS 67b654495f, memex-portal-ai:3.0.0-rc9.ci.7604) left the old pod 58c88885f8-lhgp7 terminating for 27 minutes (terminationGracePeriodSeconds=1800), still holding circuits: 11.8 GB, .NET Runtime Platform stalled GC pauses of 3.3–5.0 s every ~4 s (310 stalls in 25 min), readiness /alive deadline-exceeded (events 14:58Z).

While draining it kept running the prebuilt-bundle adoption sweep against the SHARED NodeType nodes:

ADOPTION REFUSED for Underwriting/Workbench (#2813): bundle fingerprint aa0e63c8… vs live 572495ee…
→ coordinates cleared, Roslyn compile dispatched
→ LATE_NACK_TERMINAL code=Unknown TimeoutException (+10 s)   [UpdateQueue] FAILED path=Underwriting/Workbench elapsedMs=12344
→ ShippedPrebuiltBundles: seeding … did not complete — the sweep compiles it instead
→ next access repeats (Workbench/Guideline every 30–90 s; Crm/Client 388× in 30 min)

All 1424 ADOPTION REFUSED and 350 [UpdateQueue] FAILED lines of the window were on the dying pod; the two new pods logged 0 of either and 0 GC stalls.

Cross-generation effect — this IS the reported navigation: because the old pod kept clearing Underwriting/Workbench's coordinates, the NEW pods logged Overlay self-heal: instance 'Underwriting/Desk' still stuck on NodeType 'Underwriting/Workbench' after 120s (14:56:44; same for Underwriting/Rulebook/Guideline at 14:57:03) → self-heal compile 14:58:26 → Release 14:58:33. A ≥120 s hang per Underwriting type, per roll (three rolls today).

Root cause (framework)

A hub that is shutting down still runs the NodeType adoption/compile sweep and still WRITES coordinates on shared nodes. #3109 stopped HandleInitialize from running BuildupActions when IsShuttingDown; the adoption sweep and the coordinate-clear on ADOPTION REFUSED are the same class of side effect and need the same guard: a draining silo must not mutate shared NodeType state, and must not dispatch compiles whose result write will be NACKed by its own flush bound (#3112, fixed by #3115 — not in ci.7604).

The 1800 s grace period (Memex chart) is the amplifier, not the cause; cutting it alone would be a band-aid.

Also seen in the same window (separate items)

• The new image adopted 37 bundles and none of the Reinsurance family (Underwriting.zip, Crm.zip; the previous generation had both) — the Reinsurance release-follow bake failed at 14:28Z on registry answered 503 from memex.meshweaver.cloud (degraded replicas, RCA 2026-08-25: sync/bake recompile waves fatten type-hosting silos to 20 GB — superseded builds stay rooted behind the Recycle banner, and a degraded pod never leaves rotation #2194) and was re-run at 15:13Z. Until it lands, all 12 Underwriting types compile from source on first access after every roll (7–10 s each).
• Underwriting/_GitSync last synced 09-01 01:17Z; Reinsurance main moved 09-02 08:56Z. Filed on MeshWeaver.Plugins.
• [CrossSchema] SLOW ≈30/min/pod, 1.0–2.7 s each over 143 schemas (AccessAssignment path:- scope:Subtree 362, Notification 358, Thread 183 per 25 min) — Every page render fires unanchored cross-schema queries (access, notifications, threads, mesh_nodes) — ~2 s each over 186 partition schemas #2640 / RCA 2026-08-25: sync/bake recompile waves fatten type-hosting silos to 20 GB — superseded builds stay rooted behind the Recycle banner, and a degraded pod never leaves rotation #2194.

Measured by the fleet sweep session, 2026-09-02. Not yet conserved as a doc page (belongs in NodeTypeCompilation.md / ReadingAWriteVerdict.md — to be done with the fix PR).