Repository navigation
fix(bake): an unready replica neither re-keys shared NodeType records by adoption nor deadlocks its own activation sweep - #5643
Conversation
… by adoption nor deadlocks its own activation sweep #5544 — memex's 9260 pod (cbbj8) never became Ready across five restarts: - every boot adopted 168 prebuilt bundles and stamped CompiledFrameworkVersion onto the SHARED records before its sweep began (PrebuiltAssemblySeeder bypassed MeshPublicationGate), so the serving 9218 replicas' census read 21 records re-keyed since boot; - its batched source discovery failed, and the activation fallback's per-type wait (WarmOne) read only the shared record — whose compile stamp this process's own admission HOLDS until the bake passes. Every type timed out at 300 s; 107 x 5 min outlasts the 3 h startup probe. Fixes: the adoption stamp goes through MeshPublicationGate.Offer (new: reports the verdict the publication was decided under; a held adoption still feeds the local adoption ledger); WarmOne merges LocalNodeTypeBuilds (as WatchForRecovery already did) and reads content via ContentAs; NodeTypeBakeGateState.MarkProgress names the step, type and bound the sweep waits on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical publication, compatibility, and recovery-path issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 4
Open (4)
What changed in this PR
Fixes readiness-gated NodeType baking and prevents unready replicas from publishing adoption stamps.
Changes:
- Gates prebuilt adoption through mesh admission.
- Adds local-build awareness and detailed progress reporting.
- Adds integration tests and architecture documentation.
| File | Summary |
|---|---|
test/MeshWeaver.Hosting.Test/AnUnadmittedProcessDoesNotReKeyByAdoptionTest.cs |
Tests gated prebuilt adoption. |
test/MeshWeaver.Hosting.Test/AGatedSweepHearsItsOwnCompileTest.cs |
Tests local compile detection. |
src/MeshWeaver.Mesh.Contract/Services/MeshPublicationGate.cs |
Adds admission-aware publication offers. |
src/MeshWeaver.Hosting/NodeTypeBakeGate.cs |
Adds dynamic progress details. |
src/MeshWeaver.Hosting/DynamicTypePreWarmerHostedService.cs |
Wires progress reporting. |
src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs |
Uses local build signals and content conversion. |
src/MeshWeaver.Hosting/BuildProtocolDriver.cs |
Reports build coordination progress. |
src/MeshWeaver.Documentation/Data/Architecture/MeshAdmission.md |
Documents admission and sweep behavior. |
src/MeshWeaver.Compiler.Pipeline/PrebuiltAssemblySeeder.cs |
Gates adoption record stamps. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if (admission is MeshAdmission.Provisional) | ||
| Apply(node); |
There was a problem hiding this comment.
Not changed, on purpose. It is the same lifecycle the two held COMPILE stamps have had since #3478, and it heals the same way. Ledger: NodeTypeAdoptionRegistry is only read by OverlayOnto when the sweep classifies, and that sweep has already run by the time a refusal and then a retraction can happen, so a stale ledger entry cannot misclassify anything later. Record: if the adoption was discarded and the process is re-admitted, the shared record still names the serving replicas' framework, so it reads as Foreign to this process (NodeTypeBuildIdentity.RefusalReason). The next activation of that type here takes DecideFrameworkStale -> Recompile (the process is not leaving, so there is no #5353 yield). That compile's stamp goes through the now-admitted gate and lands. Re-offering discarded publications on re-admission would give a refused image a replay path for everything it produced while refused. Discarding them is the property #3478 asks for.
| Func<IObservable<PreWarmOutcome>> bake, | ||
| ILogger? logger) | ||
| ILogger? logger, | ||
| Action<string>? progress = null) |
There was a problem hiding this comment.
Kept as an optional parameter on purpose. The only consumer outside core is MeshWeaver.Plugins, and it compiles against a pinned core source (MW_PLATFORM_REF), not against a shipped binary, so there is no already-compiled caller to break. I tried the overload route first (in this branch) and it breaks SOURCE instead: every existing cref to Run / WarmDynamicTypes turns into CS0419 under -warnaserror (6 sites in core alone), and Plugins' DynamicTypePreWarmerTest.cs:108 has and :171 does typeof(DynamicTypePreWarmer).GetMethod(nameof(WarmDynamicTypes)), which throws AmbiguousMatchException once a second overload exists. Every current call site (BuildCoordinationTest.cs:128/256/304, DynamicTypePreWarmerTest.cs:71/235) uses positional or named arguments that still bind to the optional-parameter shape.
| bool batchBake = false, | ||
| bool buildProtocol = false) | ||
| bool buildProtocol = false, | ||
| Action<string>? progress = null) |
There was a problem hiding this comment.
Same answer as on BuildProtocolDriver.Run: the only external consumer (MeshWeaver.Plugins) compiles from pinned core source, so binary compatibility buys nothing. A second overload would break its source: DynamicTypePreWarmerTest.cs:108's cref goes CS0419 and :171's GetMethod(nameof(WarmDynamicTypes)) throws AmbiguousMatchException. Its existing calls (:71, :235, BuildCoordinationTest.cs:128) still bind to the optional parameter unchanged.
| lock (verdict) | ||
| { | ||
| if (disposed) | ||
| return Observable.Return(Unit.Default); | ||
| return notRun(MeshAdmission.Refused); | ||
| held = held.Enqueue(new HeldPublication(subject, publication)); | ||
| heldCount++; | ||
| } |
There was a problem hiding this comment.
Fixed in dd8099f: after the enqueue (outside the lock) Route re-reads Admission and calls Reconsider() when it is no longer Provisional. Any verdict turn is now either seen by that re-read or drains a queue that already contains the publication, so no window is left.
… Reconsider that drained first cannot strand it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nd-written 30 s (TestTimeoutLiteralRatchetGuard) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Refs #5544 (sev:H — closes on post-roll verification, not on merge).
What the 9260 pod on memex actually did (read from its own log,
Logsactions on memex.meshweaver.cloud)Ops/logs-memex-20260924-cbbj8-bake-stall→ 200 lines ofmemex-portal-deployment-7596dd76bb-cbbj8, boot at 03:55Z:ShippedPrebuiltBundles: 168 prebuilt assembly(ies) from 65 shipped bundle(s)…— preceded by 168 ×Prebuilt assembly ADOPTED for … (framework s9e58a…). Each one wroteCompiledFrameworkVersion+ assembly coordinates into the SHARED record while the pod was Provisional. The serving 9218 replicas'/healthLIVE RECORD CENSUS (Ops/Status/memex, 06:36Z) reads "21 NodeType record(s) were RE-KEYED to a framework this replica does not run AFTER it booted" — alls9e58afe.BuildProtocol: claim granted→107 of 232 … need building→batched source discovery did not establish the source sets — … falling back to the activation-driven sweep.→ TimedOut — 300.0 sfor EVERY type, sequentially (binary-clicker-space, BinaryClicker, BinaryClickerV2, Crm/, Edu/, Hosting/, LinkedIn/, RFPProcessor/, rsalzmann/…). 107 × 300 s ≫ the 3 h startup probe → killed → same boot again. Readiness saidenumerating dynamic NodeTypesthe whole time, because that label was set once and never changed.So the enumeration never stalled — it is a sweep that cannot succeed on a gated pod, plus an ungated write path.
Root causes and fixes
(b) Adoption bypassed the join point. #3478 gated the two COMPILE stamps;
PrebuiltAssemblySeeder's adoption stamp carries the same fields and was never offered toMeshPublicationGate. It now is, via a newMeshPublicationGate.Offerthat reports the verdict the publication was decided under (no second read ofAdmission). Held ⇒ the adoption still counts for THIS process (bytes on the store, definition inNodeTypeAdoptionRegistry, so the sweep classifies it Baked as before) and the shared stamp lands on admission; Refused ⇒ not adopted, nothing written. This is the inverse of #5353's bind-path yield (leaving process re-keying backwards vs. not-yet-joined process re-keying forwards).(a) The activation sweep deadlocked against its own admission.
WarmOnewaited only for the shared record to show a usable build — whose stamp this process's gate holds until the bake passes. It now mergesLocalNodeTypeBuilds(the process-local witnessWatchForRecoveryalready uses since #3478) and reads content viaContentAsinstead ofContent is NodeTypeDefinition.NodeTypeBakeGateState.MarkProgressmakes the readiness detail name the step, the type and the bound (static import, bundle adoption, enumeration count, claim election, following another replica's GO — the one unbounded wait, batched discovery,building k of n … waiting on <type>, for up to <budget>).Tests (real monolith mesh, negative controls run on the unfixed source)
AGatedSweepHearsItsOwnCompileTest— provisional: pre-fixTimedOutat exactly the 60 s budget; post-fixCompiledin seconds while the shared record provably does NOT carry the build (stamp held). Unarmed control compiles from the record either way.AnUnadmittedProcessDoesNotReKeyByAdoptionTest— provisional: pre-fixHeldCount == 0(stamp written immediately); post-fix held, ledger carries it, record untouched for 3 s, lands onMarkComplete. Refused: pre-fix adopted=true; post-fix nothing adopted/written. Unarmed control stamps at once.Not fixed here / not established
System.TimeoutExceptions.Pairs-with: none — adds
MeshPublicationGate.OfferandNodeTypeBakeGateState.MarkProgress; removes nothing public.🤖 Generated with Claude Code