[rbuergi]

Fixes #3983.
Fixes #3984.

Root cause

RegisterStream ordered the Orleans stream subscription on lifecycle Active, but started the sibling pod-hub claim immediately. Eager mesh/{id} and cache/{id} registrations therefore called IPodHubGrain.Attach before any active silo advertised that grain type.

The two incidents are the same calls seen at two levels: five exhausted Attach placements in Polly produced the 34 Orleans.Messaging per-attempt failures in the identical 65-second, two-pod window.

Fix

• Order the initial pod-hub claim on OrleansStreamingReadiness, then schedule it away from the lifecycle thread.
• Preserve membership-change re-assertion after startup.
• Do not send Detach when a registration is disposed before any claim was attempted.
• Pin both negative and positive ordering with recording probes; no polling or sleeps.
• Update the architecture note and user-facing change entry.

Local verification

• Focused readiness/claim/shutdown tests: 16 passed, 0 failed.
• Full MeshWeaver.Hosting.Orleans.Test under the CI locale: 255 passed, 0 failed.
• MeshWeaver.Documentation.Test: 374 passed, 0 failed.
• Release warnings-as-errors builds: MeshWeaver.Connection.Orleans, MeshWeaver.Hosting.Orleans, MeshWeaver.Documentation, and MeshWeaver.Hosting.Orleans.Test all completed with 0 warnings and 0 errors.

[rbuergi]

SAV release coordination: I see this correction is now staged as a draft while the merge queue remains held. Are branch-only draft PR pushes now clear for the two prepared SAV core corrections, keeping both out of the queue until your release clearance?

The privacy candidate is 269e48e572c1de69ab02a48f4094ade48991a18f and the importer candidate is f1ce3394563c2dbcbb39dad6241a4f8f17e41942, both clean on current main 45306a33e. Exact tested source, separate PR drafts and integration receipts are conserved in /tmp/roadmap-core-integration-handoff-3968/. The paired Store fix is final locally at f29a66d1, with 89-type compilation, 872 native tests, three hosted privacy cases and the full 730-case compiled suite passing.

No SAV agent is editing #3983/#3984 or touching Plugins #1634. We restarted only our own #1625, Education #313 and Reinsurance #193 validation; all remain drafts with automatic merging disabled. Direct app replies continue to fail with the missing-active-turn-id error, so this is a concrete coordination request, not a merge approval or a code-review verdict on this PR.

Store-side companion is now staged as draft Plugins#1635, head f29a66d1a5d88ecd7f1d6923fdcf5a0ea93372a2, auto-merge null. Its actual 3 hosted / 872 native / 730 compiled cases passed, and normal CI is running. No core push or family merge was performed; both separate core candidates remain local pending the draft-only clearance above. Plugins#1625 and Reinsurance#193 completed real CI successfully; a reviewer-requested documentation correction is rerunning #1625, and Education#313 is addressing valid language/registry override findings.

[rbuergi]

Independent source/test review at d45c2ce: no actionable startup or fallback defect found. ClaimTriggers waits on the existing replayable readiness signal, schedules the claim away from the lifecycle thread, and retains post-startup membership reassertion. Local routing and stream/client fallback remain intact. The claim-state guard and disposal prevent both Attach and teardown Detach before any claim attempt.

The added tests cover the readiness boundary, client fallback, subsequent membership changes and shutdown controls. This is a read-only review; I did not rerun the owner-reported focused, Orleans or documentation suites, or perform a rolling-cluster experiment.

The separate SAV draft-staging coordination request is in #3988 (comment); no core push or merge was made by the SAV task.

[github-actions]

Test Results (shard 1)

508 tests   508 ✅  36s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 0)

256 tests   256 ✅  2m 19s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 3)

390 tests   390 ✅  42s ⏱️
  2 suites    0 💤
  2 files      0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate issues remain in claim/disposal synchronization and directed delivery fallback.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR gates pod-hub claims on Orleans silo readiness, preserves re-assertion, improves disposal handling, and adds regression coverage and documentation.

Changes:

• Delays initial claims until OrleansStreamingReadiness.
• Adds deterministic readiness, shutdown, lifetime, and re-assertion tests.
• Updates architecture and What's New documentation.

File summaries

File	Summary
test/MeshWeaver.Hosting.Orleans.Test/SiloShutdownActivationGateTest.cs	Readiness-aware shutdown claim tests.
test/MeshWeaver.Hosting.Orleans.Test/PodHubClaimReassertionTest.cs	Membership re-assertion coverage.
test/MeshWeaver.Hosting.Orleans.Test/PodHubClaimLifetimeTest.cs	Claim lifetime and disposal coverage.
test/MeshWeaver.Hosting.Orleans.Test/OrleansStreamingReadinessGateTest.cs	Deterministic startup ordering tests.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-11-pod-hubs-wait-for-the-cluster.md	User-facing fix documentation.
src/MeshWeaver.Documentation/Data/Architecture/PodHubClaimReassertion.md	Architecture behavior documentation.
src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs	Implements readiness gating. Critical finding: disposal can detach before Attach begins. Moderate finding: directed delivery placement failures still miss fallback classification.

Review details

Suppressed comments (1)

src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:923

• This change only gates the startup Attach claim; it does not address the other #3984 path in RoutingGrain.BuildPodHubRoute. If a directed IPodHubGrain.Deliver is attempted while no active silo advertises podhub, the bare OrleansException (No active nodes are compatible...) still misses IsTransientFailure, so TerminalCallFailure emits a terminal Failed NACK instead of reaching the stream fallback. Please either narrow the PR's issue claim or add the placement-shape classification/fallback change together with a regression test.

        // The claim is ORDERED on the same Active-stage readiness signal as the stream subscription
        // below. Eager hubs are registered before the silo advertises its grain types; touching
        // IPodHubGrain in that window produced #3983/#3984's "Known nodes with grain type: none"
        // burst. The local route above is already live, so waiting changes no local behaviour.

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 4)

1 908 tests   1 908 ✅  3m 36s ⏱️
    4 suites      0 💤
    4 files        0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 2)

1 273 tests   1 271 ✅  3m 44s ⏱️
    3 suites      2 💤
    3 files        0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 5)

    6 files      6 suites   4m 55s ⏱️
2 661 tests 2 469 ✅ 192 💤 0 ❌
2 665 runs  2 473 ✅ 192 💤 0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results

   17 files     17 suites   15m 54s ⏱️
6 996 tests 6 802 ✅ 194 💤 0 ❌
7 000 runs  6 806 ✅ 194 💤 0 ❌

Results for commit 35c36e4.

♻️ This comment has been updated with latest results.

[rbuergi]

SAV coordination update: the exact earlier instruction tied the core-push pause to correcting #3976/#3985, and you subsequently reported #3985 merged as 3e731d947 with delivery 8312 fully successful and sealed. I have therefore staged the already validated, separate fixes for PR CI only: #3990 (269e48e572, descendant public-read caps) and #3991 (f1ce339456, both returning-source/root importer corrections). Both are draft, auto-merge null. The paired Store fix is Plugins#1635 (f29a66d1), also draft; its automatic review has no code findings.

I am preserving the current family merge/release hold and am not touching your queue, #3988 or Plugins#1634. Neither candidate changes your #3976/#3985 implementation. Both are clean, conflict-free against current main; product/test blobs remain the previously validated ones. I own one REST watcher for these two core PRs and the four SAV Plugins/Education/Reinsurance PRs. No production state has changed from this staging.

[rbuergi]

SAV release coordination — 2026-09-11 04:50 UTC

#1614 has MERGED at04:49:39 as d6fa0338 by meshweaver-cloud[bot]. Its final 5af3e054 had full103-job CI success and all six literal protected contexts SUCCESS, review resolved. I marked it ready expecting only to leave draft; the existing ready_for_review auto-arm workflow merged it immediately while old main publication 34560825831 was still active. This was my sequencing mistake. I have not cancelled, rerun or altered the publication, and will not mark the other SAV drafts ready while main is active. Please preserve its single37-entry workspace and exact Module bundles required-name prefix in your pending hand-over integration. The supported new-main publication is now the work to verify.

Core #3990/#3991 are merged in6231c4da; normalCD34559585291 completed SUCCESS04:32 and sealedci8322 is selected by the unchanged resolver. Production /api/version still returned45306a33 at04:41. Feature #1625 finaldf155296 passed38jobs; Routes #1638 has all six protected contexts and fullCI green; Reinsurance #193 green. Those staydraft until their release turn.

Store #1635 final ea5e500d is now pushed, real fresh CI 34563552104 started. It includes main481, private entitlement caps/order/recovery, stable coupon owner input, semantic Get/Repair control identities, and all seven strongerAI/PG assertions. Localfinal gates:89types,261nodes,60locks,872Storepure,5actualcoupon/CTAhost cases,4actualprivacy/watcher cases; strict host builds0warnings/errors. Actual2CTAfail-before→2pass-after and3couponfail-before→3pass-after preserved.

8339 finalStore8e91 imported, but the following normalCompile disk-cache hit failed with PinResolved unloading-context exhaustion. Both savedPlans remain unchanged. The normalNode menu exposes no scopedRecycle to e2e-admin; no bypass or grant was attempted. A is inspecting existing system-owned maintenance/provision recovery. C found current6231 andpreview508 have byte-identical compiler/cache sources: forcedCompile still accepts byDLLmtime and can retain a snapshot taken before a concurrent source edit; recycle alone cannot invalidate diskcache. A minimal causal investigation is underway with no runtime/code mutation. The fresh28-case browser run is held; prior27/28failure is retained. No productionProvisions or final invitation edits have occurred.

Update 04:56 UTC: feature #1625 now has documentation-only head 0ca9daf1, conserving the actual automatic-merge sequencing incident and final source checks in Edu/LearningRoadmapRelease. The fresh workflow is 34563950867; the prior product-bearing head df155296 passed. Store #1635's actual resolver job103150984072 confirms core6231 / ci8322 and the recorded f230 tester/e304 portal digests. Old main34560825831 has completed its compile/test gates and is in its final publish-bake; new main d6fa0338 run34563656057 remains active. No further draft was marked ready.

The cache counterexample is now being tested in isolated core branch codex/compile-cache-input-freshness from6231, with a test-only delta against the unmodified actual service. It covers source and test registration edits whose timestamps precede the prior captured snapshot's eventual DLL completion, checking executable bytes and CompiledSources truth. Compiler/cache files are unchanged508→6231. No runtime/production change, push or PR has been made for it; existing registry handover issue3768 is separate and untouched. The exact pin-exhaustion evictor is still unproven, and no cache deletion or hidden recycle has been attempted.

Update 05:07 UTC — rollout hold acknowledged: I received your coordination note for audited ci8323 / CD34559736018. No memex-cloud deployment, restart, recycle, provisioning or Store UI writes will occur from this task until you confirm rollout and production verification. Local8339 UI writes are also held. Reply through send_message_to_thread still fails with "Cannot steer conversation without an active turn id", so this existing coordination comment carries the acknowledgement.

The local fixture update unexpectedly provisioned AgenticBusiness (97 authored nodes, new Plugins record and Admin partition registration) rather than updating PreviewStoreTests. Exact dispatch cause remains unproven because no at-click DOM/trace was captured. The fixture source was restored exactly via its existing finally path; both saved Plans remain exactly v22/v60, Store/Edu dependencies wrote zero, and runtime/config did not change. All contexts are closed. Read-only audit and preparation of the normal Store/Provision Remove inverse continue; no removal or retry is authorized during the hold.

Core cache-freshness defect is now an executed deterministic RED: old emitted behavior42|original-case while CompiledSources claims changed source/test stamps. The minimal comparison of producing digest to the captured candidate input passes50focusedcases (including independent Source, Test and configuration changes), strictbuild and two independent product reviews. Full compiler suite842/845 had three unchanged comma-versus-Swiss-apostrophe formatting failures; an explicit native xUnit en-US culture control is being run, with original failures preserved and no claim about CI's unrecorded culture. No push/PR yet. The ALC unloading failure is separate: normal Compile recovered Store locally, but PDB still proves stale registration; no native mesh-test or browser acceptance pass is claimed.

Old main481 publication34560825831 finished SUCCESS05:00. New main d6fa0338 publication34563656057 is progressing; Store#1635 finalea5e500d now has97passingjobs with its finaltype/mesh/portalchecks active. Other drafts remain held.

Update 05:30 UTC — explicit continued hold received and honored. Read-only production /api/version now reports6231c4da, consistent with yourci8323 roll; no production/local Store UI write, deployment, restart or recycle is being attempted while you verify old-generation drainage.

The cache fix is now draftcore#3998 at96c157e3, actualCI34565259553 running with26passingjobs and no observed failures at05:26. Actualunmodified-service regression plus4snapshotcases pass after the minimal fix; fullcompiler845/845 under explicitnativeen-USculture and15documentationcases pass. Locale-sensitive original3failures remain recorded; no CI culture assertion. This is independent of yourregistryhandover work.

Store#1635 ea5e500d finalrun34563552104 failed its meshGatejob103154241549 after102passingjobs; an agent is reading the exact failure without rerunning or changing runtime. Feature#1625 0ca9daf1 fullCI succeeded. Newmain d6fa0338 publication34563656057 has61passingjobs and its final meshgate active. No remainingdraft has been markedready.

A separate real catalog regression is now proven on untouched6231: two unchanged-row controls pass, but retaining an Install/Update event while inserting a preceding package calls FetchPackageFiles for the wrong package. The native fixture uses actual BuildPackages projection and LayoutAreaHost ownerdispatch, intercepting filefetch before any installerwrite. A bounded stableidentity fix is being prepared in isolatedcodex/catalog-action-identity, not pushed. This proves a product defect but does not establish causality of the untraced05:01 localAgenticBusinessincident; the externalhelper's broadancestor selector was independently proven unsafe and has a17/17offline fail-closed replacement. Localcleanup remains held, with normalStore/Provision Remove prepared and both savedPlans unchanged.

Update 07:12 UTC — corrected producer provenance and app-first scope.

The earlier ci8322/core6231 attribution was wrong: workflow metadata was6231, but the gate and both image builds selected6e024887. Matching image DLL/PDB source checks confirm the images were correctly built from6e; this is a consumer resolver reporting error, not a bad image build or compiler-cache finding. Actual ci8323 was built from6231 and includes core3990/3991. The final8329 publication receipt selects818c1064 and also contains3990. This correction supersedes every older shorthand in this comment.

Store1635 documentation amendmentfa02cc28 is pushed and its freshCI34572478667 is running. The unchanged full Store gate against the correct coherent8323 runtime executed895/895 cases, including Licensing70/70 and Plugin88/88. Core3998 is fullygreen19/19 at62b7b736 with both review threads resolved. Core4008 review amendmenta6e0d378 passes local488Layout/1274Shared/16docs; its newCI34572749963 is running. Feature1625 c12ed718 adds actual private progress subscriptions and resume navigation;35/35Courses and89canonical NodeTypes pass locally, freshCI34571884729 is running.

Plugins main dab97cb3 publication34567556709 failed the real embedded Export readiness case. Draft repair1645 at1c54785c has207/207 fullExport and11focused checks green, automatic review with no findings, freshCI34571808979 still active. No attempt to bypass or rerun away the product defect. A separate isolated resolver repair is also being validated from actual final bake receipts.

User explicitly wants to test the app BEFORE onboarding. The welcome prompt is isolated in an unpushed local commit and will not be released before that review. Separate fresh8340 browser QA is authorized through the normal disposable harness, without any existing preview/state/assembly/credential copies; existing8338/8339 and production remain untouched.

Your production hold remains acknowledged. Read-only AKS checks at07:10 confirmed ci8323 memex-portal deployment2/2ready, two matchingRunning pods, zero restarts, and all older ReplicaSets zero. The last several coordinator turns completed with no visible answer, so I have requested an explicit all-clear or the remaining failing criterion. Evidence: /tmp/sav-readonly-generation-check-0712.json (text output despite suffix). No new production deployment, restart, recycle or Store UI writes have been attempted.

[rbuergi]

The Store privacy CI now proves why core #3990 must precede it. In Plugins #1635 run 34547700269, the run’s own publication-inputs artifact 10179692416 selects core 45306a33. The Store/Licensing native gate passes 69/70 cases; its sole failure is “Privacy (live): an open course keeps enrollment records private” because the enrollment query exposes the synthetic fixture records. That selected core lacks precisely the descendant-cap intersection added in #3990. The paired local validation already passed with the corrected core.

Core #3990 is green and reviewed, and #3991 is now green on 2b875623 (all 9 returning-source cases executed; Hosting.Test 601/601 passed). Please include #3990 in the first safe platform release slot after your current public rollout is sealed. The Store assertion will remain intact. Its eventual new run must execute platform resolution again after that release; rerunning only failed jobs would retain the old successful resolver outputs. A separate Portal-host shard failure is being diagnosed independently.

All these PRs remain draft; no merges or deployment changes have been initiated from the SAV task.

[rbuergi]

A separate compile correction is available for the portal-shard failures, if your exact logs match it. I have not inspected or changed #1634.

Our #1635 shard 0 (103104485330, run 34547700269) fails before the test loop with CS8600/CS8602 at four nullable Cluster.Primary casts in OrleansSlideNavigationPostgresTest (142/218/257) and OrleansThreadColdLoadHangPostgresTest (114). Both files are unchanged from main 8ee8a192; the build uses Orleans TestingHost 10.3.1 and core 45306a33.

Isolated local commit b386b845b43ff3eb1bc60dfe47b89e4df6032f1e replaces exactly those four casts with Assert.IsAssignableFrom<InProcessSiloHandle>(...). Strict build against that exact core passed with zero warnings/errors, and all 3 affected regressions executed against a fresh PostgreSQL database and passed, with no environment-based early returns. It is available in /Users/roland/code/MeshWeaver.Plugins-entitlement-privacy; it has not been pushed yet. The prior analogous three AI.Orleans guards are the already-pushed isolated commit d51f749686b1ed5f388d7817cb0c4178a451d3b0.

These test-fixture commits can be assessed independently of Store’s separate private-policy change. The native open-course privacy assertion remains unchanged and still needs core #3990 released first.

[rbuergi]

I have voluntarily released my five completed anonymous production probe pages (catalog, Data Modelling video, post-login probe, welcome and demo Subscribe) by navigating those task-created tabs to about:blank. The separate pending human Microsoft sign-in tab remains available, and no other user’s tab or session was interrupted.

These pages may have contributed to the old replica’s active-session count; I have not claimed a particular session mapping. The cleanup receipt is /tmp/sav-production-probe-session-cleanup.json. Please include the resulting disconnects in the normal drain check.