Orleans shard-3 tests time out on fresh branches while main is green (PartitionRootActivationTest, OrleansScheduledPostTest)

Three PR runs today, each on a branch cut from main AFTER #2482 (0596387d, whose own Build and Test was green), each touching only workflow YAML / docs / a structural test — none can reach Orleans:

PR	run	failing test
#2493 (pre-merge, after a main merge)	33061664517	`PartitionRootActivationTest.BarePartitionPath_NoMeshNode_RespondsToPing` — "Expected value not to be because the partition-root fallback must synthesize…"
#2496	33064991182	`OrleansScheduledPostTest.ASystemWrittenPost_IsNotArmed_AndSaysWhyOnThePost` — "The operation has timed out"
#2497	33065380857	`PartitionRootActivationTest.BarePartitionPath_NoMeshNode_RespondsToPing`

Both are activation/timing shaped (a 5 s wait for a partition-root fallback; a scheduled-post arm that never observes). Either #2482's cluster change made activation slower under the shard's load, or the tests' waits are too tight for a shared runner. Re-runs are being used to land unrelated PRs, which is exactly the habit a flaky suite trains. Needs a control: run shard 3 alone on main a few times and read the timing of the fallback.

CD failed on main: run 33067007359 for commit 0ad5f922ba7aa9dce0c7165ec513feb7732ccea9 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33068366200 for commit 6b2d834491a0ef3f2408951eada33af42a85ca9d (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33074224467 for commit 8798a6db98b8bfb3d57be4643c2cb9fb6b588bad (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Two more data points, both on ci/reconcile-recovers-release-version (YAML/doc/test-only diffs, branch refreshed onto current main):

run	failing test
33064991182 attempt 2	`OrleansDelegationStartTest.Delegation_CreateCellsThenThread_ExecutionStartsAndCompletes`, `OrleansThreadAccessTest.ThreadMessageNodes_AreChildrenOfThread`
33072539763	`OrleansCrossSiloCompilationTest.CompiledArtifact_IsVisibleFromBothSilos`

Five runs, five different Orleans shard-3 tests, zero overlap — this is shard-level resource contention (multi-silo clusters on a shared runner), not any single test.

Attempt 3 on run 33072539763: OrleansGrainTeardownStragglerTest.GrainLifetimeCallbacks_AfterActivationIsInvalid_AreGracefulNoOps_NotThrows — a FOURTH distinct test on the same branch. Six data points now, all shard 3, all Orleans, no overlap.

🔁 CD reconcile attempt 1/3 for 7126b08 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 7126b08. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

Two more samples, both on feat/social-leaves-core (#2508 — deletes MeshWeaver.Social + its tests; the diff removes OrleansScheduledPostTest from the suite and cannot reach either failing test): run 33083759894 attempt 1 → OrleansGrainTeardownStragglerTest.GrainLifetimeCallbacks_AfterActivationIsInvalid_AreGracefulNoOps_NotThrows; attempt 2 → PartitionRootActivationTest.BarePartitionPath_NoMeshNode_RespondsToPing. Both 3/3 green in isolation locally; full local suite run with DOTNET_PROCESSOR_COUNT=4 in flight. Same shard-3 shape as every sample above.

✅ Healed 7126b08 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc8.ci.6124), verified against ACR. Installs pick it up on their next 6 h self-update poll. Close this issue if nothing else is outstanding.

CD failed on main: run 33087472581 for commit 46fde128c48de5106c3c9f530c4fc2f9bdbdd767 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Third distinct member of the population, same shape: PR #2524 (run 33106164603 / job 98638473092, branch cut from today's f4b48f5d6, diff touches only FrameworkBuildIdentity + the PluginTester — cannot reach Orleans): OrleansNodeChangePropagationTest.Delegation_NodeChanges_PropagateFromSubThread failed with "Expected value not to be null or empty because delegation should have a sub-thread path" at 8s into the class — the delegation sub-thread was never created (the sub-thread cold-start shape). Shard 3 again. Branch updated onto current main rather than re-run.

Merge main (post-2515/2520) — shard-3 re-sample rides the update, #2500

CD failed on main: run 33106541165 for commit 96a022b13ab6afd0a318f8b53f65675abbac21dc (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33112114437 for commit 246cce0198e52a690d0b38047e90b54fcec8e13e (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33112455261 for commit ec58d985bae7b2d67e55274054ad39883b4953ed (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33203219455 for commit e6e7d5ccca82055d17c84eb227c3719fe39b129e (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for aaf95af (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for aaf95af. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

CD failed on main: run 33209932873 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed aaf95af (run) — main's HEAD now has the complete image set (portal 3.0.0-rc8.ci.6360), verified against ACR. Installs pick it up on their next 6 h self-update poll. Close this issue if nothing else is outstanding.

CD failed on main: run 33215857120 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33216943470 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33220309534 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33224903139 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33226418068 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33229165977 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33231559262 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33233940783 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33236186273 for commit aaf95afc37e111fbbf41e9a56d21cbb1b27c59ca (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Not reproducing, and one of the two named tests has left this repo

Swept every MeshWeaver Build and Test run on main, 2026-08-26T07:38Z → 2026-08-28T20:46Z (61 h,
121 non-cancelled attempts, 19 failures, read from each shard's own annotations).

OrleansScheduledPostTest no longer exists in this repo. git grep -l "class OrleansScheduledPostTest" origin/main returns nothing; it moved to MeshWeaver.SocialMedia (now
src/MeshWeaver.Social.Test/OrleansScheduledPostTest.cs) via "MeshWeaver.Social leaves core"
(d825905d9, merged 2026-08-27T13:13Z) — about two hours after this issue was filed. It cannot
recur here.

PartitionRootActivationTest still exists (test/MeshWeaver.Hosting.Orleans.Test/PartitionRootActivationTest.cs)
and appears once in the whole sample: run
32951361671, 2026-08-26T09:07:54Z
— on shard 0, not shard 3, and about 2 hours before this issue was filed (2026-08-27T11:20Z).
Since filing: 66 valid main attempts, 0 recurrences.

Both signals point the same way — closing. If PartitionRootActivationTest times out again, it
would be a fresh occurrence against a much smaller shard-0 Orleans suite now that the AI-specific
tests have moved to MeshWeaver.Plugins (#786, merged 2026-08-28T11:59Z), so worth a new issue
rather than reopening this one if it happens.

Navigation Menu

Repository navigation

Orleans shard-3 tests time out on fresh branches while main is green (PartitionRootActivationTest, OrleansScheduledPostTest) #2500

16 remaining items

Not reproducing, and one of the two named tests has left this repo

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Orleans shard-3 tests time out on fresh branches while main is green (PartitionRootActivationTest, OrleansScheduledPostTest) #2500

Description

Activity

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

rbuergi commented on Aug 27, 2026

rbuergi commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

rbuergi commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

rbuergi commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

16 remaining items

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 28, 2026

github-actions commented on Aug 29, 2026

github-actions commented on Aug 29, 2026

github-actions commented on Aug 29, 2026

github-actions commented on Aug 29, 2026

github-actions commented on Aug 29, 2026

github-actions commented on Aug 29, 2026

rbuergi commented on Aug 29, 2026

Not reproducing, and one of the two named tests has left this repo

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions