Race: RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles — the recycled address's GetDataRequest is access-checked on the DISPOSED hub's LifetimeScope (shard 1, first seen after #2438)

Measured

PR #2721 (a CI/tooling diff: deletes src/MeshWeaver.Plugin.Build pack types, .github/scripts/node-repo-scope.py, docs, scripts/type-forwards.allow — nothing MeshWeaver.Hosting.Monolith.Test can see), run 33297409320 shard 1, 06:51 UTC. The PREVIOUS run on the same branch (33296632888, same test code) passed shard 1. Base = main fa35046df, which includes #2438 (e6e7d5ccc, "hub lifetime scopes now actually close").

MeshWeaver.Hosting.Monolith.Test.RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles [FAIL]
  AssertionException : Expected value not to be <null> because the recycled address must answer again within the budget.
  at RecycleSurvivesItsOwnDisposeTest.cs(87)
Output:
  [06:51:27.081] MeshNodeTypeSource: Created for hubPath=TestData/recycle-survivor (routingSuppliedStream=True)
  [06:51:27.140] [Error] ROUTER_TRAFFIC: RawJson has the mesh hub as sender (sender: mesh/2WRLlj…, target: cache/2WRLlj…)
  [06:51:27.140] [Error] ROUTER_TRAFFIC: StreamEndedEvent has the mesh hub as sender (…, target: sync/LsN-_JJQs…)
  [06:51:27.143] MeshNodeTypeSource: Created for hubPath=TestData/recycle-survivor (routingSuppliedStream=True)   ← the recycled hub
  [06:51:27.158] [Warning] AccessControlPipeline: could not evaluate access for GetDataRequest on TestData/recycle-survivor — reporting UNAVAILABLE
     System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
        at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
        at AutofacServiceProvider.GetRequiredService(Type)
        at MeshWeaver.Hosting.Security.AccessControlPipeline.<>c__DisplayClass2_0.<AddAccessControlPipeline>b__1(IMessageDelivery, CancellationToken, AsyncDelivery)
  [06:51:27.192] === TEST FAILED

The shard also took two hang dumps (06:49:48 and 06:51:57, 1.4 GB / 1.9 GB "Target process is alive") — the run completed normally afterwards (exit=1 TESTFAIL, 1 failing test).

Reading

The address TestData/recycle-survivor is recycled: the second MeshNodeTypeSource: Created line is the new hub. The GetDataRequest that the test then sends is access-checked by the OLD hub's AccessControlPipeline delegate, whose ServiceProvider is the disposed scope — so the pipeline fails closed (UNAVAILABLE → null) and the test's "answers again within the budget" assertion reads null. Before #2438 that scope never actually closed, so the same ordering resolved the service from a zombie scope and passed. This is the #2666 class ("callbacks that resolve services after their container is disposed"): a delivery that reaches a hub's pipeline after the hub was recycled must be re-routed to the replacement (or refused with a routing NotFound that the caller re-probes), not evaluated on the corpse.

Not re-run silently: the failed jobs on #2721 were re-run AFTER filing this, because the diff cannot reach the test and the previous run on identical test code passed.

Related: #2438, #2666, #2543 (per-node hubs stop answering mid-bake).

This fired on main once (`81f8869d5`), it took the host with it, and the next main tip is green — please don't read that green as a fix

Recording the main-side occurrence and the attribution, since this issue was filed off a PR run.

On main, run 33298742141 (81f8869d5, the #2638/#2726 merge), shard 1:

[CI] MeshWeaver.Hosting.Monolith.Test (part 1/2) exit=1 MASKED
     (trx records 2 failing test(s), which does not explain exit=1 — host crashed after
      streaming results) elapsed=462s

with the synthetic MeshWeaver.Hosting.Monolith.Test.HOST_CRASHED entry. The two recorded failures were this test and UserHomeLegacyPathResolutionTest.LegacyUserPrefix_RendersHomeArea_NotAreaNotFoundPlaceholder. So this is not only a failing assertion — on that run the host did not survive it.

Attribution, by reachability rather than adjacency: neither merge under the red can reach it. test/MeshWeaver.Hosting.Monolith.Test.csproj has no Orleans ProjectReference (Fixture, GitSync, Hosting.Monolith, Hosting, Kernel.Hub, Mesh.Contract, Monolith.TestBase, Layout, Graph, Documentation) and the monolith host runs no Orleans at all, so #2726 (MeshWeaver.Hosting.Orleans only) is out; #2721 is out for the reason this issue's own body already gives. The preceding red (5e73bfcaa) was a different shape entirely — exit=1 TESTFAIL, one SilentReadNackTest flake, host completed normally.

The next main tip, d3272b5f8, is green (run 33299286168, shard 1 + Consolidate). It is the first main commit carrying #2728 (issue #2679), which removed PermissionEvaluator.GetEffectivePermissions' per-emission hub.ServiceProvider resolution from inside its SelectMany fold and added the shared probe-gated ScopeTeardown. That is the same hazard family as this issue's title — an access check resolving on a disposed LifetimeScope.

🚨 Suggestive, not proof. One green run never settles a race, and there is a specific reason to expect a residual here: AccessControlPipeline's synchronous prologue is already guarded (the #974 sweep, AccessControlPipeline.cs:80-105) — on a disposing hub it refuses the delivery and reports Unavailable rather than throwing. That behaviour is exactly what produces this test's assertion failure ("the recycled address must answer again within the budget"). So the open question is recycle/dispose ordering — whether the confirming hub reliably outlives the hub it recycles — not an unguarded resolve, and #2728 does not touch it.

Suggested next step for whoever takes it: run this test repeatedly against d3272b5f8 (and against 81f8869d5 as the control) rather than closing on the green.

Correction to my previous comment: `81f8869d5` did NOT crash — the host completed, and the `MASKED` marker was a misclassification

@meshweaver-61 caught this and I verified it. My earlier statement "on that run the host did not survive it" is wrong; please disregard that part.

The assembly printed its own execution summary on that run (33298742141, shard 1):

=== TEST EXECUTION SUMMARY ===
   MeshWeaver.Hosting.Monolith.Test  Total: 331, Errors: 0, Failed: 2, Skipped: 0, Not Run: 0, Time: 461.193s

so it ran to completion and reported. The MASKED (host crashed after streaming results) marker is derived, not observed: the harness infers a crash from rc != <failures in trx>, assuming xUnit v3 exits with the failure count — it exited 1 with 2 failures. The classifier rule is being fixed in #2738.

What that changes here: this test failed as an ordinary assertion failure on main, alongside UserHomeLegacyPathResolutionTest.LegacyUserPrefix_RendersHomeArea_NotAreaNotFoundPlaceholder, and took nothing else down with it. So the occurrence is still real and still worth this issue — but it is not evidence of a process-level crash, and it should not be folded into any host-crash cluster.

Unchanged: the attribution (neither #2721 nor #2726 can reach MeshWeaver.Hosting.Monolith.Test — no Orleans ProjectReference, monolith runs no Orleans), and the caution about the following green tip: d3272b5f8 carries #2728, which removed PermissionEvaluator's per-emission hub.ServiceProvider resolution — same hazard family as this issue's title — but one green run does not settle a race, and AccessControlPipeline's prologue guard (#974, AccessControlPipeline.cs:80-105) refusing with Unavailable on a disposing hub is exactly what produces this assertion. The measurement worth making is recycle/dispose ordering: does the confirming hub reliably outlive the hub it recycles.

Two more single Monolith.Test failures on shard 2 today, on a PR with zero C# in its diff (#2734, run 33299828739 and its re-run of the failed jobs):

SilentReadNackTest.OwnerDisposedWithReadOutstanding_IsNacked_NotLeftHanging — got GetDataResponse where a DeliveryFailureException NACK was expected (the SilentReadNackTest.OwnerDisposedWithReadOutstanding is 35% flaky on main — two disposal arms race, and the test asserts a deterministic outcome over it #1599 shape, reopened by the same disposal-arms race).
UserHomeLegacyPathResolutionTest.LegacyUserPrefix_RendersHomeArea_NotAreaNotFoundPlaceholder — 30 s methodTimeout, i.e. a wedge, not an assertion; the test started at 08:07:29.523Z and never produced a second output line.

Both land in MeshWeaver.Hosting.Monolith.Test on the shard this issue names, after #2438; recording them here as further measurements of the same cause rather than opening a third 'flaky test' issue (#2700).

The access-pipeline half is fixed in #2740 — and a correction to the `SilentReadNackTest` attribution in the comment above

Fixed (#2740): the seam this issue's own captured stack names —
AccessControlPipeline → GetRequiredService → ObjectDisposedException on the recycled hub's
closed scope. It was classified Unavailable with a sentence matching none of the markers
MeshNodeStreamCache.IsTransientOwnerFailure / RoutingGrain.IsTransientFailure /
AreaErrorClassifier.IsTransientHubFailure key on, and posted through a hub whose Post is gated
shut past DisposeHostedHubs — so the caller got a terminal-looking answer (or silence) about an
address that is alive one activation later, and never re-probed. It is now refused in the same
vocabulary MessageService's shutting-down gate uses, NACKed through the parent. Refusal is
unchanged (next is still never invoked); only the classification and the route change.

🚨 Correction: `SilentReadNackTest.OwnerDisposedWithReadOutstanding` is a DIFFERENT terminal

The comment above records it here as "the #1599 shape, reopened by the same disposal-arms race".
Measured against the fix branch, that attribution does not hold — please don't let it send the
next person to the access pipeline:

With fix(#2727): a delivery that reaches a RECYCLED hub is refused as transient, not as a permission verdict #2740 applied, a full MeshWeaver.Hosting.Monolith.Test run still fails it (1 of 767).
The same test passes 8/8 in isolation (--filter FullyQualifiedName~SilentReadNackTest) —
the bulk-only signature, so the full-suite run is the repro vector, not the test alone.
The failure's answer is GetDataResponse { Error = "Exception has been thrown by the target of an invocation." }. HandleGetDataRequest produces that shape in exactly one place: the Catch
that converts a non-hub-disposal fault into a response value. So
HubDisposingException.IsHubDisposal(ex) returned false for that fault, the read never
reached the NACK arm, and the access pipeline is not on the path at all.

Refuted while checking: the natural explanation — that the recognition walk gives up too early
— is wrong. IsHubDisposal → ExceptionChain.Contains walks up to 256 nodes, follows
AggregateException fan-out and tolerates cycles, so a HubDisposingException would be found at
any realistic depth.

Still unconfirmed: which exception it actually is. The failure carries only the
TargetInvocationException wrapper text. The test disposes the data-source stream directly and
then posts the recycle, so a plausible candidate is the stream's own ObjectDisposedException —
not a hub disposal, therefore answered as data rather than NACKed. If that is right, the fix is
this issue's principle one layer down (a disposed-resource fault is not a read result and must not
claim the once-only answer slot — the #1362/#1470 reasoning already written into that Catch's own
comment), but it is a distinct change in a path with a regression history and I am not shipping it
on an unconfirmed cause.

How to confirm it cheaply, for whoever picks it up: the terminal is reached under the full
suite, so log the unwrapped ex.GetType().Name / innermost message at that Catch (or break on
it) during one bulk run — that names the exception in a single iteration instead of guessing.

A paired flake-repro.yml rate run (100 iterations, this test, main vs the fix branch) is in
flight; I will post both numbers here so the residual rate is a measurement rather than an
impression.

fix(#2727): a read that faults on a CLOSED DI SCOPE is NACKed, not an…

Merge pull request #2740 from Systemorph/fix/2727-recycled-hub-nack

The other half is measured and fixed: 20/100 → 1/100 (#2753)

The exception is named — the thing #1470, #2673 and this issue have all discussed without ever identifying, because it arrives as TargetInvocationException whose message is the useless "Exception has been thrown by the target of an invocation." Unwrapping it (first commit) and one flake-repro run gave:

TargetInvocationException → ObjectDisposedException:
  "Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope
   as it (or one of its parent scopes) has already been disposed."

The hub's Autofac scope was closed underneath an in-flight read — so my earlier comment here guessed the right class (a disposed resource) and the wrong object (I said the stream). Correcting that for the record.

It is not a HubDisposingException — a scope closed under a live delivery announces nothing — so HandleGetDataRequest's Catch sent it to the value arm, fabricated a GetDataResponse{Error}, claimed the once-only answer slot, and told the caller a node that exists does not. The #1362/#1470 shape, reached by a different cause.

Measured on the 4-vCPU runner, rate mode, 100 iterations, same filter throughout:

branch	failures / 100
`main`	20
fix, first draft	10 — the rethrow worked, but I had reworded the NACK and dropped the `"shutting down"` marker `IsTransientOwnerFailure` keys on; all 10 were that regression
fix, marker restored	1 (z = −4.38)

The remaining 1 is GetMeshNode_WhenOwnerIsDisposedMidRead_RecoversOnTheReProbe — #2673, which the class filter also matches. OwnerDisposedWithReadOutstanding is 0/100.

So this issue's three terminals are now all fixed

terminal	fix
drain ordering — scopes closed before pooled I/O joined	#2735 ✅ merged
access gate — refusal not transient, posted through a dead hub	#2740 ✅ merged
read path — a closed scope answered as data	#2753 (armed)

One root cause — work touching a disposed scope, classified as an answer instead of a teardown — at three layers.

A gap this exposed, worth its own issue

38 sites emit "is shutting down" as a contract marker and MeshNodeStreamCache.cs:1374 classifies on it, but nothing pins the pairing. My regression was caught only by a 20 %-flaky integration test — it would have been missed on 4 runs in 5. A deterministic guard (every teardown-NACK producer's wording must satisfy IsTransientOwnerFailure) belongs with RecycledHubRefusalTest. Noted at the call site; not folded into the fix.

Merge pull request #2753 from Systemorph/probe/silentread-cause

Closing: all three terminals this issue names are fixed and merged, and the measured residual belongs to #2673, which is closed.

One root cause — work touching a disposed scope, classified as an answer instead of a teardown — at three layers:

terminal	fix	state
drain ordering — scopes closed before pooled I/O joined	#2735	merged `f71fcfbb8`
access gate — refusal not transient, posted through a dead hub	#2740	merged `619927afd`
read path — a closed scope answered as data	#2753	merged `de321a440`

Measured, not asserted: 20/100 failures on main → 1/100 after the fix (z = −4.38). The intermediate 10/100 is itself worth recording — the rethrow worked, but the NACK had been reworded and lost the "shutting down" marker IsTransientOwnerFailure keys on; all ten were that regression.

The remaining 1/100 is GetMeshNode_WhenOwnerIsDisposedMidRead_RecoversOnTheReProbe — #2673, which the class filter also matches and which is closed. OwnerDisposedWithReadOutstanding measured 0/100.

The gap this exposed — 38 sites emitting "is shutting down" as a contract marker with nothing pinning the pairing — is now filed separately rather than left in this thread.

State after today's landings

The chain that this issue, #1470 and #2673 turned out to share is in: #2735 (hub scopes close after the drains — the LifetimeScope … already disposed root), #2740 (a delivery reaching a recycled hub is refused as transient), #2753 (a read that faults on a closed DI scope is NACKed, never answered as data), #2757 (an undetermined permission check on a hub that is gone is transient). Measured on the SilentReadNackTest class: 20/100 → 1/100 → 0/100 (flake-repro rate, 100 iterations each).

What is not measured is a 100-iteration rate on this test, RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles, after #2752 widened its assertion to a terminal answer. Leaving open for that number; nothing else is pending on it.

fix(#2795): pin the teardown marker to its classifiers, executably

Navigation Menu

Repository navigation

Race: RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles — the recycled address's GetDataRequest is access-checked on the DISPOSED hub's LifetimeScope (shard 1, first seen after #2438) #2727

Measured

Reading

This fired on main once (`81f8869d5`), it took the host with it, and the next main tip is green — please don't read that green as a fix

Correction to my previous comment: `81f8869d5` did NOT crash — the host completed, and the `MASKED` marker was a misclassification

The access-pipeline half is fixed in #2740 — and a correction to the `SilentReadNackTest` attribution in the comment above

🚨 Correction: `SilentReadNackTest.OwnerDisposedWithReadOutstanding` is a DIFFERENT terminal

The other half is measured and fixed: 20/100 → 1/100 (#2753)

So this issue's three terminals are now all fixed

A gap this exposed, worth its own issue

State after today's landings

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Race: RecycleSurvivesItsOwnDisposeTest.TheHubThatConfirmsTheRecycle_OutlivesTheHubItRecycles — the recycled address's GetDataRequest is access-checked on the DISPOSED hub's LifetimeScope (shard 1, first seen after #2438) #2727

Description

Measured

Reading

Activity

rbuergi commented on Aug 30, 2026

This fired on main once (81f8869d5), it took the host with it, and the next main tip is green — please don't read that green as a fix

rbuergi commented on Aug 30, 2026

Correction to my previous comment: 81f8869d5 did NOT crash — the host completed, and the MASKED marker was a misclassification

rbuergi commented on Aug 30, 2026

rbuergi commented on Aug 30, 2026

The access-pipeline half is fixed in #2740 — and a correction to the SilentReadNackTest attribution in the comment above

🚨 Correction: SilentReadNackTest.OwnerDisposedWithReadOutstanding is a DIFFERENT terminal

rbuergi commented on Aug 30, 2026

The other half is measured and fixed: 20/100 → 1/100 (#2753)

So this issue's three terminals are now all fixed

A gap this exposed, worth its own issue

rbuergi commented on Aug 30, 2026

rbuergi commented on Aug 30, 2026

State after today's landings

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

This fired on main once (`81f8869d5`), it took the host with it, and the next main tip is green — please don't read that green as a fix

Correction to my previous comment: `81f8869d5` did NOT crash — the host completed, and the `MASKED` marker was a misclassification

The access-pipeline half is fixed in #2740 — and a correction to the `SilentReadNackTest` attribution in the comment above

🚨 Correction: `SilentReadNackTest.OwnerDisposedWithReadOutstanding` is a DIFFERENT terminal