Repository navigation
DataContext's transient-fault retirement races its own disposal for the deferred backlog — the requester gets the generic disposal nack instead of the cause (#4162) #4261
Description
The failure
MeshWeaver.Data.Test.TransientInitializationFaultRetiresTheActivationTest.ATransientFault_RetiresTheActivation_AndTheNextDeliveryIsServedByAFreshOne(which: DataContext), verbatim:
MeshWeaver.Reactive.Assertions.AssertionException :
Expected "Hub host/1 was disposed while ProbeRequest (id=b2fedf570e6142e7b623e9af19a785a3) was
still deferred; initialization gates closed at deferral: [DataContextInit] — the message was
never processed. The address may reactivate (recycle / restart); retry to get the authoritative
answer."
to contain "transient infrastructure fault"
because the refusal names the cause the caller should expect to clear.
at TransientInitializationFaultRetiresTheActivationTest.cs(184,0)
The requester got the disposal nack instead of the transient-infrastructure-fault nack. Both are ErrorType.ShuttingDown and both say "retry", so the behaviour #4067/#4068 shipped is intact — what is lost is the CAUSE, which is the half of the fix a caller reads.
The mechanism, read from the code (not inferred from the failure)
DataContext.SettleInitializationGate, the transient branch:
// Dispose FIRST so the refusal is classified ShuttingDown (the reporters read IsShuttingDown),
// FailGate SECOND so the caller's error names the database rather than the recycle.
Hub.Dispose();
Hub.FailGate(InitializationGateName, reason);The comment states the required ordering; the code cannot enforce it, because the two calls do not run on the same thread:
SettleInitializationGateruns on the thread pool —OpenInitializationGateends with.ObserveOn(TaskPoolScheduler.Default).Subscribe(_ => SettleInitializationGate(allInit), …).Hub.Dispose()only starts the teardown: it postsShutdownRequestand returns;MessageService.Dispose()runs later, inside that request's turn, on the hub's action block.- Both ends then drain the same queue through
DrainDeferredDeliveries: the disposal drain answers with"Hub {Address} was disposed while {Message} … was never processed", andFailGate→FailDeferredBackloganswers with the transient reason.
So the requester's answer is decided by whether the hub's action block reaches MessageService.Dispose before the thread-pool thread reaches FailGate. Under load the action block can win — and then the reason the retirement exists to carry is replaced by a generic one. FailDeferredBacklog also Clear()s the parked turns precisely so nothing is answered twice, which confirms both paths are expected to compete for the same backlog.
(There is a second, narrower branch of the same race: SettleInitializationGate opens with if (Hub.IsShuttingDown), which answers with the shutdown reason. A disposal that progresses far enough before the settle runs takes that branch instead of the transient one.)
Denominator, honestly
- 1 failure in 6 consecutive full-project
MeshWeaver.Data.Testruns (macOS, Release, alone on the machine), 2026-09-14. - 0 in 8 further full-project runs since, split across two trees (
origin/mainand a feature branch), and 0 in 8--filtered runs of this test alone — it wants the concurrent load of the whole project, which is why a filtered re-run is not a control. - Not reproduced yet on
origin/mainspecifically; the mechanism above is entirely insideDataContext.cs+MessageService.csatorigin/main(1c1d62adff), and the tree it was observed on differs fromorigin/mainonly inSynchronizationStream's fault-flag publication (fix(data): a faulted stream is refused from the instant its terminal is delivered (#4180, #4244) #4255), which has no path to the deferred-queue drain.
Filed rather than catalogued: it is load-dependent, i.e. a race, i.e. a defect.
What a fix has to preserve
Both halves of the comment, at once:
- the refusal must classify as
ErrorType.ShuttingDown(which is whyDispose()is called first today), and - the backlog must be answered with the transient cause, exactly once.
Answering the backlog before starting the teardown, with a reason and a classification chosen explicitly rather than read off hub.IsShuttingDown at drain time, satisfies both without depending on which thread wins. That is one shape; the point of this issue is that any fix has to stop the two drains from racing, because ordering two calls on two threads cannot.
Related
- fix: a transient infrastructure fault at init retires the activation instead of latching it FAILED (#4067, #4068) #4162 — the change that introduced this branch and this test (PostgreSQL database at 10.42.18.4:5432 is unreachable — all MessageHub sync initializations fail with connection timeout #4067, Transient DNS failure at PostgreSQL connect time permanently latches mesh DataContext hubs in FAILED state #4068).
- fix(data): a faulted stream is refused from the instant its terminal is delivered (#4180, #4244) #4255 / StreamResyncGivesUpTest: a faulted stream reads USABLE for the whole of its own terminal delivery, so the cache hands the corpse to the consumer it just told — #4151's store-before-flag window (consolidates #4244; 6 occurrences) #4180 — a different faulted-stream race, found in the same session; listed so the two are not merged by resemblance. Its own test class documents a prior "assertion pinning one side of a legal ordering" in this test; this is a third ordering, and unlike those it is the PRODUCT that picks the wrong side, not the test.
Activity
rbuergi commentedon Sep 14, 2026
Confirmed from a failing run's own trail — the two drains, 1 ms apart
That is the sequence this issue describes, observed:
Hub.Dispose()returned on the thread-poolsettle thread, the hub's action block reached
MessageService.Disposeone millisecond later, andits drain answered the deferred
ProbeRequestbefore the settle thread's very next statement —Hub.FailGate(InitializationGateName, reason)— got there. The requester is told "the message wasnever processed" instead of "transient infrastructure fault". The fence trail confirms the request
was parked exactly where the test intends (
DEFERRED gates=[DataContextInit] … QUEUED queue=deferred pos=1), so the test's own setup is not at fault.Measured rate, and one honest loose end
A/B, full-project
MeshWeaver.Data.Test, alternating trees, alone on the machine, macOS Release,30 pairs (2026-09-14):
origin/main(1c1d62adff)Plus one earlier failure in 6 runs on the #4255 tree — 3 in 36 vs 0 in 30.
🚨 That asymmetry is NOT significant and must not be read as a cause. If both arms had the same
rate, all three failures landing on one arm has p ≈ 0.16. It is recorded because it is what was
measured, and because the next person should neither dismiss it nor act on it. Two things argue it is
sampling rather than coupling:
SynchronizationStreampublishesits terminal fault relative to
Store.OnError. The race here is between a thread-pool continuationexecuting its next statement and the hub's action block reaching its
ShutdownRequestturn —neither side is touched. What a diff can do is shift how many threads are busy at that instant,
which changes the odds of an already-unguarded race without being its cause.
Settling it would need ~60 more pairs. The better use of that time is to remove the window, after
which neither arm can fail.
What a fix has to solve, stated precisely
Calling
FailGatebeforeDispose()is not sufficient on its own:AnswerUnreleasableDeliveryderives the classification at drain time —
hub.IsShuttingDown ? ErrorType.ShuttingDown : ErrorType.Failed— so a backlog answered before the teardown starts would be classifiedFailed,which is the terminal verdict #4067/#4068 exist to avoid (and the test's first assertion). Hence the
current
Dispose()-first ordering, and hence why it cannot simply be swapped.So the classification has to become explicit — carried with the reason rather than read off the
hub's state at the moment the drain happens — and the backlog answered once, before the teardown can
answer it differently. Then the order stops being a race, and
Dispose()can follow.rbuergi commentedon Sep 14, 2026
The test that shipped with #4278 fails on another PR's branch — and it is the SAME race, from the other side
RetirementNamesTheCauseWhenTheTeardownWinsTest.TheBacklogIsAnswered_BeforeTheTeardownThatCouldAnswerItDifferentlyfailed on #4282's run 34817271676, shard 1:#4282's diff is
resolve-gate-platform.shand its shell tests — it cannot reachDataContext's retirement path.mainis green on the same commit range (34817860374, 07:26Z).What the failure says is the interesting part. The assertion is not the ordering this test exists to pin — it is the precondition for having an ordering at all: the teardown did not happen, so there was nothing to race. That is this issue's own mechanism observed from the other side. #4261 measured
Hub.Dispose()(async: postsShutdownRequestand returns) racingHub.FailGate(...)on a thread-pool continuation, with the disposal drain winning by ~1 ms. Here the disposal drain apparently did not arrive within the test's window at all, soNotBeNullfired instead of the order assertion.So the new guard is itself timing-dependent on the same window it guards, which makes it a source of unattributed reds on unrelated PRs — exactly the cost pattern recorded in #4244 (six occurrences, four PRs, two dequeues). Flagging rather than fixing, because the shape of the fix is this issue's open question: #4261 already records that swapping the two calls is insufficient while
AnswerUnreleasableDeliveryderivesShuttingDownvsFailedfromhub.IsShuttingDownat drain time — the classification has to become explicit before any ordering, including this test's, can stop being a race.Until then: a red on this assertion on a PR that cannot reach
DataContextis this issue, not that PR's diff.