Repository navigation
Two teardown tests are duplicated across repos and drifted: #3291 updated core's copies only (not a regression) #3345
Description
What breaks
Two long-standing tests in MeshWeaver.Hosting.Monolith.Test (MeshWeaver.Plugins) now fail with
System.TimeoutException — the waiter is never answered:
NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCallerLateNackReenqueueTest.LateOwnerDisposingNack_AfterOptimisticEmit_ReenqueuesAndLands
Both migrated from core on 2026-08-30 / 09-01, so this is existing behaviour that changed, not
new assertions written against a new contract. The test's own trace stops right after
[fence] caller is armed on the late watch (ArmedCount=1).
Bisected to #3291
Controlled A/B: same Plugins tree, same two tests, only MeshWeaverRoot differs. Each arm was a
clean build + run, so this is not a flake re-run — the code under test changes between arms.
| core commit | what it is | result |
|---|---|---|
7d644de95 |
Plugins' current MW_PLATFORM_REF |
2 / 2 pass |
f9d2a335a^1 |
the commit before #3291 | 2 / 2 pass |
f9d2a335a |
merge of #3291 fix/robust-disposal |
fail |
fc4adc666 |
#3293 | fail |
684cc6ac1 |
#3302 | fail |
d99165850 |
the set ci.7808 was cut from |
0 / 2 fail |
The transition is at f9d2a335a. Everything after it stays broken.
🚨 #3302 is NOT the cause, and I said it was before I measured. Its titles line up seductively —
97a0284f5 "a reply routing cannot deliver is handed to the waiter instead of dropped", then
e719c74e3 dropping the intake gate 25 minutes later — and the failing tests are about exactly
that. But the arm before #3302 already fails. A story assembled from commit titles is not a cause;
recording it here so nobody re-derives the same wrong lead.
Where to look
#3291 is the "teardown lets accepted work finish, never forced" change. Its diff touches the right
surfaces: MeshTeardownHostedService.cs, ActivityTracker.cs (+271), DataExtensions.cs,
JsonSynchronizationStream.cs, MeshNodeStreamExtensions.cs, plus TeardownLayers.md (+236) and
HubDisposalModel.md.
It also carries an internal revert — 4628df971 "Revert the late-verdict sink alongside an
accepted post — it reorders the ack ahead of the state" (DataExtensions.cs,
PatchVerdictRoutingTest.cs). A revert inside the same PR is worth checking first: the tests here
are precisely about a late verdict reaching a waiter, and the two commits either side of that revert
(62be91698, ef42bed55 — "an armed late watch is served through the sink first") name the
mechanism the failing test prints.
Repro
git worktree add --detach /tmp/core-before f9d2a335a^1
git worktree add --detach /tmp/core-after f9d2a335a
# in a MeshWeaver.Plugins checkout:
dotnet build src/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj \
-c Release -p:MeshWeaverRoot=/tmp/core-before # then --no-build test → 2/2 pass
dotnet test src/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj \
-c Release --no-build \
--filter "FullyQualifiedName~NackReachesTheWaiterDuringTeardownTest|FullyQualifiedName~LateNackReenqueueTest"
# repeat with -p:MeshWeaverRoot=/tmp/core-after → both fail, TimeoutExceptionWhy it went unnoticed, and what it blocks
This is AGENTS.md break shape 7 — a method's BEHAVIOUR changing behind an unchanged signature.
No surface detector can see it, the Cross-repo pair gate does not fire, and the dependent's own CI
does not catch it either, because MeshWeaver.Plugins pins core: its platform-ref job resolves
the same pin on a release event as on a PR. The pin bump IS the integration test (#3259), and this
is what it caught on its first run in a day.
It blocks MeshWeaver.Plugins#1359 (the 7755 → 7808 pin bump), which is held as a draft rather
than merged, because landing it would import this regression into every Plugins build. #1359 in turn
gates #3327 and part of #3203.
Activity
rbuergi commentedon Sep 5, 2026
Corrected: this is not a regression. The title is wrong and so was my root cause.
I narrowed the bisect to a single commit and it refuted the premise. Nothing in #3291 broke a
behaviour. What broke is that these two tests exist twice, and #3291 updated one copy.
The bisect, exact
The Plugins test project was held constant; only
MeshWeaverRootvaried across detached coreworktrees. Filter: both test names,
MeshWeaver.Hosting.Monolith.Test.6163dfacbf92305ae146e6f7941ef42bed557d644de95d99165850So the entire delta is
f92305ae1— "Teardown lets accepted work finish and names what it had tostop — no forced teardown" — the maintainer directive of 2026-09-04. One commit, not a merge.
Why it looked like a product regression
NackReachesTheWaiterDuringTeardownTestandLateNackReenqueueTestlive in both repos:test/MeshWeaver.Graph.Test/here, andMeshWeaver.Plugins/src/MeshWeaver.Hosting.Monolith.Test/where a monolith mesh can actually be stood up. Core's copies carry the note "Core twin of … Keep
the two in step."
f92305ae1rewrote core's copies to the new contract and left the Pluginsoriginals asserting the contract it had just deleted.
The divergence is one clause. Core's parked turn releases itself when the owner starts shutting
down:
The Plugins copy waited only for the test's own
finally. With the 8 s force-teardown deleted, aturn blind to the shutdown holds the owner at
DisposeHostedHubs— honestly pending — and thecaller burns its verdict budget, which is the
VERDICT_TIMEOUTin the log. Core's rewritten copypredicts this exact failure in prose; nothing measured it.
The fix, and the control
In Systemorph/MeshWeaver.Plugins#1359 (the pin bump, where it belongs — the guard below compares
against
MW_PLATFORM_REF, so it can only be green once the pin moves):d99165850; namespace and header comment are the onlyedits. They pass at both pins — the release-on-shutdown turn needs no watchdog either way —
so this was never pin-gated, it was just stale.
TeardownTwinParityTest— the control the comment was standing in for. It compares each bodybelow the
namespaceline against the core checkout at the pin, never core'smain, so itreddens only in the change that MOVES the pin, which is the change that owes the update. A
missing platform checkout fails rather than skips.
Verified it discriminates rather than merely passing: 4/4 at
d99165850; at7d644de95the twoparity theories fail, naming the file, the line and both sides.
What this says about the guard surface
This is break shape 7
— behaviour changing behind an unchanged signature — with a twist worth recording: the dependent
held a duplicated test asserting the old behaviour. No surface detector can see it, the
cross-repo pair gate has nothing to resolve, and the symptom surfaces in an unrelated-looking suite
one pin bump later. The marker that was supposed to prevent it was a comment, and it sat on the copy
whose editor already knew. I will carry that into
CrossRepoPairGate.mdhere.Closing when #1359 merges.
[-]Regression: a disposing owner stops answering its waiter — #3291 breaks two long-standing Nack/teardown tests, bisected[/-][+]Two teardown tests are duplicated across repos and drifted: #3291 updated core's copies only (not a regression)[/+]rbuergi commentedon Sep 5, 2026
Closed — twins synced, and all four portal-host shards are green on
mainNot a regression, as the corrected title says. The two tests exist in both repos because core's
CI cannot stand up a monolith mesh; #3291 rewrote core's copies to the no-forced-teardown contract
and left the Plugins originals asserting the contract it had just deleted. The bisect, with the test
project held constant and only
MeshWeaverRootvarying, put the entire delta on one commit:6163dfacbf92305ae1One clause: core's parked turn releases itself on
|| owner.IsShuttingDown; the Plugins copy waitedfor the test's own
finally. With the 8 s force-teardown deleted, a turn blind to the shutdown holdsthe owner at
DisposeHostedHubs— honestly pending — and the caller burns its verdict budget.Landed:
pins, so they were never pin-gated, and the stale copies were reddening live PRs (An install can be torn down by DisposeHostedHubs two seconds in, writing 0 nodes — disposal does not wait for work in flight #1360 lost
Portal hosts (shard 2)to exactly this 10 sTimeoutException). All four shards are green onPlugins
mainnow.Doc/Architecture/CrossRepoPairGategains "Shape 7's worst form: the dependent holds a COPY of thetest".
TeardownTwinParityTest, the control the comment was standing in for. Itcompares each body below the
namespaceline against the core checkout atMW_PLATFORM_REF,so it reddens only in the change that moves the pin — the change that owes the update. A missing
platform checkout fails rather than skips. Verified it discriminates: 4/4 at the current pin,
and at the previous pin both parity theories fail naming the file, the line and both sides.
The lesson worth keeping
The marker was one-sided. Core's copy said "keep the two in step"; the Plugins copy said nothing
— so the note was addressed to the party who already knew, and the one who needed it never saw it.
Prose that asks a future reader to keep two files in step only fires when someone is already looking
at the right file. A duplicated test needs a parity guard, and the marker goes on both copies.