MeshWeaver.FutuRe.Test SIGSEGVs after all tests pass: SourcesWatcher keeps requesting against a hub disposed 2s earlier, and reddens a required Plugins gate #3026

MeshWeaver.FutuRe.Test intermittently crashes its test host with SIGSEGV (exit 139) after every test has passed. It reddens Build + test the portal hosts — a required context on MeshWeaver.Plugins — so a green PR becomes an un-mergeable one for a reason nothing in the diff can reach. Observed on Plugins PR #1122 (run 33570183207, shard 3) while shards 0/1/2 passed and a sibling PR's run had all four green, so it is intermittent rather than deterministic.

The CI harness's own annotation states the class exactly, and I am quoting it rather than re-deriving it:

THE TEST HOST EXITED 1 WITH NO FAILING TEST. Every test passed and the host STILL exited non-zero, so nothing failed ON a test thread and there is no failing assertion to find. An exception escaped on a non-test thread and reached AppDomain.UnhandledException … xUnit v3 does NOT hook TaskScheduler.UnobservedTaskException, so this can only be a raw throw on a timer / ThreadPool / dedicated thread — typically a DI resolve inside a single-argument Subscribe(onNext) body, which Rx RETHROWS rather than routing to onError. Fix it by GATING the callback on the hub's teardown state, not by catching the exception; do not re-run.

The measured mechanism: a watcher that keeps running after dispose

From the shard's _meshweaver-test-trace.log (pid 51286), one fixture's teardown:

23:30:20.819  DISPOSE_CLIENTS_DONE elapsed=13ms
23:30:20.825  DISPOSE_INVOKED      elapsed=19ms
23:30:22.877  [FAULT] [Warning] [MeshWeaver.Graph.CompileWatcher] SourcesWatcher: the @@-include
              closure for FutuRe/LocalAnalysis could not be established …
              MeshWeaver.Graph.Configuration.SourceIncludeUnavailableException
               ---> System.ObjectDisposedException: Hub FutuRe/LocalAnalysis was disposed before
                    the response arrived (request type GetDataRequest,
                    target FutuRe/GroupAnalysis/Source/ExternalDependencies).
23:30:22.894  DISPOSE_DONE elapsed=2088ms teardown clean — all pooled I/O joined, async dispose queue drained

So SourcesWatcher is still issuing GetDataRequests two seconds after DISPOSE_INVOKED, against a hub that is already gone. It happens 22 times in this one shard, every one of them for FutuRe/LocalAnalysis, spanning 23:30:22.877 → 23:31:47.274 — i.e. it continues for a minute and a half, across subsequent fixtures.

Note what the teardown assertion says while this is happening: teardown clean — all pooled I/O joined, async dispose queue drained. The straggler is not in the pooled-I/O set the fixture knows how to join, which is why the existing teardown gate does not catch it.

The 22 logged instances are caught (they surface as [FAULT] [Warning]). The crash is the one that is not — same window, same watcher, a path where the throw reaches a raw thread instead of the log. A core dump for the crashing process is in the run's artifact teardown-stragglers-33570183207-1-shard3 (MeshWeaver.Futu-51286.dmp, 1.3 GB) if you want the stack.

Where it lives

Core, not Plugins: MeshWeaver.Graph.CompileWatcher / SourcesWatcher, and SourceIncludeUnavailableException is src/MeshWeaver.Compiler.Pipeline (SourceFingerprintIncludeReader.cs, NodeTypeCompilationHelpers.cs). The suite is merely the observation site — every frame in the failure is core code.

What the fix is not

Per the standing directive, exit 139 is use-after-dispose, never a flake to retry, so please do not close this by re-running or by wrapping the throw in a catch. Catching would convert a crash into 22 more warnings and leave the watcher doing work for a disposed hub. The watcher needs to observe the hub's teardown state and stop — and the fixture's teardown clean verdict should not be able to read clean while a watcher of one of its hubs is still issuing requests.

Possibly the same subject

MeshWeaver.Plugins#1141 (now closed) recorded a one-sample 50s timeout in this same suite where the area resolved to a RedirectControl pointing at itself. I disproved its stated cause (the type-registration warnings appear on green runs too — 114 of them across 28 types on a 2.9s passing run). A compile watcher that cannot establish a source-include closure is a plausible route to a view that never renders its control, so the two may share a root. Not asserted — flagging it so whoever fixes this checks.