Fix re-entrant hub construction (FutuRe.Test SIGSEGV / exit=139) by rbuergi · Pull Request #774 · Systemorph/MeshWeaver

rbuergi · 2026-08-04T07:21:47Z

The crash

MeshWeaver.FutuRe.Test crashes intermittently on CI with exit=139 — a SIGSEGV that names no test, so the shard only reports "a test host exited non-zero". It has been one of the recurring reds gating the image channel.

Root cause — from the core dump, not inference

CI already collects mini dumps (DOTNET_DbgEnableMiniDump, copied into the shard artifact). The faulting thread shows the problem literally:

ThreadPool worker ← Rx: MeshQuery result arrives
  MonolithRoutingService.CreateHub
    HostedHubsCollection.GetHub → Lazy → CreateHub
      MessageHubConfiguration.Build                      ← construction #1
        MeshDataSourceExtensions.SubscribeToOwnDeletion
          MeshNodeStreamHandle.Subscribe → GetStream
            Workspace.GetStream → ReduceManager → CreateReducedStream
              HostedHubsCollection.GetHub → Lazy → CreateHub
                MessageHubConfiguration.Build            ← construction #2, NESTED
                  Autofac ComponentRegistryBuilder.Build
                    Autofac ResolvePipelineBuilder.BuildPipeline   ← access violation

SubscribeToOwnDeletion was registered through the synchronous WithInitialization(Action<IMessageHub>) overload, whose actions run inside MessageHubConfiguration.Build ("BEFORE message processing starts"). But it resolves the own-node stream, and resolving a stream can create another hub — so hub construction re-entered hub construction, nesting an Autofac container build inside the in-progress one. ComponentRegistryBuilder is not re-entrant; the nested build corrupts state and the process dies with an access violation (the dump carries AccessViolation + FailFast markers).

It only fires when a query result lands mid-construction on a threadpool thread — which is why it is load-dependent and has never reproduced locally.

The fix

Register it through the observable WithInitialization overload so it runs as a BuildupAction on InitializeHubRequest — after Build has returned — where resolving a stream can no longer nest a container build. Otherwise identical: same subscription, same cache, same disposal registration.

The wrapper is a named static method, not a lambda, deliberately: that overload de-duplicates on delegate identity (RegisteredObservableInits), so a fresh lambda per configurator call would stack N subscriptions instead of collapsing to one — the exact "running on N stacked subscriptions" bug the neighbouring docs warn about.

Verification

dotnet build src/MeshWeaver.Graph -c Release -warnaserror → 0 warnings, 0 errors
MeshWeaver.Graph.Test 779/779
MeshWeaver.FutuRe.Test 59/59

The ordering change means the own-node cache populates on init rather than during Build; Graph.Test covers the consumers of that cache and is green.

⚠️ A green local run does not prove the crash is gone — it never reproduced on macOS/arm64. The claim is narrower: the re-entrant path is removed. Confirmation is FutuRe.Test exit=139 no longer appearing in CI shards.

Release note

Skipped — internal correctness fix, no user-visible change.

🤖 Generated with Claude Code

…init, not a sync one MeshWeaver.FutuRe.Test crashes intermittently on CI with exit=139 - a SIGSEGV that names no test, so the shard only reports "a test host exited non-zero". The CI core dump (DOTNET_DbgEnableMiniDump, collected into the shard artifact) gives the faulting thread directly: ThreadPool worker <- Rx: MeshQuery result arrives MonolithRoutingService.CreateHub HostedHubsCollection.GetHub -> Lazy -> CreateHub MessageHubConfiguration.Build <- construction #1 MeshDataSourceExtensions.SubscribeToOwnDeletion MeshNodeStreamHandle.Subscribe -> GetStream Workspace.GetStream -> ReduceManager -> CreateReducedStream HostedHubsCollection.GetHub -> Lazy -> CreateHub MessageHubConfiguration.Build <- construction #2, NESTED Autofac ComponentRegistryBuilder.Build Autofac ResolvePipelineBuilder.BuildPipeline <- AV SubscribeToOwnDeletion was registered through the SYNCHRONOUS WithInitialization(Action<IMessageHub>) overload, whose actions run INSIDE MessageHubConfiguration.Build ("BEFORE message processing starts"). But it resolves the own-node stream, and resolving a stream can create another hub - so hub construction re-entered hub construction and nested an Autofac container build inside the in-progress one. ComponentRegistryBuilder is not re-entrant; the nested build corrupts state and the process dies with an access violation (the dump carries AccessViolation + FailFast markers). It only fires when a query result lands mid-construction on a threadpool thread, which is why it is load-dependent and has never reproduced locally. Fix: register it through the OBSERVABLE WithInitialization overload so it runs as a BuildupAction on InitializeHubRequest - after Build has returned - where resolving a stream can no longer nest a container build. Behaviour is otherwise identical: same subscription, same cache, same disposal registration. The wrapper is a NAMED STATIC METHOD, not a lambda, on purpose: that overload de-duplicates on delegate identity (RegisteredObservableInits), so a fresh lambda per configurator call would stack N subscriptions instead of collapsing to one - the exact "running on N stacked subscriptions" bug the neighbouring docs warn about. Verified: Release -warnaserror clean; Graph.Test 779/779; FutuRe.Test 59/59. A green local run does NOT prove the crash is gone - it never reproduced on macOS/arm64. The claim is that the re-entrant path is removed; confirmation is FutuRe.Test exit=139 no longer appearing in CI shards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses an intermittent CI-native crash (SIGSEGV / exit=139) by preventing re-entrant hub construction during MessageHubConfiguration.Build, moving a hub init hook from synchronous build-time execution to reactive init-time execution (InitializeHubRequest).

Changes:

Switched the own-node deletion watcher registration to the reactive WithInitialization(Func<IMessageHub, IObservable<Unit>>) overload to avoid nested container builds.
Added a named SubscribeToOwnDeletionInit wrapper to preserve init de-duplication (delegate identity) while deferring execution until hub initialization.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+using System.Reactive;
 using System.Collections.Immutable;
 using System.Reactive.Linq;


github-actions · 2026-08-04T07:39:41Z