A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991

Summary

MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal intermittently fails with a real GC-root detection: a single-shot Observable.Timer closure keeps a TimerQueueTimer alive on the process-wide TimerQueue, and that chain pins a disposed mesh's MessageHub.

This is filed to preserve the root chain — CI logs expire, and the guard already did the hard part of naming the culprit.

The chain (ClrMD, run 31300373896, shard 1, Linux)

ROOT[StrongHandle] System.Object[]
  → System.Threading.TimerQueue[] → TimerQueue → TimerQueueTimer
  → System.Reactive.Concurrency.ConcurrencyAbstractionLayerImpl+Timer
  → UserWorkItem<ValueTuple<ObservableImpl.Timer+Single+_, …>>
  → System.Reactive.Linq.ObservableImpl.Timer+Single+_
  → System.Reactive.AnonymousSafeObserver<Int64> → System.Action<Int64>
  → MeshWeaver.Mesh.ActivityControlPlaneExtensions+<>c__DisplayClass2_1<MeshNode>
  → System.Action
  → MeshWeaver.Mesh.ActivityControlPlaneExtensions+<>c__DisplayClass2_0<MeshNode>
  → System.Func<IObservable<MeshNode>>
  → MeshWeaver.Graph.Configuration.NodeTypeCompilationHelpers+<>c__DisplayClass5_0
  → MeshWeaver.Messaging.MessageHub  [RunLevel=6] [Address=AccessAssignment]

visited=253524 hubFound=True, i.e. the analysis ran for real (not the Unavailable skip path from #674).

Reading it

An Observable.Timer subscription created in ActivityControlPlaneExtensions — reached through NodeTypeCompilationHelpers — is not disposed when the mesh is disposed. Rx registers the timer on the shared TimerQueue, which is a strong-handle root, so for as long as the timer is pending it transitively holds the two display-class closures, the Func<IObservable<MeshNode>>, and finally the AccessAssignment MessageHub.

That explains the intermittency without making it a flake: if the timer has already fired by the time the probe forces GC, the hub is collected and the test passes; if it is still pending, the hub survives and the guard fires. The leak is present either way — the probe just samples it.

Why it matters beyond the test

MeshHubDisposalLeakTest's own docstring ties this class of leak to the Acme bulk failure (UpdateNodeRequest@…/DefinePersona never replying, but only once another test class ran first in the same process) — a disposed mesh's hub graph accumulating across classes. A leaked timer subscription is exactly that shape, and in a portal it accumulates per compiled node type rather than per test.

Suggested direction (not prescriptive)

The subscription needs to be tied to the owning hub's disposal — the timer's IDisposable added to the hub's disposables rather than left to run free. Worth checking every Observable.Timer / Interval in ActivityControlPlaneExtensions and NodeTypeCompilationHelpers for the same missing disposal, not just the one arm the probe happened to catch.

Provenance

Observed on PR #984, whose diff is 5 non-C# files (workflow YAML, client catalogues, a client .ts test, one markdown doc) and cannot create a GC root. main at 906fc8827 — the same commit that branch merges with — ran shard 1 and passed this test 68 minutes earlier (run 31298376804), so this is pre-existing and intermittent, not a regression from that PR.