Repository navigation
ActivityControlPlane watcher reports the boot-time content-type probe as a transient ERROR and arms a re-establish #2990
Description
Symptom
Every mesh start writes one Error-level line per static NodeType whose HubConfiguration
installs an Activity Control Plane:
Error MeshWeaver.Kernel.Hub.KernelContainer: ActivityControlPlane subscription faulted on
content-type-registration/65936dcbcfa348bd921eb49885dea848 — re-establishing
The address is the giveaway: content-type-registration/{guid} is minted only by
ContentTypeRegistration.ProbeRegister (src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs:70).
Nothing is actually wrong with the mesh — the line reports the boot-time content-type sweep against
itself.
Evidence
Reproduced locally on origin/main (24254ad65), captured off the mesh's own ILoggerFactory in
Memex.Portal.Shared.Test (an ILoggerProvider registered through MeshBuilder.ConfigureServices,
so this is what a deployment's log sink receives, not test output). The exception attached to the
record:
System.InvalidOperationException: Failed to create stream
at MeshWeaver.Data.Workspace.ReduceLocalStream[TReduced](WorkspaceReference`1 reference, Func`2 configuration) in src/MeshWeaver.Data/Workspace.cs:line 925
at MeshWeaver.Data.Workspace.<>c__DisplayClass33_0`1.<GetStream>b__1() in src/MeshWeaver.Data/Workspace.cs:line 874
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at MeshWeaver.Data.Workspace.GetStream[TReduced](WorkspaceReference`1 reference, Func`2 configuration) in src/MeshWeaver.Data/Workspace.cs:line 881
at MeshWeaver.Mesh.MeshNodeStreamHandle.AcquireStream() in src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs:line 311
at MeshWeaver.Mesh.MeshNodeStreamHandle.Subscribe(IObserver`1 observer) in src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs:line 406
It also shows up in CI: MeshWeaver.Plugins job Module bundles (floor) / Module bundle (MeshWeaver.AI), run 33523800964 / job 99911922886, captured inside an unrelated assertion's
log dump.
Mechanism
ContentTypeRegistrationSweep.StartAsyncwalks every static node definition carrying a
HubConfigurationand callsProbeRegister, which builds a probe hub with
AsTransientNodeProbe(startDataSources: false)and disposes it immediately. Registration is a
side effect of building the configuration — nothing is meant to run.- For an Activity-shaped NodeType the configuration includes
KernelContainer's
WithInitialization(hub => … StartActivityControlPlane(hub)), which runs on the
InitializeHubRequestturn and callshub.WatchControlPlane(...)→
SubscribeWithReEstablish(() => workspace.GetMeshNodeStream() …). - The probe has no data sources, so
ReduceManager.ReduceStreamfinds no matching reducer and
returnsnull.Workspace.ReduceLocalStreamturns that into a bare
InvalidOperationException("Failed to create stream")— no reference, no owner, no inner cause. SubscribeWithReEstablish's classifier (ActivityControlPlaneExtensions.cs) tests for
HubDisposingExceptionnaming this address (IsOwnHubDisposing), a deserialization
MeshNodeStreamException(IsPoisonedContent) and a routingNotFound
(IsOwnNodeGone). A bareInvalidOperationExceptionmatches none of them, so it falls
through to "everything else = genuinely transient":LogError+ a 1 s re-establish timer.
That fourth branch exists for a hub that is alive and briefly unreachable. Here the hub is a
throwaway probe that was never going to have a stream, so "re-establishing" is a promise about a
state that cannot improve.
Impact
- Error-level log noise at every portal boot, once per swept Activity-shaped NodeType.
Informationand above ship to Loki;Erroris what operators alert on. This is the same
"routine lifecycle reported as a prod ERROR" drip theIsOwnHubDisposingbranch was written to
remove (see its own remarks) — the guard is present and simply cannot classify this path. - A brief re-establish timer armed on a hub that is already being disposed. This is A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991's
TimerQueue-root shape. It does not leak today:ProbeRegisterdisposes the probe, and
MessageHub.RegisterForDisposaldisposes a late registrant immediately, which cancels the
pending schedule. But the ordering is incidental —Establish()arms the timer before
WatchControlPlanereturns the disposable the caller registers — so the safety here is timing,
not design.
What the fix is (and is not)
Not a level change and not a catch. Two candidates, both root-cause:
AsTransientNodeProbe(startDataSources: false)should mean the control plane does not start.
A probe built explicitly without data sources should not install a watcher that requires one —
this is the narrower and more honest fix.ReduceLocalStreammust not throw a causelessInvalidOperationException. "Failed to create
stream" names no reference, no owner and no reason, which is precisely why three deliberate
classifiers all miss it. Whatever it throws should carry the reference and the owner, and the
no-data-sources case should be distinguishable from a real failure.
Either way the classifier keeps its meaning: transient means the node is alive and will come back.
Not in scope
Found while restoring #841's Warning trace to ExecuteScript's up-front refusal (the noise appears
in that test's captured-log dump). Deliberately not fixed there — a re-establish that recovers
from a state that cannot improve is a design question about the probe, not a line to patch inside an
unrelated change.
Activity
test(probe): take the sweep-probe signal from DisposalCompleted, not …