Skip to content

ActivityControlPlane watcher reports the boot-time content-type probe as a transient ERROR and arms a re-establish #2990

Description

@rbuergi

Symptom

Every mesh start writes one Error-level line per static NodeType whose HubConfiguration
installs an Activity Control Plane:

Error MeshWeaver.Kernel.Hub.KernelContainer: ActivityControlPlane subscription faulted on
content-type-registration/65936dcbcfa348bd921eb49885dea848 — re-establishing

The address is the giveaway: content-type-registration/{guid} is minted only by
ContentTypeRegistration.ProbeRegister (src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs:70).
Nothing is actually wrong with the mesh — the line reports the boot-time content-type sweep against
itself.

Evidence

Reproduced locally on origin/main (24254ad65), captured off the mesh's own ILoggerFactory in
Memex.Portal.Shared.Test (an ILoggerProvider registered through MeshBuilder.ConfigureServices,
so this is what a deployment's log sink receives, not test output). The exception attached to the
record:

System.InvalidOperationException: Failed to create stream
   at MeshWeaver.Data.Workspace.ReduceLocalStream[TReduced](WorkspaceReference`1 reference, Func`2 configuration) in src/MeshWeaver.Data/Workspace.cs:line 925
   at MeshWeaver.Data.Workspace.<>c__DisplayClass33_0`1.<GetStream>b__1() in src/MeshWeaver.Data/Workspace.cs:line 874
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at MeshWeaver.Data.Workspace.GetStream[TReduced](WorkspaceReference`1 reference, Func`2 configuration) in src/MeshWeaver.Data/Workspace.cs:line 881
   at MeshWeaver.Mesh.MeshNodeStreamHandle.AcquireStream() in src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs:line 311
   at MeshWeaver.Mesh.MeshNodeStreamHandle.Subscribe(IObserver`1 observer) in src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs:line 406

It also shows up in CI: MeshWeaver.Plugins job Module bundles (floor) / Module bundle (MeshWeaver.AI), run 33523800964 / job 99911922886, captured inside an unrelated assertion's
log dump.

Mechanism

  1. ContentTypeRegistrationSweep.StartAsync walks every static node definition carrying a
    HubConfiguration and calls ProbeRegister, which builds a probe hub with
    AsTransientNodeProbe(startDataSources: false) and disposes it immediately. Registration is a
    side effect of building the configuration — nothing is meant to run.
  2. For an Activity-shaped NodeType the configuration includes KernelContainer's
    WithInitialization(hub => … StartActivityControlPlane(hub)), which runs on the
    InitializeHubRequest turn and calls hub.WatchControlPlane(...) →
    SubscribeWithReEstablish(() => workspace.GetMeshNodeStream() …).
  3. The probe has no data sources, so ReduceManager.ReduceStream finds no matching reducer and
    returns null. Workspace.ReduceLocalStream turns that into a bare
    InvalidOperationException("Failed to create stream") — no reference, no owner, no inner cause.
  4. SubscribeWithReEstablish's classifier (ActivityControlPlaneExtensions.cs) tests for
    HubDisposingException naming this address (IsOwnHubDisposing), a deserialization
    MeshNodeStreamException (IsPoisonedContent) and a routing NotFound
    (IsOwnNodeGone). A bare InvalidOperationException matches none of them, so it falls
    through to "everything else = genuinely transient": LogError + a 1 s re-establish timer.

That fourth branch exists for a hub that is alive and briefly unreachable. Here the hub is a
throwaway probe that was never going to have a stream, so "re-establishing" is a promise about a
state that cannot improve.

Impact

  • Error-level log noise at every portal boot, once per swept Activity-shaped NodeType.
    Information and above ship to Loki; Error is what operators alert on. This is the same
    "routine lifecycle reported as a prod ERROR" drip the IsOwnHubDisposing branch was written to
    remove (see its own remarks) — the guard is present and simply cannot classify this path.
  • A brief re-establish timer armed on a hub that is already being disposed. This is A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991's
    TimerQueue-root shape. It does not leak today: ProbeRegister disposes the probe, and
    MessageHub.RegisterForDisposal disposes a late registrant immediately, which cancels the
    pending schedule. But the ordering is incidental — Establish() arms the timer before
    WatchControlPlane returns the disposable the caller registers — so the safety here is timing,
    not design.

What the fix is (and is not)

Not a level change and not a catch. Two candidates, both root-cause:

  1. AsTransientNodeProbe(startDataSources: false) should mean the control plane does not start.
    A probe built explicitly without data sources should not install a watcher that requires one —
    this is the narrower and more honest fix.
  2. ReduceLocalStream must not throw a causeless InvalidOperationException. "Failed to create
    stream" names no reference, no owner and no reason, which is precisely why three deliberate
    classifiers all miss it. Whatever it throws should carry the reference and the owner, and the
    no-data-sources case should be distinguishable from a real failure.

Either way the classifier keeps its meaning: transient means the node is alive and will come back.

Not in scope

Found while restoring #841's Warning trace to ExecuteScript's up-front refusal (the noise appears
in that test's captured-log dump). Deliberately not fixed there — a re-establish that recovers
from a state that cannot improve is a design question about the probe, not a line to patch inside an
unrelated change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions