Repository navigation
ThreadAgentIntegrationTest blocks shard 4: a per-node hub's GetDataRequest handler runs and never answers #1362
Description
ThreadAgentIntegrationTest blocks PRs on shard 4: a per-node hub's GetDataRequest handler runs and never answers.
Not a flake in the test — a read that is owed a reply and never gets one
MeshWeaver.AI.Test.ThreadAgentIntegrationTest.FullFlow_CreateThread_SendMessage_StreamResponse_SaveReply
fails intermittently on shard 4 with:
System.TimeoutException : GetMeshNode('ACME/ProductLaunch') timed out after 60.0s (budget 60s)
— the owning per-node hub never answered the GetDataRequest. This is NOT 'node not found'.
Target: NO LOCAL HUB at 'ACME/ProductLaunch' — it never activated here …
at ThreadAgentIntegrationTest.cs:190 (StartChatWithOwnAgent → ReadNode(AgentContextPath)).
Reproduced on main, not only on PRs — dabfecba9 (merge of #1315), run
31611972654, shard 4.
It has since blocked at least PR #1343, whose diff cannot reach it.
The trace contradicts the message: the hub DID activate, and the handler DID run
The stale-callback dump is unambiguous:
GetDataRequest → ACME/ProductLaunch
→ ROUTED onTarget=False state=Forwarded@mesh/…(+89ms)
→ RECEIVED runLevel=Starting@ACME/ProductLaunch(+6690ms)
→ ENQUEUED@ACME/ProductLaunch(+6690ms)
→ ROUTED onTarget=True state=Submitted@ACME/ProductLaunch(+7247ms)
→ HANDLER_ENTER@ACME/ProductLaunch(+7247ms)
→ HANDLER_EXIT state=Processed@ACME/ProductLaunch(+7272ms)
⇒ a handler was entered and no reply, completion or fault has been recorded since
So NO LOCAL HUB is what the final diagnostic sees 53 s later, after the hub has gone —
it is not what happened. The hub activated at +6.7 s, the handler entered and exited
"Processed" at +7.27 s, and then nothing for the remaining 53 s.
DataExtensions.HandleGetDataRequest is the shape that permits this: it subscribes
GetDataResponseObservable(...), posts the response on emission, and returns
request.Processed(). It has a .Catch for faults — but no terminal answer for "the stream
never emits at all". GetDataFromWorkspaceCore returns workspace.GetStream(reference, x => x.ReturnNullWhenNotPresent()), so an absent node is supposed to surface as a null
emission; here the workspace stream produced no initial state whatsoever.
Very likely upstream cause, worth confirming before fixing
Immediately before the handler, on the same hub:
[Warning] MeshWeaver.Graph.MeshNodeTypeSource — MeshNodeTypeSource[ACME/ProductLaunch]:
content discriminator '$type': 'Project' is not a registered type on the owning hub —
content stays an untyped JsonElement (renders empty, not editable).
ACME/ProductLaunch is nodeType: ACME/Project, a dynamic NodeType compiled at runtime
(samples/Graph/Data/ACME/Project/Source/). Under shard-4 contention the instance hub appears
to activate before its NodeType's assembly is bound, the content cannot be typed, and the data
source never reaches an initial state — so the read is owed a reply that never comes. This is
the class AGENTS.md describes as parking "every instance hub for the full 60 s activation
budget".
Two separable defects
- The read.
HandleGetDataRequestcan sit forever on a stream that never emits. Per
A gate that can never open must fail what it holds; and delete the mesh-op budget that was never applied #1298's rule — a gate that can never open answers now; it does not sit on a timeout — a
one-shot read whose workspace has no initial state for the reference should answer, not
hang. Today the only bound is the caller's, 60 s away, and the failure it produces names
the wrong thing ("NO LOCAL HUB"). - The activation. Why the per-node hub reaches
Startedwith its NodeType's content type
unregistered, and why nothing re-enriches it afterwards. Related to, but not covered by,
A successful recompile now rebinds the instances — the healthy binding was the one nobody watched #1343 (WithStaleAssemblySelfHealarms only when aLatestAssemblyPathwas bound, which is
precisely not this case) and to theWithOverlaySelfHealdegraded branches.
Filing rather than half-fixing: (1) is a small, well-scoped change, but choosing its terminal
answer needs a call about read semantics, and (2) sits in the compile/activation area under
active work. Neither belongs inside an unrelated PR.
Activity
fix: only claim "the owner is shutting down" when it is TRUE — gate t…
test: prove the consumer-visible outcome — a read whose owner is recy…
Merge pull request #1364 from Systemorph/fix/1362-silent-read-nack
fix(#2727): a read that faults on a CLOSED DI SCOPE is NACKed, not an…
fix(#2778): deliver the teardown NACK through the first ancestor that…