Skip to content

fix(mesh): ContentAs<T> recovers same-named Content from a foreign dynamically-compiled assembly — cures the empty deployed dashboards - #218

Merged
rbuergi merged 5 commits into
mainfrom
fix/contentas-foreign-assembly-recovery
Jul 3, 2026
Merged

rbuergi merged 5 commits into
mainfrom
fix/contentas-foreign-assembly-recovery

Conversation

@rbuergi

@rbuergi rbuergi commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes Systemorph/agentic-pensions#12 — on the deployed portal, every virtual-data-source collection whose loader filters on content fields stayed empty forever (AgenticPension/Dashboard EntryNode = 0 of 200, /AgenticPension/ExtraktionsReview stuck on "Noch keine Bilanzwerte geladen"), while metadata-only collections populated.

Root cause (NOT the suspected stream drop)

The issue's working theory (large snapshot silently dropped by the RouteStreamMessage Ignored() + idle-release race) is falsified: a diagnostic NodeType run in-grain on client-i showed the 200-item Query<MeshNode> result arriving completely — but Content.GetType() on every row was BalanceSheetEntry @ DynamicNode_AgenticPension_BalanceSheetEntry, and ContentAs<BalanceSheetEntry> returned null for every row, silently.

The chain:

  1. Serialize-time auto-registration (PolymorphicTypeInfoResolver.ComputeDerivedTypes → GetOrAddType) teaches the root registry a short-name $type → the owning NodeType assembly's CLR type, the first time such content is serialized at root level (persistence writes).
  2. The query layer (MeshQuery.Options = root hub options) then materializes every result row's Content as that CLR type.
  3. Consumer NodeType hubs compile their own copy of the shared record (@@-included sources) — same short name, different assembly — so ContentAs<T>'s default: branch (not T, not JsonElement) returned null, and the loaders' .Where(Content is not null …) filtered every row.

This is order/restart-dependent (the registry learns on the first root-level serialize), which is why A/B revert experiments misattributed it to the framework build diff, and why "small collections work" (their loaders never touch Content, or filter only on non-null) masqueraded as a size-dependent stream bug.

Fix (three layers — read-site recovery + the arming root-fix + observability)

  1. PolymorphicTypeInfoResolver never auto-registers collectible-assembly types (Assembly.IsCollectible guard). Dynamic node / kernel-script compilations produce a NEW CLR identity per recompile; adopting one into a long-lived hub's registry as a serialization side effect is what armed the outage — and it also pins the collectible assembly, defeating unload. The $type discriminator is still formatted and written; reads on non-owning hubs degrade to JsonElement. This kills the poisoning at its source.
  2. MeshNodeContentExtensions.ContentAs<T>: the default: branch now recovers any foreign materialized shape by a JSON round-trip (SerializeToElement(content, content.GetType(), options) → Deserialize<T>), symmetric with the existing degraded-JsonElement recovery — serialized AS the concrete runtime type so it routes through the guarded resolver, not ObjectPolymorphicConverter's unguarded GetOrAddType (per Copilot review). Recovery logs at Debug (healthy per-row path); unrecoverable shapes return null with an assembly-qualified error (dynamic types have no namespace — the bare name is identical on both sides of the mismatch).
  3. RouteStreamMessage's absent-sync-hub drop logs one Warning (StreamId + message type + hub) instead of a fully silent Ignored() — the issue's explicit observability ask.

This PR reconciles two independent investigations that converged on the same RCA (see the coordination trail in the commits).

Tests

  • TypeRegistryTest collectible-assembly facts (against a REAL emitted collectible assembly): serializing never poisons the registry.
  • ContentAsForeignAssemblyContentTest (red→green): the full pollute → materialize → consume chain through real hub options, incl. registry-survival of the consumer's own registrations.
  • MeshNodeContentAsTest: the plain ContentAs contract facts (typed / JsonElement / foreign-typed).
  • VirtualDataSourceContentTypeTest (Query.Test): the full virtual-data-source loader corridor.
  • Suites green: Messaging.Hub 81, Hosting.Test 61, Data.Test 274, Query.Test 343; solution Release -warnaserror clean.

Notes

  • MeshNodeStreamCache.DeserializeContent has the mirror gap (returns the node unchanged when Content is already foreign-typed) — benign today because the cache hub's registry only knows framework types; noted as follow-up.
  • Once this merges and the portals self-update, client-i's dashboards and the ExtraktionsReview table populate without any change to the customer's node code.

🤖 Generated with Claude Code

…namically-compiled assembly — cures the empty deployed dashboards

ROOT CAUSE of agentic-pensions#12 ("virtual-data-source workspaces stay empty
after the 2026-07-02 deploy") — proven live on atioz with an in-grain
diagnostic NodeType, and NOT the suspected stream-message drop: the 200-item
Query<MeshNode> arrives completely in the grain workspace, but every row's
Content is typed as the OWNING NodeType assembly's CLR type
(BalanceSheetEntry @ DynamicNode_AgenticPension_BalanceSheetEntry).

Chain: (1) serialize-time auto-registration (PolymorphicTypeInfoResolver.
ComputeDerivedTypes -> GetOrAddType) teaches the ROOT registry the short-name
$type -> owning assembly's type on the first root-level serialize of such
content; (2) the query layer (MeshQuery.Options = root hub options) then
materializes every result row's Content as that type; (3) consumer NodeType
hubs compile their OWN copy of the shared record (@@-included sources) - same
short name, different assembly - so ContentAs<T> hit the default: branch (not
T, not JsonElement) and silently returned null. Every loader that filters on
content fields dropped all rows -> permanently empty EntryNode/RevEntry ->
"Noch keine Bilanzwerte geladen", while metadata-only collections populated
(the false "size-dependent snapshot drop" signature). Order/restart-dependent
(registry learns on first write), which is why A/B revert experiments
misattributed it to the framework build diff.

Fix: the default: branch now recovers by a JSON round-trip
(SerializeToElement(content, content.GetType(), options) -> Deserialize<T>),
symmetric with the degraded-JsonElement recovery — cross-assembly identity
mismatch is the NORMAL shape for shared query results in a mesh of
dynamically compiled NodeTypes; the JSON shape is authoritative and the read
site owns typing (the documented ContentAs contract). Unrecoverable shapes
still return null, now logging the assembly-qualified ActualType.

Repro red->green: ContentAsForeignAssemblyContentTest (3 facts) reproduces
the full pollute -> materialize -> consume chain through real hub options.
Hosting.Test 56/56, ThreadViewModelJsonElementTest green, solution Release
-warnaserror clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a production issue where MeshNodeContentExtensions.ContentAs<T>() returned null for MeshNode.Content materialized as a same-named CLR type from a different dynamically-compiled NodeType assembly, causing content-filtered collections (dashboards) to remain empty after deployment.

Changes:

  • Extend ContentAs<T>() to attempt recovery of non-T, non-JsonElement typed content via a JSON round-trip.
  • Add a regression test that reproduces the root-hub materialization + consumer-hub ContentAs<T>() mismatch scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
test/MeshWeaver.Hosting.Test/ContentAsForeignAssemblyContentTest.cs Adds a regression test covering cross-assembly same-name typing recovery (and related shape-binding behavior).
src/MeshWeaver.Mesh.Contract/MeshNodeContentExtensions.cs Implements JSON round-trip recovery for foreign-typed Content in ContentAs<T>(), with additional logging.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/MeshWeaver.Mesh.Contract/MeshNodeContentExtensions.cs Outdated
Comment thread src/MeshWeaver.Mesh.Contract/MeshNodeContentExtensions.cs Outdated
…'s TypeRegistry — kill the registry poisoning that armed agentic-pensions#12

Completes the cure with the ARMING root-fix (reconciled from a parallel
investigation that reached the same RCA independently):

1. PolymorphicTypeInfoResolver never auto-registers a type from a COLLECTIBLE
   assembly (dynamic node / kernel-script compilations — every recompile is a
   NEW assembly with a NEW CLR identity for the "same" class). Adopting one
   into a long-lived hub's registry as a serialization side effect poisoned
   the hub for the rest of the process: every later $type resolution (e.g.
   mesh-query row deserialization) yielded THAT foreign/stale CLR type, and
   consumers holding their own compilation read Content as null. It also
   pinned the collectible assembly, defeating unload. The discriminator is
   still formatted and written; reads on non-owning hubs degrade to
   JsonElement, which ContentAs<T> recovers at the consumer.

2. ContentAs<T> recovery hardened per Copilot review: serialize AS the
   concrete runtime type (the object-declared path routes through
   ObjectPolymorphicConverter.GetTypeName's unguarded GetOrAddType and would
   defeat the resolver guard), recovery logged at Debug (per-row on a healthy
   path — Warning would storm at snapshot size), unrecoverable-shape error now
   carries the ASSEMBLY identity (dynamic types have no namespace — the bare
   name is identical on both sides of a cross-assembly mismatch).

3. RouteStreamMessage's absent-sync-hub drop now logs one Warning with
   StreamId/message type/hub (the silent Ignored() made data-sync drops
   unfindable in production — issue #12's explicit ask).

Tests: TypeRegistryTest collectible-assembly facts (real emitted collectible
assembly), MeshNodeContentAsTest (contract facts), ContentAsForeignAssembly-
ContentTest (real-hub-options pollute→materialize→consume chain),
VirtualDataSourceContentTypeTest (full loader corridor). Suites green:
Messaging.Hub 81, Hosting 61, Data 274, Query 343; solution Release
-warnaserror clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

799 tests  ±0   691 ✅ ±0   2m 40s ⏱️ +42s
 13 suites ±0   108 💤 ±0 
 13 files   ±0     0 ❌ ±0 

Results for commit f843470. ± Comparison against base commit 85dbdeb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

1 305 tests   - 97   1 304 ✅  - 97   5m 40s ⏱️ - 1m 7s
   14 suites ± 0       1 💤 ± 0 
   14 files   ± 0       0 ❌ ± 0 

Results for commit f843470. ± Comparison against base commit 85dbdeb.

This pull request removes 99 and adds 2 tests. Note that renamed tests count towards both.
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveAgents_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveAgents_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveModels_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_FromMeshHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_SpaceAndUserSet_SurfacesSpaceAgent_UserAgent_AndPlatform
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveModels_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveModels_FromMeshHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.CessionLayoutAreaTest ‑ BusinessRulesDoc_RelativeReference_ResolvesToMotorXL
MeshWeaver.Hosting.Monolith.Test.CessionLayoutAreaTest ‑ Cession_Trace_HubConfiguration
…
MeshWeaver.Messaging.Hub.Test.TypeRegistryTest ‑ SerializingCollectibleType_DoesNotPoisonRegistry_AndDegradesReadsToJsonElement
MeshWeaver.Query.Test.VirtualDataSourceContentTypeTest ‑ TypedForeignContent_StillPopulatesVirtualCollection

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

1 536 tests  +8   1 536 ✅ +8   3m 15s ⏱️ -2s
   12 suites ±0       0 💤 ±0 
   12 files   ±0       0 ❌ ±0 

Results for commit f843470. ± Comparison against base commit 85dbdeb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

   15 files  ±  0     15 suites  ±0   7m 29s ⏱️ -29s
1 174 tests  - 193  1 171 ✅  - 193  3 💤 ±0  0 ❌ ±0 
1 183 runs   - 223  1 180 ✅  - 223  3 💤 ±0  0 ❌ ±0 

Results for commit f843470. ± Comparison against base commit 85dbdeb.

This pull request removes 193 tests.
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_FutuReSubPaths_ShouldResolve(path: "FutuRe/Analysis", expectedPrefix: "FutuRe/Analysis", expectedRemainder: null)
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_FutuReSubPaths_ShouldResolve(path: "FutuRe/EuropeRe", expectedPrefix: "FutuRe/EuropeRe", expectedRemainder: null)
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_FutuReSubPaths_ShouldResolve(path: "FutuRe/EuropeRe/LineOfBusiness", expectedPrefix: "FutuRe/EuropeRe/LineOfBusiness", expectedRemainder: null)
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_FutuReSubPaths_ShouldResolve(path: "FutuRe/EuropeRe/Overview", expectedPrefix: "FutuRe/EuropeRe", expectedRemainder: "Overview")
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_FutuRe_ShouldNotReturnNull
MeshWeaver.Persistence.Test.ApplicationPageResolutionTest ‑ ResolvePathAsync_UnknownDeepPath_ShouldReturnNull
MeshWeaver.Persistence.Test.DataContextIntegrationTest ‑ GraphHub_InitializesWithConfiguration
MeshWeaver.Persistence.Test.DataContextIntegrationTest ‑ MeshNode_ChildrenAvailable_ViaPersistence
MeshWeaver.Persistence.Test.DataContextIntegrationTest ‑ Persistence_CanCreateNodeWithContent
MeshWeaver.Persistence.Test.DataContextIntegrationTest ‑ Persistence_CanUpdateNodeWithContent
…

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

   54 files  ±  0     54 suites  ±0   19m 6s ⏱️ -57s
4 814 tests  - 282  4 702 ✅  - 282  112 💤 ±0  0 ❌ ±0 
4 823 runs   - 312  4 711 ✅  - 312  112 💤 ±0  0 ❌ ±0 

Results for commit f843470. ± Comparison against base commit 85dbdeb.

This pull request removes 292 and adds 10 tests. Note that renamed tests count towards both.
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveAgents_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveAgents_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionPartitionedTest ‑ PartitionedPath_ObserveModels_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_FromMeshHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveAgents_SpaceAndUserSet_SurfacesSpaceAgent_UserAgent_AndPlatform
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveModels_FromHostedSubHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.AgentPickerProjectionTest ‑ ObserveModels_FromMeshHub_PopulatesCombobox
MeshWeaver.Hosting.Monolith.Test.CessionLayoutAreaTest ‑ BusinessRulesDoc_RelativeReference_ResolvesToMotorXL
MeshWeaver.Hosting.Monolith.Test.CessionLayoutAreaTest ‑ Cession_Trace_HubConfiguration
…
MeshWeaver.Hosting.Test.ContentAsForeignAssemblyContentTest ‑ ContentAs_DifferentlyNamedTypedContent_StaysNull_PreservingProbeDispatch
MeshWeaver.Hosting.Test.ContentAsForeignAssemblyContentTest ‑ ContentAs_ForeignRecovery_DoesNotBreakTheConsumersOwnRegistrations
MeshWeaver.Hosting.Test.ContentAsForeignAssemblyContentTest ‑ ContentAs_SameNamedTypeFromForeignAssembly_RecoversViaJsonRoundTrip
MeshWeaver.Hosting.Test.MeshNodeContentAsTest ‑ JsonElementContent_IsDeserialized
MeshWeaver.Hosting.Test.MeshNodeContentAsTest ‑ NullContent_ReturnsNull
MeshWeaver.Hosting.Test.MeshNodeContentAsTest ‑ TypedForeignContent_SameShape_IsRecovered
MeshWeaver.Hosting.Test.MeshNodeContentAsTest ‑ TypedSameClrContent_IsReturnedAsIs
MeshWeaver.Hosting.Test.MeshNodeContentAsTest ‑ UnrecoverableContent_ReturnsNull_WithoutThrowing
MeshWeaver.Messaging.Hub.Test.TypeRegistryTest ‑ SerializingCollectibleType_DoesNotPoisonRegistry_AndDegradesReadsToJsonElement
MeshWeaver.Query.Test.VirtualDataSourceContentTypeTest ‑ TypedForeignContent_StillPopulatesVirtualCollection

♻️ This comment has been updated with latest results.

rbuergi and others added 3 commits July 3, 2026 11:53
…es — preserve probe-dispatch

The first cut recovered ANY typed-but-not-T content by shape, which defeated
probe-dispatch call sites: the token validator's
`ContentAs<ApiTokenIndex>() ?? treat the node as the token itself` got a
half-populated ApiTokenIndex out of an ApiToken (both carry TokenHash), followed
the null TokenPath, and every API-token login failed "Token not found" — the 7
McpAccessControlTests failures on shard 0.

Cross-assembly identity means the SAME type name with a different CLR identity
(the same class compiled into two dynamic node assemblies). A differently-named
registered type is a legitimate "this is another domain type" answer and must
stay null; by-shape binding of differently-named content remains the degraded-
JsonElement path's job (untyped content only).

The DifferentlyNamedTargetType fact now pins the strict-null contract instead
of the by-shape binding it used to assert.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR #217 shipped Data/Skill/access.md without updating the two catalog
expectation lists. The miss never turned CI red because MeshWeaver.AI.Test hits
the 6-minute wall-clock cap on CI (exit=124) and a timed-out project's trx never
reaches the consolidated results — the failures are masked, the shard stays
green. The timeout-masking gap and the other two real AI.Test failures it hides
(McpNegativeOperationsTest.DeletedNode_EveryVerb_FailsCleanly,
CodeCellOutputCaptureTest.Progress_Of_Succeeded_Activity_Renders_Output_For_
Anonymous_VUser) are separate defects and stay open.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rbuergi

rbuergi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

CI shard 0 caught a real regression in the first cut of the recovery: all 7 McpAccessControlTests failed with "Token not found". Root cause: the by-shape recovery also fired for differently-named types, defeating probe-dispatch call sites — the token validator's ContentAs<ApiTokenIndex>() ?? treat node as the token itself got a half-populated ApiTokenIndex out of an ApiToken (both carry TokenHash) and followed a null TokenPath.

Fixed in 2787d1e: recovery is now gated on short-name equality — exactly the cross-assembly identity case it exists for. Differently-named typed content stays null (the probe contract); by-shape binding of differently-named content remains the degraded-JsonElement path's job. The DifferentlyNamedTargetType fact now pins the strict-null contract. Locally green: McpAccessControlTests 7/7, ContentAs suites 8/8, Release -warnaserror clean.

f843470 additionally catches the two skill-catalog expectation lists up with the /access skill (#217 omission). Note for a follow-up: that miss never turned CI red because MeshWeaver.AI.Test hits the 6-min wall-clock cap on CI (exit=124) and a timed-out project's failures never reach the consolidated trx — the run on main reads green while AI.Test actually fails inside (also hides McpNegativeOperationsTest.DeletedNode_EveryVerb_FailsCleanly and CodeCellOutputCaptureTest.Progress_Of_Succeeded_Activity_Renders_Output_For_Anonymous_VUser).

🤖 Generated with Claude Code

@rbuergi
rbuergi merged commit e093354 into main Jul 3, 2026
11 checks passed
@rbuergi
rbuergi deleted the fix/contentas-foreign-assembly-recovery branch August 5, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants