Repository navigation
UserIdentityCache type confusion is not Blazor-specific: the AspNetCore Portal host caches a NodeTypeDefinition under a User key #2358
Description
What is failing
The identity cache in the AspNetCore Portal hosting layer (MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache) holds a MeshWeaver.Graph.Configuration.NodeTypeDefinition in a slot it expects to hold a User. Its As<User>() type guard rejects the value on read and logs at fail, returning nothing — so the identity lookup misses and every caller that needed a resolved User falls through.
Probable cause
High confidence that this is the same defect as the known MeshWeaver.Blazor.Infrastructure.UserIdentityCache family — and medium-to-high confidence that the existing fixes will NOT cover it. That is the reason this ticket exists.
The message is byte-identical to the Blazor-layer family (same expected type, same actual type, same assembly), but the log category is a different UserIdentityCache — one under MeshWeaver.Hosting.AspNetCore.Portal rather than MeshWeaver.Blazor.Infrastructure. Two conclusions follow:
- Either the type-confusion originates below both shells, in whatever populates identity from the graph, and both caches merely observe it; or there are two
UserIdentityCacheimplementations carrying the same bug independently. Either way, a fix scoped to the Blazor assembly (as proposed on StreamConfiguration<T>.GetHashCode recurses through SynchronizationStream<T>, corrupting UserIdentityCache lookups #2163 and the sibling drafts) leaves this path broken. - Per "Choose your portal: Blazor, the new Next shell, or both", a Next-only portal carries no Blazor at all. This sighting is on a different deployment (
memex-cloud, notmemex), which is consistent with the non-Blazor hosting path now being exercised there. Worth confirming whethermemex-cloudis a Next-only or both-shells deployment — that single fact tells you whether the bug is shared-below or duplicated-per-shell.
What is NOT established: the writer that puts a NodeTypeDefinition into a user-keyed slot. No stack trace was captured for this burst, and I did not find the framework source mirrored in the mesh, so I cannot name the offending call site. Do not carry over the "cache-key collision" root cause asserted in the draft on a95282b4d12cfd2b — that was inferred, not evidenced.
Also note: unlike the siblings, this burst carries no GetHashCode recursion and no stack overflow. That is a useful negative — it shows the cache type-confusion occurs independently of the StreamConfiguration<T> / SynchronizationStream<T> hash cycle, so the two are genuinely separate defects rather than one symptom.
Impact
This fingerprint on its own is negligible: a couple of events, one pod, a two-second window, no crash. Do not size the problem from this row. Its value is as a signal — the sibling fingerprint a95282b4d12cfd2b has logged over 667,000 occurrences across roughly 190 pods since 2026-08-21 and is still firing, and the crash-variant 786f05aa5de0c166 terminates pods outright with an uncatchable StackOverflowException. This ticket is the first sighting of that same condition on the AspNetCore Portal host and on the memex-cloud deployment, i.e. evidence of spread into a code path the current fixes do not touch.
Related — already filed
Same symptom, different layer or different top frame. This is not a straight duplicate of any of them, because the category differs:
a95282b4d12cfd2b— Blazor layer, no stack, very high volume. Tracks the type-confusion trigger. Filed.- StreamConfiguration<T>.GetHashCode recurses through SynchronizationStream<T>, corrupting UserIdentityCache lookups #2163 (
6e643370636ed34a) — Blazor layer,StreamConfiguration<T>.GetHashCodeon top. - Break the mutual GetHashCode recursion between StreamConfiguration<T> and SynchronizationStream<T> #2306 (
557ddfbfbfbc70e8) — Blazor layer,SynchronizationStream<T>.GetHashCodeon top; noted as a likely duplicate of StreamConfiguration<T>.GetHashCode recurses through SynchronizationStream<T>, corrupting UserIdentityCache lookups #2163. - InstanceCollection.GetHashCode recurses to stack overflow when UserIdentityCache holds a NodeTypeDefinition #2171 (
786f05aa5de0c166) — Blazor layer,InstanceCollection/EntityStorehash cycle, ends in stack overflow.
A fifth ticket for one symptom is a lot. If an engineer would rather fold this into #2163, that is reasonable — but please first confirm the fix there actually reaches MeshWeaver.Hosting.AspNetCore.Portal, because on the evidence it does not.
Worth noting separately: because the fingerprint incorporates topFrame, one unbounded recursion keeps minting a fresh incident per frame that happens to land on top. That fingerprinting behaviour is itself worth a look — it is what turned one bug into this pile.
Where to look
MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache— the log site here. Confirm whether it is a distinct implementation from the Blazor one or a shared type logging under a host-specific category.- Whatever writes into it — validate the type at insert time and refuse a non-
User, so this is caught at the writer instead of surfacing as a silent read miss. This is the one change that fixes the class of bug rather than an instance of it. - The shared identity-resolution path below both shells — if a
NodeTypeDefinitioncan be returned where aUseris expected, that conversion is the real defect and both caches are downstream of it.
Framework source does not appear to be mirrored into the mesh, so these are type references rather than links.
Evidence
| Fingerprint | 75d8e54a0f563fa1 |
| Category | MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-788cffc666-swfdl, memex-portal-deployment-788cffc666-xbmfs |
| Occurrences | 139 |
| First seen | 2026-08-26 10:08:32Z |
| Last seen | 2026-08-26 10:12:50Z |
Recent log lines
2026-08-26 10:10:27Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:10:33Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:10:56Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:10:56Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:11:05Z memex-portal-deployment-788cffc666-xbmfs fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:12:50Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:12:50Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:12:50Z memex-portal-deployment-788cffc666-xbmfs fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:12:50Z memex-portal-deployment-788cffc666-xbmfs fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
2026-08-26 10:12:50Z memex-portal-deployment-788cffc666-swfdl fail: MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCache[0]
As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
Opened automatically from Admin/_LogIncident/75d8e54a0f563fa1. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 385224 more since the last update (385363 total).
memex-portal-deployment-788cffc666-swfdl,memex-portal-deployment-788cffc666-xbmfs,memex-portal-deployment-788cffc666-46jqr,memex-portal-deployment-5d4bfc98b7-ff447,memex-portal-deployment-5d4bfc98b7-5r56d,memex-portal-deployment-788cffc666-kh7sl,memex-portal-deployment-788cffc666-9zhzw,memex-portal-deployment-64cc8b9b6d-z9w44,memex-portal-deployment-64cc8b9b6d-rk5gw,memex-portal-deployment-64cc8b9b6d-q264s,memex-portal-deployment-64cc8b9b6d-hr5hh,memex-portal-deployment-6c954fd5bf-czv78,memex-portal-deployment-54cc8d6b4d-qsxvp,memex-portal-deployment-6c954fd5bf-fphtz,memex-portal-deployment-6c954fd5bf-7b4xq,memex-portal-deployment-6c954fd5bf-b5q52,memex-portal-deployment-54cc8d6b4d-cjnqn,memex-portal-deployment-6c954fd5bf-k6q74,memex-portal-deployment-6c954fd5bf-k525s,memex-portal-deployment-6c954fd5bf-h9lnk,memex-portal-deployment-6c954fd5bf-kmhpv,memex-portal-deployment-6c954fd5bf-5tk8b,memex-portal-deployment-565c5d6b97-zwvjs,memex-portal-deployment-565c5d6b97-wp8g7,memex-portal-deployment-7f6d66995c-dbzj8,memex-portal-deployment-7f6d66995c-r8zmq,memex-portal-deployment-7f6d66995c-tvd9l,memex-portal-deployment-d999f9888-sfpph,memex-portal-deployment-d999f9888-f86ts,memex-portal-deployment-748c6d4456-29chg,memex-portal-deployment-748c6d4456-bn2gh,memex-portal-deployment-58b5fbf775-fjbvn,memex-portal-deployment-58b5fbf775-pvbmd,memex-portal-deployment-59ddc494b7-frdz9,memex-portal-deployment-59ddc494b7-r6gqg,memex-portal-deployment-85bd9f5d85-6th26,memex-portal-deployment-85bd9f5d85-5fcnk,memex-portal-deployment-5998589dbd-tzx4w,memex-portal-deployment-5998589dbd-9v8mj,memex-portal-deployment-5678bbc9c8-pl7dq,memex-portal-deployment-5678bbc9c8-4rvjb,memex-portal-deployment-dbc8997cd-q2n4s,memex-portal-deployment-dbc8997cd-w55hb,memex-portal-deployment-6c74bf4c66-88jg8,memex-portal-deployment-6c74bf4c66-5xvlt,memex-portal-deployment-84765b45c8-f4j9m,memex-portal-deployment-84765b45c8-fjsv2,memex-portal-deployment-fbfc4b45-mqzbp,memex-portal-deployment-fbfc4b45-w4vfr,memex-portal-deployment-856d4474c-hr5rd,memex-portal-deployment-856d4474c-4rzkz,memex-portal-deployment-7858d9f8cc-h6gp2,memex-portal-deployment-7858d9f8cc-r2v5v,memex-portal-deployment-7d8fd6f84b-s7sr8,memex-portal-deployment-7d8fd6f84b-47cmq,memex-portal-deployment-7d8fd6f84b-mjp87,memex-portal-deployment-7d8fd6f84b-slmzp,memex-portal-deployment-7d8fd6f84b-25m2m,memex-portal-deployment-7d8fd6f84b-tq9bw,memex-portal-deployment-7657b88b9-fc4rq,memex-portal-deployment-7657b88b9-qwcxz,memex-portal-deployment-69f86967d8-9xxb7,memex-portal-deployment-69f86967d8-vmj29,memex-portal-deployment-69f86967d8-54p5q,memex-portal-deployment-69f86967d8-bglpq,memex-portal-deployment-69f86967d8-kdnlb,memex-portal-deployment-69f86967d8-9f8nx,memex-portal-deployment-6f76b9c79b-4wtcx,memex-portal-deployment-6f76b9c79b-vnpw5,memex-portal-deployment-6f76b9c79b-9spms,memex-portal-deployment-6f76b9c79b-gx2fc,memex-portal-deployment-865476c896-vxsr4,memex-portal-deployment-865476c896-6bv8p,memex-portal-deployment-865476c896-d64rf,memex-portal-deployment-865476c896-xbbsw,memex-portal-deployment-865476c896-strcr,memex-portal-deployment-865476c896-ssghx,memex-portal-deployment-5f9479f848-ffrpw,memex-portal-deployment-5f9479f848-hmmbn,memex-portal-deployment-5f9479f848-fsvjr,memex-portal-deployment-7ccdd994cf-sbrjg,memex-portal-deployment-7ccdd994cf-wwwsj,memex-portal-deployment-768b8dff99-cn8nn,memex-portal-deployment-768b8dff99-h48bj,memex-portal-deployment-7c6b48957d-xqzdq,memex-portal-deployment-7c6b48957d-c58hl,memex-portal-deployment-7c6b48957d-xml97,memex-portal-deployment-7c6b48957d-n5vkr,memex-portal-deployment-7c6b48957d-9zc78,memex-portal-deployment-7c6b48957d-gkzfs,memex-portal-deployment-7c6b48957d-rpkhr,memex-portal-deployment-7c6b48957d-xl99h,memex-portal-deployment-7c6b48957d-fllqp,memex-portal-deployment-78cd7b4f8f-g85gw,memex-portal-deployment-767d8d6684-gdl2g,memex-portal-deployment-767d8d6684-s6kcf,memex-portal-deployment-78cd7b4f8f-npfqs,memex-portal-deployment-767d8d6684-n8pfw,memex-portal-deployment-767d8d6684-vrc78,memex-portal-deployment-6d55467d9d-46jfk,memex-portal-deployment-6d55467d9d-x6xr2,memex-portal-deployment-c4499cbcc-fq9xr,memex-portal-deployment-c4499cbcc-qwtl2,memex-portal-deployment-67c44ccd96-z9g2b,memex-portal-deployment-cfc9f8448-d4txb,memex-portal-deployment-67c44ccd96-f7dhr,memex-portal-deployment-cfc9f8448-vjlb9,memex-portal-deployment-8d856d8c-nbbkv,memex-portal-deployment-8d856d8c-txmhq,memex-portal-deployment-554545c8cc-xw7xn,memex-portal-deployment-554545c8cc-m7kw2,memex-portal-deployment-56699545bf-jzbvt,memex-portal-deployment-56699545bf-pk948,memex-portal-deployment-76c67b6b45-5n2zg,memex-portal-deployment-76c67b6b45-v4hdf,memex-portal-deployment-66cfbf759-svxh9,memex-portal-deployment-66cfbf759-cnwnb,memex-portal-deployment-66cfbf759-v4grm,memex-portal-deployment-66cfbf759-gmgl4,memex-portal-deployment-66cfbf759-ks9hlRecent log lines
rbuergi commentedon Sep 18, 2026
Closing — fixed at the writer in current
origin/main, and quiet 19 daysDormancy. The incident record's newest occurrence is
Last seen: 2026-08-30 11:55:23Z(the bot's own reopen comment above). Measured 2026-09-18T05:52Z that is 19 days quiet, and log ingest is live (incidents on this mesh updated 2026-09-18T05:42Z), so the silence is the fault having stopped — not an ingest gap. The portals have not rolled in that window either, so it stopped while the same image kept running.The defect this issue asked for is in
origin/main. This ticket's own "Where to look" asked for the class fix — "validate the type at insert time … the one change that fixes the class of bug rather than an instance of it" — and both halves of it exist, checked againstorigin/mainat229930bf2f29845323e0f350e73829b1bdb4928e:The
Userdeclaration no longer names itself as an instance ofUser, so it no longer answers the very query this host's cache is built on —UserIdentityCache.cs:234,MeshQueryRequest.FromQuery("nodeType:User"). That is the source of theNodeTypeDefinitionthis ticket reported, and it is upstream of both shells, which is exactly the "shared-below" branch the ticket said had to be settled.NodeTypeDeclarationSelfTypingValidatorrefuses the collision at the write boundary for every declaration present or future (its XML doc names this issue by number as the reason it is a validator rather than a retype), andSelfTypedDeclarationDurableRepairheals rows persisted with the collision before the guard existed. Registered, not merely present.Also a duplicate. The same fault is tracked by the already-closed family #2160 / #2161 / #2162 / #2425 / #2506 — this ticket exists only because the log category differed (
MeshWeaver.Hosting.AspNetCore.Portal.UserIdentityCachevs the Blazor one). Fixing the writer covers every category, so the distinction the ticket was opened on no longer separates anything. The lowest-numbered member of the family is #2160.Not closed here: #2171 (the
InstanceCollection.GetHashCodestack-overflow variant) stays open on its own merits — this ticket's body records the useful negative that the hash recursion is a separate defect.Closing as redundant. If the fault recurs, the recurrence bot reopens this issue with fresh evidence — that is the designed behaviour and it is why closing a dormant incident costs nothing: a real recurrence comes back carrying a
Last seenthat postdates this close, which is evidence that means what it says.