Skip to content

Break the mutual GetHashCode recursion between StreamConfiguration<T> and SynchronizationStream<T> #2306

Closed

Description

@meshweaver-cloud

What is failing

MeshWeaver.Blazor.Infrastructure.UserIdentityCache.As<User>() is handed a value whose runtime type is MeshWeaver.Graph.Configuration.NodeTypeDefinition, and its type guard rejects it. The captured stack shows the read path then enters an unbounded hash cycle:

SynchronizationStream<T>.GetHashCode()
  -> ObjectEqualityComparer<T>.GetHashCode(T)
     -> StreamConfiguration<T>.GetHashCode()
        -> SynchronizationStream<T>.GetHashCode()
           -> ... (repeats to truncation)

StreamConfiguration<T>.GetHashCode() and SynchronizationStream<T>.GetHashCode() each hash the other, with no depth guard, cycle check or memoized hash.

Probable cause

High confidence on the recursion — it is not inferred, it is the literal repeating frame sequence in the sample, and it is structural rather than data-dependent: any code path that hashes either type (or puts one in a dictionary/set, which is what ObjectEqualityComparer in the trace means) can walk it forever. Sibling incident 786f05aa5de0c166 captured the same class of cycle one layer down (InstanceCollection.GetHashCode <-> EntityStore.GetHashCode) and there it terminated in an explicit Stack overflow. — i.e. a process kill, which .NET cannot catch.

Low confidence on the trigger — why a NodeTypeDefinition occupies a slot the cache expects to hold a User is not established by this evidence. Do not invent a cache-key collision from this ticket; that thread is already being pulled on separately (see below).

Likely duplicate — read before starting work

This is almost certainly the same defect as #2163 (incident 6e643370636ed34a), which describes this exact two-type cycle. The only difference is which frame of the cycle happened to be on top when the burst was sampled — SynchronizationStream<T> here, StreamConfiguration<T> there. Because the fingerprint incorporates topFrame, one infinite recursion keeps minting new incidents, one per frame that lands on top. Related: #2171 (786f05aa5de0c166, the EntityStore variant) and a95282b4d12cfd2b (already Filed, very high volume, no stack captured — that one tracks the cache type-confusion trigger).

A human can likely close this as a duplicate of #2163 in seconds. It is worth keeping open only if you want the SynchronizationStream<T> entry point covered by a regression test explicitly.

Impact

This fingerprint on its own is a single event on a single pod — negligible in isolation. The reason to act is the family, not this row: the untriggered form is a StackOverflowException, which is an uncatchable process termination and a pod restart, and the trigger condition it rides on is running at very high volume on the same deployment. Severity is high per event, low in frequency so far, and will rise if the trigger worsens.

Where to look

Framework source is not mirrored into the mesh, so these are type/method references rather than links:

  1. MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() — make it hash only its own fields, or cache the hash.
  2. MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() — make the delegation one-way; two types must not hash through each other.
  3. Consider whether either type should be hashable at all — a mutable stream used as a dictionary key is a smell independent of the recursion.
  4. MeshWeaver.Blazor.Infrastructure.UserIdentityCache — validate type at write time so a non-User is refused on insert rather than detected on read.

Evidence

Fingerprint 557ddfbfbfbc70e8
Category MeshWeaver.Blazor.Infrastructure.UserIdentityCache
Severity Error
Top frame MeshWeaver.Data.Serialization.SynchronizationStream1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()`
Namespace memex
Pods memex-portal-deployment-86779d495b-ksxqq
Occurrences 1
First seen 2026-08-26 02:35:48Z
Last seen 2026-08-26 02:35:48Z
Recent log lines
2026-08-26 02:35:48Z memex-portal-deployment-86779d495b-ksxqq fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
      As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral…[truncated]

Opened automatically from Admin/_LogIncident/557ddfbfbfbc70e8. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. rbuergi commented on Aug 26, 2026

    @rbuergi
    Contributor

    Already fixed on main — this was auto-filed from production logs that predate the fix.

    Closed by #2239 (merge 4585c5188), which landed ~02:00 UTC; this issue was opened at 02:38 from a log capture on a pod still running the older image.

    Verified on origin/main:

    SynchronizationStream.cs:1637  public virtual bool Equals(SynchronizationStream<TStream>? other) => ReferenceEquals(this, other);
    SynchronizationStream.cs:1644  public override int GetHashCode() => RuntimeHelpers.GetHashCode(this);
    

    Root cause, since this issue's stack shows only half of it: both types are records, so the compiler synthesized structural equality over every field — and the field graph contains a cycle (StreamConfiguration<T>'s primary-ctor property Stream is a back-pointer to its owning stream). The fix gives the live/identity object reference semantics, which is correct independently of the cycle: a SynchronizationStream owns a ReplaySubject, mutable state and subscriptions — two distinct streams are never "equal".

    🚨 The entry point is the part this issue's stack hides: SynchronizationStream's synthesized hash also walked its private ChangeItem<TStream>? current, so hashing a stream descended SynchronizationStream → ChangeItem → EntityStore → InstanceCollection. Both recorded production stacks were entered by hashing a STREAM, which is why reference identity removes the entry point rather than patching one arm. The container hashes (keys+count, never the arbitrary values) were fixed too, as belt-and-braces for any future ISynchronizationStream implementation.

    The As<User> line in this report is a separate defect — a NodeType declaration filed under the very type it declares — fixed by #2245 (0a0560501).

    ⚠️ Expect more re-files of both until CD rolls the new image. The red-log pipeline files from live pod logs, so a fixed defect keeps arriving until every replica is on a build that carries the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions