Repository navigation
SynchronizationStream<T>.GetHashCode recurses through StreamConfiguration<T> and will StackOverflow on any log-scope hashing #2174
Description
What is failing
MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() and MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() call each other in an unbounded cycle. Any code path that hashes one of these objects (typical trigger: it lands as a key/member of a Dictionary<,> or HashSet<>, where EqualityComparer<T>.Default — i.e. ObjectEqualityComparer<T> — invokes GetHashCode) recurses until the stack is exhausted.
Here it fired on the logger's structured-state path while RoutingGrain was formatting a routine NACK — the routing message itself ("NotFound DeliveryFailure ... no live subscriber") is expected behaviour and is not the bug; the bug is that a SynchronizationStream reference travelling through the log scope gets hashed.
Probable cause — high confidence
The stack is self-evident and repeats verbatim:
SynchronizationStream<T>.GetHashCode()
→ ObjectEqualityComparer<T>.GetHashCode(T)
→ StreamConfiguration<T>.GetHashCode()
→ SynchronizationStream<T>.GetHashCode()
→ …
That pattern is the classic "record / compiler-generated GetHashCode across a back-reference": SynchronizationStream folds its StreamConfiguration into its own hash, and StreamConfiguration folds its owning stream (probably captured on a configuration/delegate field) back in. Either type declared as a record — or a hand-written GetHashCode that combines the other — will produce exactly this trace. Confidence: high; the stack trace alone establishes the cycle.
What cannot be told from the evidence: which of the two types owns the offending field. That needs eyes on the source.
Impact
Single occurrence, single pod (memex-portal-deployment-7cfbf545d9-jk5qc, namespace memex), one log line — no user-visible outage yet. But this is StackOverflowException territory in .NET, which is unrecoverable and fast-fails the process. The trigger is a routine RoutingGrain NACK log; every future NACK that carries a stream reference is a coin-flip on the process. Treat as latent-critical rather than actual-critical.
Where to look
MeshWeaver.Data.Serialization.SynchronizationStream<TStream, TReference>— itsGetHashCode/ equality contract, especially any field of typeStreamConfiguration<...>.MeshWeaver.Data.Serialization.StreamConfiguration<TStream>— same, especially any field that references the owningSynchronizationStream(delegate targets captured via lambdas count).- If either is a
record/record struct, overrideGetHashCode(andEquals) explicitly and hash on stable identity only (e.g.StreamIdentity/ owner + reference key), not on the graph. - The logger call site in
MeshWeaver.Hosting.Orleans.RoutingGrainthat emits the[ROUTE] Cannot deliver the NotFound DeliveryFailure …message — verify noSynchronizationStream/StreamConfigurationinstance is being placed into the log scope; even after the hash bug is fixed, streams don't belong in log state.
Evidence
| Fingerprint | e089a7d2379666fb |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Error |
| Top frame | MeshWeaver.Data.Serialization.SynchronizationStream1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()` |
| Namespace | memex |
| Pods | memex-portal-deployment-7cfbf545d9-jk5qc |
| Occurrences | 4 |
| First seen | 2026-08-24 18:09:17Z |
| Last seen | 2026-08-24 18:09:17Z |
Recent log lines
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for n_uVS5E99kW-rA-rEAdcXQ to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(Sy…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Stream-routed delivery to 'portal/xXFPuF3UcAIJCvUl_PCsni8xS_98NY3AUqls6oi4nJU' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered. Message RawJson (UsSuFujm80yPxm4bOuIn2w) from sglauser was NOT posted — a publish to a subscriber-less stream succeeds and discards, which is why this had to be checked rather than observed. Surfacing DeliveryFailure to the sender.
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
a…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for IiiBt1HDAkyLhho1bOu_NA to sender sglauser: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/xXFPuF3UcAIJCvUl_PCsni8xS_98NY3AUqls6oi4nJU' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(Sys…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for B_4e7sF-OkSmh4-5aee-Uw to sender sglauser: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/xXFPuF3UcAIJCvUl_PCsni8xS_98NY3AUqls6oi4nJU' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(Sys…[truncated]
Opened automatically from Admin/_LogIncident/e089a7d2379666fb. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 25, 2026
Fixed by #2239. Root cause: these types are
records, so the compiler synthesized structuralGetHashCodeover every field — and the field graph contains a cycle.StreamConfiguration<T>'s primary-ctor propertyStreamis a back-pointer to the owningSynchronizationStream<T>, andSynchronizationStream's own hash walks its privateChangeItem<TStream>? current, descendingChangeItem → EntityStore → InstanceCollection. Both recorded production stacks were entered by hashing a STREAM — cycle 1 fed cycle 2. Fix: a live/identity object gets reference identity (ReferenceEquals+RuntimeHelpers.GetHashCode); the containers hash keys+count, never the arbitrary instance values. Proven red-before/green-after by reverse-applying the patch (cycle 1 crashed the test host with SIGABRT). Also fixed a latentInvalidOperationExceptionfromEntityStore.GetHashCode()'s seedless.Aggregate()on an empty store.