Repository navigation
Infinite recursion between StreamConfiguration<T>.GetHashCode() and SynchronizationStream<T>.GetHashCode() #2173
Description
What is failing
MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() and MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() call each other through the default ObjectEqualityComparer<T> and never terminate. Any code path that hashes a stream key blows the stack.
Probable cause
High confidence. The captured stack is a tight, repeating cycle:
ObjectEqualityComparer<T>.GetHashCode(T)
→ StreamConfiguration<T>.GetHashCode()
→ SynchronizationStream<T>.GetHashCode()
→ ObjectEqualityComparer<T>.GetHashCode(T) // back to the top
The shape says each type’s GetHashCode() delegates to a field/property whose own hash resolves via the default comparer back into the other type — most likely StreamConfiguration hashes its owning SynchronizationStream and SynchronizationStream hashes its StreamConfiguration (or one of them falls back to this through the default comparer). Fix is to break the cycle: hash on identity/immutable key fields (Owner, Reference, stream id) rather than delegating to the peer object, or drop one of the two overrides entirely.
Impact
Low so far but load-bearing risk. 4 occurrences over ~4 ms on a single pod (memex-portal-deployment-…-jk5qc) at 18:09:17 UTC — a single request storm, not an ongoing outage. However, StackOverflowException from recursive GetHashCode is unrecoverable and will crash the process the moment a hot path (routing table lookup, dictionary keyed on a stream) hits it under real traffic. Treat as a latent crash bug, not a nuisance.
Where to look
The log category is MeshWeaver.Hosting.Orleans.RoutingGrain — misleading. That’s just where the invocation originates (routing a message to a portal/… hub whose stream key it hashes). The defect is entirely in the MeshWeaver.Data.Serialization assembly:
MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode()MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode()
Start from either override and follow what it delegates to — the two ends of the cycle are in these two files.
No prior incident found on this fingerprint or on the StreamConfiguration<T> / SynchronizationStream<T> pair.
Evidence
| Fingerprint | 52c67215dc2ff138 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Error |
| Top frame | MeshWeaver.Data.Serialization.StreamConfiguration1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()` |
| Namespace | memex |
| Pods | memex-portal-deployment-7cfbf545d9-jk5qc |
| Occurrences | 4 |
| First seen | 2026-08-24 18:09:17Z |
| Last seen | 2026-08-24 18:09:17Z |
Recent log lines
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for hV-ypLdXVEesO8W4wBzPIA to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Ge…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered. Message RawJson (ZxYwup-rVE6CJcRB04HOjQ) from rsalzmann was NOT posted — a publish to a subscriber-less stream succeeds and discards, which is why this had to be checked rather than observed. Surfacing DeliveryFailure to the sender.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for zQpV38tzR0eAjMFK8s1nEA to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for XTYlYjYse0aTZUSk1IWHAg to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Ge…[truncated]
Opened automatically from Admin/_LogIncident/52c67215dc2ff138. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 25, 2026
Duplicate — same root cause, fixed by #2239 (see #2163/#2170/#2172/#2174).
rbuergi commentedon Aug 25, 2026
Duplicate of #2172 — same fingerprint
52c67215dc2ff138, filed twice in the same second. Both fixed by #2239 (merged as 4585c51); see #2172 for the root-cause writeup.