fix(mesh): heartbeat is fire-and-forget — stop the callback leak that wedges the cache hub by rbuergi · Pull Request #152 · Systemorph/MeshWeaver

rbuergi · 2026-07-01T14:38:16Z

The wedge

Reproduced by a documentation crawl (and matching the prod wedge-reproduced-local-k3s-e2e-portal): after ~127 open sync streams, the cache hub accumulates 163 HeartBeatEvent callbacks pending >30 s ([STALE-CALLBACK]) → RoutingGrain TimeoutExceptions → /healthz probe times out (1 s) → pod goes 0/1 → 503s.

Root cause

The sync-stream keep-alive did Post and Observe the HeartBeatEvent. The Observe existed to catch a terminal NotFound and stop heart-beating a dead owner. But a healthy owner never acks a heartbeat, so that Observe registered a hub callback that never resolved — the Rx .Timeout completed the observable but left the underlying callback pending on the cache hub. One leak per heartbeat per live stream → the pile-up that wedges the hub. The heartbeat was waiting for an ack it must never wait for.

Fix — truly fire-and-forget

The code already claimed fire-and-forget (it had added the change-feed resubscribe as "the sole recycled-grain detector") but left the leaky Observe in. This completes that migration:

JsonSynchronizationStream — post the heartbeat, don't observe it. No callback, no leak. Keep-alive still works (the Post reaches live owners); recycled owners are re-detected by the change-feed resubscribe; a permanently-gone owner is heart-beaten into routing's ignore path each interval — a benign dropped post, not a storm.
RoutingGrain.PostFailureToSender — skip [CanBeIgnored] messages (HeartBeatEvent, Shutdown/Dispose): they're fire-and-forget with no awaiting sender, so a DeliveryFailure/NACK is meaningless and, for a permanently-gone owner, would be the NotFound storm. Makes the Orleans router agree with the monolith RoutingServiceBase, which already guarded [CanBeIgnored]. (The "when not deliverable ⇒ ignore, don't fail the delivery" half.)

Tests

HeartbeatStopsWhenOwnerDiesAfterSubscribeTest pinned the removed observe-and-stop behaviour → renamed to HeartbeatFireAndForgetTest and inverted: after the owner dies (NotFound reply), the fire-and-forget heartbeat ignores the reply and keeps flowing (count climbs) instead of freezing — goes RED if the leaky Observe returns. The storm (never-existent owner) + resubscribe (change-feed) tests are unaffected and still pass. Both changed projects build under Release -warnaserror.

🤖 Generated with Claude Code

… wedge the cache hub Root cause of the cache-hub wedge (reproduced by a documentation crawl: 163 HeartBeatEvent callbacks pending >30s on the cache hub → RoutingGrain TimeoutExceptions → /healthz probe times out → pod 0/1 → 503s): The sync-stream keep-alive did `Post` AND `Observe` the HeartBeatEvent. The Observe was there to catch a terminal NotFound and stop heart-beating a dead owner. But a HEALTHY owner never acks a heartbeat, so that Observe registered a hub callback that never resolved — the Rx `.Timeout` completed the observable but left the underlying callback pending on the (cache) hub. Across many live sync streams (e.g. one per open doc page) those leaked callbacks accumulated until the hub's action block / liveness probe stalled. The heartbeat effectively waited for an ack it must never wait for. Fix — make it truly fire-and-forget (the code already claimed to be, and had already added the change-feed resubscribe as "the sole recycled-grain detector", but left the leaky Observe in): - JsonSynchronizationStream: post the heartbeat, do NOT observe it. No callback, no leak. Keep-alive still works (the Post reaches live owners); recycled owners are re-detected by the existing change-feed resubscribe; a permanently-gone owner is heart-beaten into routing's ignore path every interval — a benign dropped post, not a storm — until the subscriber hub is itself collected. - RoutingGrain.PostFailureToSender: skip [CanBeIgnored] messages (HeartBeatEvent, Shutdown/Dispose). They are fire-and-forget with no awaiting sender, so a DeliveryFailure/NACK for them is meaningless and — for a permanently-gone owner heart-beaten every interval — would BE the NotFound storm. This makes the Orleans router agree with the monolith RoutingServiceBase, which already guarded [CanBeIgnored]. This is the "when not deliverable ⇒ ignore, don't fail the delivery" half. Test: HeartbeatStopsWhenOwnerDiesAfterSubscribeTest pinned the removed observe-and-stop behaviour; renamed to HeartbeatFireAndForgetTest and inverted — after the owner dies (NotFound reply), the fire-and-forget heartbeat IGNORES the reply and keeps flowing (count climbs) instead of freezing. Goes RED if the leaky Observe is reintroduced. Storm (never-existent owner) + resubscribe tests unaffected and still pass. Both changed projects build under -warnaserror. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-01T14:57:37Z