Repository navigation
docs: one silo departure on memex-cloud, read correctly (Polly final attempt, DeliverMessage re-send vs drop) - #5450
Conversation
… final-attempt Error correctly One departed silo (S10.244.9.90:11111:149090968) was filed as seven issues across three log sites. Records the timeline, that 'fail: Polly[3] Handled: True Attempt: 3' is the FINAL placement attempt (retries exhausted, not absorbed), where RoutingGrain re-sends a rejected DeliverMessage vs drops a placement timeout, and the one read that settles graceful-vs-crash. Refs #5336 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Four documentation nits remain regarding evidence wording, timestamps, classification flow, and placement-timeout correlation.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Docs-only PR adding a second measured memex-cloud silo-departure case, including retry semantics and delivery behavior.
Changes:
- Adds the 2026-09-22 incident timeline and issue correlations.
- Clarifies Polly’s final-attempt semantics.
- Documents resend behavior and unresolved shutdown classification.
| File | Summary |
|---|---|
src/MeshWeaver.Documentation/Data/Architecture/PeerDeathIsDatedByItsExceptionClass.md |
Adds the incident analysis and operational guidance; four nit findings remain. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | ≤ 14:41:00 | Directory lookups into 9.90's partition start hanging | inferred: the first placement timeouts expire at 14:41:30 against a 30 s budget | | ||
| | 14:41:30 | First `TimeoutException: Grain placement operation timed out` on xg22d (`messagehub/Doc/Architecture/PolicyNotProse`). Its innermost frame is `PlacementService.cs:602`, which is the `_grainLocator.Lookup(...)` await: the **directory lookup**, not local placement | measured (#5334) | | ||
| | 14:41:30 | First `ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after …` on 8pdzb, raised from `LocalGrainDirectory.LookupAsync` | measured (#5333, #5336) | | ||
| | 14:41:41 | First `SocketConnectionException … Error: HostUnreachable`: the IP itself stopped answering | measured (#5338, #5342) | |
There was a problem hiding this comment.
Agreed, and fixed in 1051351. The 14:41:41 row now reads HostUnreachable as a caller-side reachability loss that is not by itself proof the pod is gone. The paragraph after the table is also reworded: the survivors never reached the address again and no new generation appeared there. That is consistent with a deleted pod, while a broken route or a network policy produces the same socket error (the caveat at the top of the page). So it does not prove deletion. The "What is NOT established" section already leaves graceful stop vs. crash open.
…he pod is gone (review) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Docs only. This adds a section to
Doc/Architecture/PeerDeathIsDatedByItsExceptionClass: a second measured case, from 2026-09-22 on memex-cloud.S10.244.9.90:11111:149090968went away at 14:41Z. The survivors logged it at three sites (Polly[3], andOrleans.Messaging[100071]as either a connect failure or a placement timeout), and each site's lines were filed separately. The section gives the timeline, which the silo generations date.fail: Polly[3] … Handled: 'True', Attempt: '3'is the final attempt. Polly logs a handled final attempt at Error (TelemetryUtil.ReportFinalExecutionAttempt), andPlacementMaxRetries= 3. So this line means Orleans' own placement retries ran out and the message was rejected. It does not mean the retry absorbed the fault, which is how three of the issues read it.RoutingGrainre-sends a connection-failed rejection 6 times (about 10 s), and when those run out it classifies the failure asShuttingDown. It never re-sends a placementTimeoutException, becauseIsResponseTimeoutmatches everyTimeoutException. Its documented premise ("the callee already holds the request") is false for a placement timeout. The doc records that as an open design question and does not change it.Logsread that settles it.Refs #5336 · Refs #5333 · Refs #5334 · Refs #5338 · Refs #5340 · Refs #5342 · Refs #5343 · Refs #5300
Local:
dotnet build -c Release -warnaserror src/MeshWeaver.Documentationgives 0 Warning(s), 0 Error(s). The change adds no internal links.🤖 Generated with Claude Code