feat(membership): detect local stalls during direct probes

[ReubenBond]

If there is a long GC pause during an inter-silo probe that subsequently times out, suppress the fault. Currently set to 25% of the window (so, a >= 1.25s GC pause will result in ignoring the failed probe by default)

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

This PR updates Orleans runtime membership probing to detect significant local GC pauses during direct inter-silo probes and to treat certain probe timeouts as inconclusive, reducing false fault detection when the local silo is unable to process a reply in time.

Changes:

• Passes the effective probe timeout into direct probing so it can be used for local GC-pause evaluation.
• Tracks GC.GetTotalPauseDuration() across a direct probe and returns ProbeResultStatus.Unknown when GC pause time exceeds 25% of the probe timeout.
• Adds a new warning log for probe failures treated as inconclusive due to detected GC pause.

Show a summary per file

File	Description
src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs	Adds GC-pause-aware handling for direct probe failures/timeouts and corresponding logging.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 1/1 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:375

• When a direct probe times out during a detected local GC pause, the failure is treated as inconclusive, but MessagingInstruments.OnPingReplyMissed(...) is not invoked. That means PingReplyMissedCounter will under-report missed replies for these timeouts, while other failure/unknown paths (including indirect probes) still record a missed reply. Consider recording the missed reply before the GC-pause suppression check so metrics remain consistent.

                // Check if a GC pause consumed a significant portion of the probe timeout.
                // If so, the local silo may have been unable to process the response in time,
                // so we treat this as an inconclusive result rather than a failure.
                var gcPauseDuring = _localSiloHealthMonitor.TotalGarbageCollectionPauseDuration - gcPauseBefore;
                if (probeTimedOut && gcPauseDuring >= probeTimeout.Multiply(0.25))

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite