fix(membership): retain recently-declared-dead entries during pruning - #10269
Merged
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
When pruning excess defunct membership entries via MaxDefunctSiloEntries, retention was based on EffectiveIAmAliveTime (max of StartTime and IAmAliveTime). Declaring another silo dead does not update its IAmAliveTime, so a silo that was just declared dead but last reported itself alive long ago would be pruned immediately and disappear from membership snapshots. Reference the entry's suspect times during pruning instead: a new EffectiveUpdateTime takes the later of EffectiveIAmAliveTime and the most recent suspect vote. Declaring a silo dead records a suspect vote, so recently-declared-dead entries are now treated as the most recently updated and are retained. The cleanup agent's selection and cutoff, and the in-memory membership providers' removal predicate, all use EffectiveUpdateTime. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c946013-eaee-456e-8d14-7db4a8ec5854
ReubenBond
force-pushed
the
reubenbond-prune-dead-members-by-recency
branch
from
July 15, 2026 23:36
e86a00d to
99c1322
Compare
This was referenced Jul 22, 2026
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
We recently added
MaxDefunctSiloEntriesto cap the number of non-active (e.g.Dead) entries kept in the membership table. When the cap is exceeded, the cleanup agent prunes the oldest defunct entries, keeping the most-recently-updated ones so they still appear in membership snapshots.Recency was measured by
EffectiveIAmAliveTime(the later ofStartTimeandIAmAliveTime). The problem is that declaring another silo dead does not update that silo'sIAmAliveTime— only a silo's own periodic self-update does. So a silo that was declaredDeadjust now, but last reported itself alive long ago, looks like the oldest defunct entry and is pruned immediately, disappearing from snapshots right after it died.Solution
Reference the entry's suspect times during pruning instead of bumping
IAmAliveTimeon death. A new internalMembershipEntry.EffectiveUpdateTimereturns the later ofEffectiveIAmAliveTimeand the most recentSuspectTimesvote. Declaring a silo dead records a suspect vote, so a recently-declared-dead entry is now correctly treated as the most-recently-updated defunct entry and is retained.EffectiveUpdateTimeis used consistently in:DefunctSiloEntryPriority) and cutoff computation, andSystemTargetclustering).This keeps the death-declaration timestamp as the source of truth without mutating
IAmAliveTime.Notes / scope
IAmAliveTimecolumn and cannot see suspect times without a schema change — that remains a pre-existing limitation and is out of scope here.Tests
Added
MembershipTableCleanupAgent_ThresholdCleanup_RetainsRecentlySuspectedEntries, which sets up twoDeadentries where the one with the oldestIAmAliveTimehas a very recent suspect vote, and asserts it is the entry that survives pruning. Full membership suite passes (67 tests).