Skip to content

fix(membership): retain recently-declared-dead entries during pruning - #10269

Merged
ReubenBond merged 1 commit into
mainfrom
reubenbond-prune-dead-members-by-recency
Jul 16, 2026
Merged

fix(membership): retain recently-declared-dead entries during pruning#10269
ReubenBond merged 1 commit into
mainfrom
reubenbond-prune-dead-members-by-recency

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jul 15, 2026

Copy link
Copy Markdown
Member

Problem

We recently added MaxDefunctSiloEntries to cap the number of non-active (e.g. Dead) entries kept in the membership table. When the cap is exceeded, the cleanup agent prunes the oldest defunct entries, keeping the most-recently-updated ones so they still appear in membership snapshots.

Recency was measured by EffectiveIAmAliveTime (the later of StartTime and IAmAliveTime). The problem is that declaring another silo dead does not update that silo's IAmAliveTime — only a silo's own periodic self-update does. So a silo that was declared Dead just now, but last reported itself alive long ago, looks like the oldest defunct entry and is pruned immediately, disappearing from snapshots right after it died.

Solution

Reference the entry's suspect times during pruning instead of bumping IAmAliveTime on death. A new internal MembershipEntry.EffectiveUpdateTime returns the later of EffectiveIAmAliveTime and the most recent SuspectTimes vote. Declaring a silo dead records a suspect vote, so a recently-declared-dead entry is now correctly treated as the most-recently-updated defunct entry and is retained.

EffectiveUpdateTime is used consistently in:

  • the cleanup agent's defunct-entry selection (DefunctSiloEntryPriority) and cutoff computation, and
  • the in-memory membership providers' removal predicate (used by the default/dev SystemTarget clustering).

This keeps the death-declaration timestamp as the source of truth without mutating IAmAliveTime.

Notes / scope

  • Cloud/ADO.NET providers are unchanged. The Azure table provider already keys removal off the row write timestamp (which reflects the death write), so recently-dead entries are retained there. The ADO.NET provider keys removal off the IAmAliveTime column and cannot see suspect times without a schema change — that remains a pre-existing limitation and is out of scope here.

Tests

Added MembershipTableCleanupAgent_ThresholdCleanup_RetainsRecentlySuspectedEntries, which sets up two Dead entries where the one with the oldest IAmAliveTime has a very recent suspect vote, and asserts it is the entry that survives pruning. Full membership suite passes (67 tests).

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

When pruning excess defunct membership entries via MaxDefunctSiloEntries,
retention was based on EffectiveIAmAliveTime (max of StartTime and
IAmAliveTime). Declaring another silo dead does not update its IAmAliveTime,
so a silo that was just declared dead but last reported itself alive long ago
would be pruned immediately and disappear from membership snapshots.

Reference the entry's suspect times during pruning instead: a new
EffectiveUpdateTime takes the later of EffectiveIAmAliveTime and the most
recent suspect vote. Declaring a silo dead records a suspect vote, so
recently-declared-dead entries are now treated as the most recently updated
and are retained. The cleanup agent's selection and cutoff, and the in-memory
membership providers' removal predicate, all use EffectiveUpdateTime.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9c946013-eaee-456e-8d14-7db4a8ec5854
@ReubenBond
ReubenBond force-pushed the reubenbond-prune-dead-members-by-recency branch from e86a00d to 99c1322 Compare July 15, 2026 23:36
@ReubenBond ReubenBond changed the title fix(membership): advance IAmAliveTime when declaring a silo dead fix(membership): retain recently-declared-dead entries during pruning Jul 15, 2026
@ReubenBond
ReubenBond merged commit 54da946 into main Jul 16, 2026
119 of 121 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-prune-dead-members-by-recency branch July 16, 2026 00:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant