fix(directory): strengthen single-activation guarantees

[ledjon-behluli]

This PR adds safety lease holds to the new strongly-consistent grain directory to prevent split-brain scenarios after silo crashes, utilizing both specific grain leases (tombstones) and broader range leases when snapshot transfers fail. Under normal graceful shutdowns the new directory performs handovers, so no range lease holds are applied.

Users can specify the lease duration, otherwise if duration <= zero than it is computed as: 2 × ProbeTimeout × NumMissedProbesLimit which should provide a good buffer so that a network-partitioned silo has enough time to realize its isolated and shut itself down before the rest of the cluster allows those grains to reactivate elsewhere.

The impact on normal registrations in a healthy cluster should be practically unobservable.

Note, this wont work with the eventually-consistent directory!

Microsoft Reviewers: Open in CodeFlow

[ledjon-behluli]

@ReubenBond addressed the feedback

[Copilot]

Pull request overview

This PR strengthens the strongly-consistent distributed grain directory’s single-activation guarantees by introducing “safety lease holds” after ungraceful silo failures and during recovery scenarios, delaying re-registrations to reduce split-brain risk.

Changes:

• Add silo-level and range-level lease hold tracking to directory partitions, including periodic cleanup of expired leases.
• Enforce lease holds during RegisterAsync by rejecting registrations with a new DirectoryLeaseHoldException.
• Add configuration (SafetyLeaseHoldDuration) and a new test validating reactivation blocking after an ungraceful shutdown.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
test/Orleans.GrainDirectory.Tests/Orleans.GrainDirectory.Tests.csproj	Adds Microsoft.Extensions.TimeProvider.Testing package for FakeTimeProvider in tests.
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryLeaseTests.cs	New test exercising lease-hold behavior after KillSiloAsync.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs	Tracks/creates lease holds, applies holds on silo death, and adds cleanup logic + logging.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.Interface.cs	Enforces lease holds on registration by throwing DirectoryLeaseHoldException.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs	Computes effective lease duration, wires TimeProvider, and runs periodic lease cleanup requests.
src/Orleans.Runtime/Configuration/Options/GrainDirectoryOptions.cs	Adds SafetyLeaseHoldDuration option and documentation; switches to file-scoped namespace.
src/Orleans.Core.Abstractions/Exceptions/DirectoryLeaseHoldException.cs	Introduces a new exception type for lease-hold rejections.

Comments suppressed due to low confidence (2)

src/Orleans.Runtime/Configuration/Options/GrainDirectoryOptions.cs:88

• Typo in XML doc comment: "clcks skues" should be "clock skews".

    /// Conditional deregistration is used for lazy clean-up of activations whose prompt deregistration failed for some reason (e.g., message failure).
    /// This should always be at least one minute, since we compare the times on the directory partition, so message delays and clcks skues have
    /// to be allowed.

test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryLeaseTests.cs:65

• Grammar in comment: "its the only one alive" should be "it's the only one alive".

            // Time has expired, we can place it now, and it will be the primary as its the only one alive.
            Assert.Equal(await leaseGrain.GetAddress(), primary.SiloAddress);

[ReubenBond]

We should add some more tests. I'll ask copilot to add some more coverage.

[ReubenBond]

@ledjon-behluli I made a minor change to not throw when a lease is still held. Now, the directory returns a 'retry after delay' result instead.

[ledjon-behluli]

@ReubenBond is there something left on my side?

[Copilot]

Copilot's findings

Suppressed comments (1)

src/Orleans.Runtime/Configuration/Options/GrainDirectoryOptions.cs:127

• This PR adds a new public option GrainDirectoryOptions.SafetyLeaseHoldDuration, but the generated public API surface file does not reflect it (see src/api/Orleans.Runtime/Orleans.Runtime.cs, GrainDirectoryOptions section). Public API changes in src/ projects typically require updating the corresponding src/api file.

    /// <summary>
    /// Gets or sets the duration for the safety lease hold applied after an ungraceful silo failure.
    /// This duration applies in two scenarios:
    /// <list type="bullet">
    /// <item>When a specific silo crashes ungracefully, grain lease holds prevent individual re-registration of its grains for this duration.</item>
    /// <item>When a directory partition can not acquire a snapshot from a previous owner, range lease holds prevent new registrations in that whole range for this duration.</item>
    /// </list>
    /// </summary>
    /// <remarks>
    /// Depending on the value of this, the duration is understood as:
    /// <list type="bullet">
    /// <item><c>SafetyLeaseHoldDuration > TimeSpan.Zero</c> - The lease duration is explicitly controlled by the user.</item>
    /// <item><c>SafetyLeaseHoldDuration = TimeSpan.Zero</c>. No leases are placed at all, effectively nullifying this safety option.</item>
    /// <item><c>SafetyLeaseHoldDuration = null</c> - The system computes a lease duration as:
    /// <c><see cref="ClusterMembershipOptions.ProbeTimeout"/> × <see cref="ClusterMembershipOptions.NumMissedProbesLimit"/></c>.
    /// This is the default value, and is designed to allow for cluster stabilization after failure detection completes.
    /// </item>
    /// </list>
    /// </remarks>
    public TimeSpan? SafetyLeaseHoldDuration { get; set; }

• Files reviewed: 14/14 changed files
• Comments generated: 1