[Runtime Configuration][Client] Dynamic endpoint selection and load-balancing policy #591

Parent tracker: #262

Analysis / implementation baseline: dev.

Goal

Allow a running multi-endpoint ISharpLinkClient to replace the endpoint-selection policy used for future attempts without rebuilding the Client, reconnecting healthy sessions, or changing the current endpoint topology.

This issue covers both existing built-in strategies and the existing custom selector SPI:

Built-in:
  PowerOfTwoChoices
  Random
  RoundRobin
  LeastPending

Custom:
  ISharpLinkEndpointSelector

The intended capability is dynamic policy replacement, not runtime registration of new enum values.

Examples:

P2C -> LeastPending
RoundRobin -> custom ISharpLinkEndpointSelector
custom selector A -> custom selector B
custom selector -> P2C

Runtime semantics

Endpoint selection is a next-attempt policy.

selection policy generation N
        ↓
Attempt A selects endpoint using N
        ↓
control path publishes N+1
        ↓
Attempt A keeps the endpoint it already selected
future attempt B selects using N+1

Do not reselect an already-started attempt merely because configuration changes.

For a retrying logical call, each new physical attempt may observe the latest selection policy unless the existing retry/attempt architecture requires a stronger capture boundary. The implementation must document the chosen boundary and keep one individual attempt internally consistent.

Scope

• runtime replacement among all existing built-in load-balancing strategies;
• runtime replacement built-in -> custom ISharpLinkEndpointSelector;
• runtime replacement custom -> built-in;
• runtime replacement custom selector A -> B;
• complete candidate validation before atomic publication;
• static endpoint-cluster support;
• dynamic-resolver endpoint-cluster support;
• future endpoint attempts use the new policy without reconnecting healthy sessions;
• current resolver/topology snapshot remains authoritative and independent;
• update vs topology replacement / endpoint retirement / Stop / Dispose races;
• no new request-path lock or per-attempt allocation solely for dynamic configuration;
• benchmark P2C/Random/RoundRobin/LeastPending and custom-selector steady state before/after dynamic support.

Fixed/single-endpoint topology has no meaningful load-balancing choice and should remain on its specialized fast path.

Important boundary: policy replacement != topology mutation

This issue changes only:

"given the current Ready endpoint snapshot, which endpoint should this attempt choose?"

It does not change:

• endpoint membership;
• resolver snapshots;
• endpoint address/authority;
• connection-pool capacity;
• endpoint readiness;
• retry eligibility;
• endpoint admission / Circuit Breaker state.

Dynamic resolver updates remain owned by the resolver/topology lifecycle. Connection-pool scaling remains tracked separately.

Custom selector semantics

ISharpLinkEndpointSelector is already the extension point for application-defined synchronous endpoint selection. Runtime replacement should reuse that SPI rather than adding a second dynamic-selector abstraction.

Requirements:

• one selection operation invokes exactly one captured selector/policy;
• a policy update cannot cause one selection to call two selectors or combine outputs;
• replaced selector state is not implicitly migrated into the new selector;
• framework ownership/disposal of custom selector objects follows the existing SPI ownership contract; do not invent disposal semantics solely for hot reload;
• selector callbacks remain synchronous/non-blocking according to the existing contract;
• invalid selector output remains rejected exactly as today.

Built-in strategy state

Built-in strategies may own small local state such as RoundRobin / LeastPending cursors.

The implementation must explicitly decide whether compatible state is preserved across updates. Correctness does not require historical cursor continuity, but updates must not create unbounded state, cross-topology stale indexes, or races with endpoint snapshot replacement.

Prefer stable runtime state separated from the immutable current selection policy when that keeps the hot path simpler.

Publication model

Conceptually:

immutable EndpointSelectionPolicy N
        ↓
atomic publish
        ↓
new attempt reads N once
        ↓
selects from one Ready endpoint snapshot

update

immutable EndpointSelectionPolicy N+1

The selection policy and endpoint topology snapshot are separate publications. One attempt must use a valid combination: the selector can only return an endpoint/index from the Ready snapshot supplied to that selection operation.

Do not introduce a global configuration lock around endpoint selection.

Correctness invariants

1. One endpoint selection operation uses one policy generation.
2. An already-selected/in-flight attempt is never moved to another endpoint because policy changed.
3. The selected endpoint must belong to the Ready snapshot used for that selection.
4. Resolver/topology replacement cannot make a stale selector result newly eligible.
5. Retry exclusion / endpoint-admission / Circuit Breaker rules remain authoritative after policy selection.
6. Custom selector replacement cannot leak or retain obsolete topology generations.
7. Stop/Dispose seals further selection-policy publication.
8. Fixed/single-endpoint fast path remains unchanged.
9. No per-attempt allocation, global lock, or unnecessary virtual dispatch is added when using built-in strategies beyond the existing selection cost.

Validation

At minimum cover:

• P2C -> Random -> RoundRobin -> LeastPending transitions;
• built-in -> custom;
• custom -> built-in;
• custom A -> custom B;
• update while many concurrent selections run;
• update while resolver publishes a new topology;
• update while endpoint is retiring/draining;
• update while retry is choosing a later attempt;
• update with endpoint admission / Circuit Breaker enabled;
• invalid custom selector result after update;
• update vs Stop/Dispose;
• repeated updates do not retain obsolete selector/policy/topology state;
• static and dynamic endpoint clusters;
• fixed topology remains allocation/performance-identical.

Performance gates

Endpoint selection is a hot path. Compare same-runner base/head for at least:

P2C
Random
RoundRobin
LeastPending
custom selector
steady state after repeated updates

at representative concurrency.

Record:

• ns/op or selection throughput;
• end-to-end RPC QPS;
• P50/P99 where practical;
• B/op / objects-op;
• contention.

Requirements:

• no request-path global lock;
• no per-attempt configuration object allocation;
• built-in steady-state performance should remain within normal benchmark noise;
• fixed endpoint fast path must not gain a dynamic-selection branch.

Explicitly out of scope

• runtime registration of new SharpLinkLoadBalancingStrategy enum values;
• endpoint add/remove/address changes;
• resolver replacement;
• connection-pool scaling;
• retry-policy changes;
• endpoint admission / Circuit Breaker policy updates;
• cross-cluster routing/failover;
• transport/TLS/authentication replacement;
• per-call user override of load-balancing strategy;
• a giant unified runtime options object.

Completion

Complete when a running multi-endpoint Client can atomically replace its built-in or custom endpoint-selection policy and future attempts use the new policy without reconnect, topology mutation, attempt inconsistency, or hot-path regression.