[Runtime Configuration][Client] Complete and dynamically update reconnect policy #592

Parent tracker: #262
Related timing foundation: #77
Related lifecycle work: #365

Analysis / implementation baseline: dev.

Goal

Turn SharpLink Client reconnect behavior into one explicit immutable ReconnectPolicy (or equivalent) and allow a running Client to replace that complete policy at runtime.

This issue intentionally combines two steps that belong together:

1. make the currently implicit/scattered reconnect rules explicit and configurable;
2. publish new reconnect policy generations at runtime so future reconnect scheduling uses the latest policy.

Do not add a second dynamic API on top of hard-coded reconnect constants.

Current behavior to characterize first

Before implementation, re-read the actual dev reconnect coordinators for fixed, static-cluster and dynamic-resolver topologies and record the real current values/ownership.

Current behavior includes concepts such as:

initial reconnect delay
exponential backoff
maximum reconnect delay
jitter
stable-ready duration after which backoff resets

Some of these are currently implementation rules rather than one explicit public/runtime policy.

The implementation must first produce a current-state matrix:

topology
reconnect owner
initial delay
backoff rule
max delay
jitter rule
reset-after-stable rule
time source
wake/signal owner

Then converge equivalent behavior on one policy model where practical.

Target policy

The exact public type/name is not fixed, but the policy should be able to express at least the existing reconnect behavior, conceptually:

ReconnectPolicy
    InitialDelay
    MaxDelay
    BackoffFactor
    JitterRatio / jitter policy
    StableResetDuration

If current topology implementations genuinely require an additional reconnect knob, add it only after characterization demonstrates the need.

Do not turn this into a generic resilience framework.

Runtime semantics

Reconnect configuration is a lifecycle/control-loop policy.

Conceptually:

ReconnectPolicy N
    ↓
connection failure
    ↓
reconnect coordinator schedules from N

runtime update publishes N+1
    ↓
future reconnect scheduling/backoff decisions use N+1

An already-armed reconnect wait must have deterministic behavior when policy changes.

Preferred rule:

• publication wakes/reschedules the existing reconnect coordinator when the new policy should materially change the next attempt time;
• do not create a second reconnect loop/timer;
• do not restart or duplicate a connection attempt already in progress;
• after the current attempt completes, subsequent scheduling uses the latest complete policy.

Backoff state vs policy

Policy and live reconnect state are different things.

policy:
  InitialDelay / MaxDelay / BackoffFactor / Jitter / StableResetDuration

state:
  current failure streak
  current backoff position
  stable-ready timestamp/history
  currently armed wait
  reconnect task/signal ownership

Runtime update must preserve meaningful live state instead of blindly resetting the backoff sequence.

Examples:

Increasing MaxDelay

If the Client is already at a 2-second backoff and MaxDelay changes from 5s to 30s, the next failure should continue from the existing sequence under the new cap; it should not reset to InitialDelay merely because configuration changed.

Decreasing MaxDelay

If current backoff state is 10s and new MaxDelay is 5s, future scheduling must clamp to the new bound.

Changing InitialDelay

Changing initial delay primarily affects a new failure sequence. It should not silently erase an existing failure streak unless the documented policy says so.

StableResetDuration

Changing the reset duration does not fabricate connection stability. Existing stable-ready history/timestamp must be preserved and evaluated against the new threshold.

The exact reconciliation rules must be documented and deterministic.

Jitter

Reconnect jitter should be part of the explicit policy rather than an unexplained hard-coded percentage.

Requirements:

• preserve a bounded jitter contract;
• no negative or unbounded delay;
• fake/deterministic tests must not depend on Random.Shared luck;
• do not introduce a general random-service abstraction into RPC hot paths;
• a tiny reconnect-only random source/test seam is acceptable if needed.

Scope

• characterize current reconnect behavior for fixed/static/dynamic topologies;
• introduce one explicit immutable reconnect policy or the smallest coherent equivalent;
• configure initial reconnect delay;
• configure max reconnect delay;
• configure backoff factor/rule;
• configure jitter ratio/policy;
• configure stable-ready reset duration;
• preserve existing reconnect state/history across compatible runtime updates;
• atomic publication of the complete policy;
• existing reconnect coordinator adopts new policy without duplicate loop/task/timer;
• prompt wake/reschedule where a shorter/new bound should take effect;
• update vs reconnect attempt / connection recovery / pool scaling / topology replacement races;
• update vs Stop/Dispose;
• fake-time deterministic coverage;
• diagnostics/docs expose effective reconnect policy where useful.

Topology scope

The common policy should apply to every built-in Client topology that supports automatic reconnect:

fixed endpoint
static endpoint cluster
dynamic resolver cluster

Do not force identical internal coordinator classes merely for API symmetry. Shared policy semantics are required; internal state owners may remain topology-specific.

Resolver observation/retry behavior that belongs to the resolver itself is not automatically part of this reconnect policy. Characterize that boundary rather than merging unrelated retry loops.

Relationship to RPC Retry

Reconnect policy and RPC retry policy are separate:

ReconnectPolicy
    -> when Client infrastructure creates/replaces physical connections

RetryPolicy
    -> whether one logical RPC gets another physical attempt

Do not reuse ISharpLinkRetryPolicy or its backoff state for reconnect scheduling.

Dynamic global retry policy remains tracked separately.

Relationship to connection-pool scaling

Connection-pool targets may change independently. Reconnect continues to converge toward the currently effective pool/readiness target.

A reconnect-policy update must not create connections beyond the current pool bounds, and pool shrink/grow must not reset reconnect policy/state unnecessarily.

Correctness invariants

1. One topology/runtime owns at most one reconnect coordinator loop per existing lifecycle design.
2. Runtime policy update cannot create duplicate reconnect attempts from duplicate loops/timers.
3. An in-progress connection attempt is not cancelled/restarted merely because reconnect policy changed unless a separately documented lifecycle reason already cancels it.
4. Existing failure streak/backoff state is preserved or reconciled deterministically; policy update is not an implicit reset.
5. Stable-ready history is preserved across policy updates.
6. Current connection-pool/topology bounds remain authoritative.
7. Stop/Dispose seals further policy publication and cancels/joins reconnect work exactly once.
8. Reconnect timing uses the existing Runtime TimeProvider / monotonic model.
9. Ordinary RPC invocation gets no new reconnect-policy read, lock or allocation.

Validation

At minimum cover with deterministic fake time:

• default/current policy reproduces existing reconnect behavior;
• InitialDelay decrease/increase;
• MaxDelay decrease below current backoff;
• MaxDelay increase while already in a failure streak;
• BackoffFactor update mid-streak;
• JitterRatio update;
• StableResetDuration decrease/increase while connection has existing stable history;
• policy update while reconnect delay is armed;
• policy update while connection attempt is in progress;
• repeated rapid updates;
• reconnect succeeds after update and future failure starts/reset behavior is correct;
• update vs connection-pool grow/shrink;
• update vs dynamic endpoint generation change;
• update vs GoAway/disconnect;
• update vs Stop/Dispose;
• invalid candidate rollback;
• no duplicate reconnect loop/task/timer;
• fixed/static/dynamic topology coverage.

Performance

Reconnect policy is lifecycle/control-plane state and must not affect the ordinary RPC hot path.

Requirements:

• no new per-RPC allocation;
• no endpoint-selection/pending/send-path policy lookup;
• update path may allocate/validate;
• reconnect scheduling itself should remain bounded and supervised;
• repeated updates must not leak timers/tasks/policy generations.

Benchmarking ordinary RPC should show no measurable change; reconnect-focused timing/load tests should validate scheduling behavior rather than optimize configuration-update throughput.

Explicitly out of scope

• RPC retry policy;
• heartbeat interval/timeout ([Runtime Configuration][Client] Dynamic heartbeat interval and liveness timeout #586);
• connection-pool sizing ([Runtime Configuration][Client] Dynamic connection-pool scaling and graceful scale-down #587);
• endpoint-selection/load-balancing policy;
• endpoint admission / Circuit Breaker;
• resolver API redesign;
• transport/TLS/authentication replacement;
• changing endpoint topology through this API;
• unbounded reconnect;
• a generalized resilience-policy framework;
• Hosting / IOptionsMonitor integration before the final [Runtime Configuration] 动态配置 / Hot Reload 长期跟踪 #262 control-surface audit.

Completion

Complete when reconnect behavior is represented by one explicit documented policy model and a running Client can atomically replace that policy so existing reconnect coordinators safely use the new values while preserving live backoff/stability state, without duplicate lifecycle work or RPC hot-path cost.