Skip to content

client: add failover broadcast listener and sentinel active probe - #27

Open
chenshi5012 wants to merge 1 commit into
valkey-io:masterfrom
chenshi5012:feature/failover-enhancement
Open

client: add failover broadcast listener and sentinel active probe#27
chenshi5012 wants to merge 1 commit into
valkey-io:masterfrom
chenshi5012:feature/failover-enhancement

Conversation

@chenshi5012

@chenshi5012 chenshi5012 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds two complementary failover-detection enhancements to the Java client, addressing the client-side aspect of valkey-io/valkey#3881.


Changes

1. Cluster mode — FailoverListener in JedisClusterInfoCache

A new inner class FailoverListener subscribes to the +switch-master Pub/Sub channel on every start node. When the server-side patch (valkey-io/valkey#4340) publishes a failover notification, the listener immediately calls renewClusterSlots() to update the slot cache in real time.

Key design points:

  • One daemon FailoverListener thread per start node.
  • Exponential back-off on reconnect (1 s → 30 s cap).
  • Performs an immediate slot refresh on (re-)connect to catch any failover that occurred while the listener was disconnected.
  • Can be disabled via setFailoverListenerEnabled(false) before first use.
  • Properly cleaned up in close().

Before this change: clients discover master role changes only through MOVED/ASK errors or periodic CLUSTER SLOTS refreshes, introducing latency proportional to the refresh interval.

After this change: clients react within milliseconds of the failover notification being published.


2. Sentinel mode — ActiveProbeTask in SentineledConnectionProvider

A single-threaded ScheduledExecutorService periodically calls SENTINEL GETMASTERADDRBYNAME on every sentinel. If the returned master address differs from the current pool target, initMaster() is called immediately.

Key design points:

  • Default interval: 10 s (configurable via the new probePeriodMillis constructor parameter).
  • Set probePeriodMillis <= 0 to disable the probe entirely.
  • Iterates sentinels in order and stops at the first reachable one.
  • Daemon thread, stopped in close().

Motivation: The existing SentinelListener relies on the Pub/Sub +switch-master event. If the subscription connection is disrupted (network partition, sentinel restart), the client may not receive the notification. The active probe provides a safety net that guarantees convergence within at most one probe interval.


Compatibility

  • Fully backward-compatible. No existing constructor signatures are changed; new constructors are additive.
  • The FailoverListener is started automatically after discoverClusterNodesAndSlots() and can be disabled.
  • The active probe is opt-in via the new constructor; the default interval is 10 s.

Related

Two complementary failover-detection enhancements:

1. Cluster mode – FailoverListener (JedisClusterInfoCache)
   Subscribe to the '+switch-master' Pub/Sub channel on every start node.
   When the server-side patch publishes a failover notification the listener
   immediately calls renewClusterSlots() so that the slot cache is updated
   in real time instead of waiting for the next periodic refresh cycle or
   an error-driven MOVED/ASK redirect.

   Key design points:
   - One daemon FailoverListener thread per start node.
   - Exponential back-off on reconnect (1s -> 30s cap).
   - Performs an immediate slot refresh on (re-)connect to catch any
     failover that occurred while the listener was disconnected.
   - Can be disabled via setFailoverListenerEnabled(false).
   - Properly cleaned up in close().

2. Sentinel mode – ActiveProbeTask (SentineledConnectionProvider)
   A single-threaded ScheduledExecutorService calls
   SENTINEL GETMASTERADDRBYNAME on every sentinel at a configurable
   interval (default 10 s).  If the returned master differs from the
   current pool target, initMaster() is called immediately.

   This provides a safety net for cases where the '+switch-master'
   Pub/Sub notification is missed (e.g. network partition, sentinel
   restart) and ensures the client converges to the correct master
   within at most one probe interval.

   Key design points:
   - Interval configurable via the new probePeriodMillis constructor
     parameter; set to <= 0 to disable.
   - Iterates sentinels in order and stops at the first reachable one.
   - Daemon thread, stopped in close().

Related: valkey-io/valkey#3881
Signed-off-by: chenshi5012 <chenshi5012@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant