[client] NotLeaderOrFollowerException persists when a stale TabletServer endpoint is reused #4256

Activity

1. gyang94

   added the

   Bug

   issue type on Sep 9, 2026
2. BackendArchitectX commented on Sep 9, 2026

   

   BackendArchitectX

   on Sep 9, 2026

   Contributor

   More actions

   Hi @gyang94, I’d like to work on this issue.

   I’ll first reproduce it on current main, specifically the case where the same TabletServer UID is advertised with a new endpoint while the old endpoint remains reachable. I’ll then narrow the fix based on the reproducer and add regression coverage.

   Could you please assign this issue to me if appropriate?
3. BackendArchitectX commented on Sep 9, 2026

   

   BackendArchitectX

   on Sep 9, 2026

   Last edited by BackendArchitectX

   Contributor

   More actions

   I reproduced the issue on current main.

   NettyClient caches connections by server UID only. When the same TabletServer UID is used with a different host/port, the client still reuses the old connection while that endpoint remains reachable.

   I’ve added a deterministic regression test for this behavior and am now working on the smallest connection-layer fix.
4. gyang94 commented on Sep 9, 2026

   

   gyang94

   on Sep 9, 2026

   ContributorAuthor

   More actions

   @BackendArchitectX Thanks for your work. Please go ahead. And I can help to review your fix.
   btw I don't have permission to assign issue. We need committers help to assign it to you.
5. BackendArchitectX commented on Sep 9, 2026

   

   BackendArchitectX

   on Sep 9, 2026

   Last edited by BackendArchitectX

   Contributor

   More actions

   Purpose

   Linked issue: close #4256

   During a rolling upgrade, a TabletServer may keep the same logical server UID while moving to a different host or port. If the previous endpoint remains reachable and is reused by another TabletServer, NettyClient currently continues using the connection cached only by server UID.

   This means refreshed metadata cannot take effect at the transport layer. Even when a caller supplies an updated ServerNode, connection lookup can return the existing connection to the previous endpoint, causing requests to continue reaching the wrong TabletServer.

   This change makes physical RPC connection identity endpoint-aware by keying connections with server UID, host and port.

   Brief change log

   • Key cached NettyClient connections by server UID, host and port instead of server UID alone.
   • Allow connections to different physical endpoints for the same logical server UID to coexist.
   • Keep disconnect(serverUid) as a logical-server operation by closing all endpoint-specific connections belonging to that UID.
   • Preserve UID-level isReady() semantics when multiple endpoint connections exist.
   • Make connection metric registration endpoint-specific so simultaneous connections sharing a UID do not overwrite each other's metric bookkeeping.
   • Add deterministic regression coverage proving that an updated endpoint for the same TabletServer UID is actually used while the previous endpoint remains reachable.
   • Add lifecycle coverage verifying that disconnect(serverUid) closes all connections belonging to that logical server.

   Why not replace the existing connection?

   Related PR #4216 handles a same-UID endpoint change by replacing and closing the connection currently associated with the UID.

   This PR deliberately keeps physical endpoints independently keyed instead.

   TabletServerGateway instances can capture a specific ServerNode. During metadata transitions, gateways referring to the previous and updated endpoints may therefore coexist temporarily. If connection ownership is still based only on UID, requests from those gateways can repeatedly replace each other's connection and close a connection that another caller is using.

   Using (serverUid, host, port) as the physical connection identity avoids making the RPC layer decide which reachable endpoint is newer. The ServerNode supplied by the caller determines where that request is routed, while existing idle-connection handling eventually cleans up unused endpoints.

   Why no NotLeaderOrFollowerException retry here?

   NotLeaderOrFollowerException exposes the stale routing condition, but the persistent behavior in #4256 comes from the transport cache preventing an updated ServerNode from taking effect.

   For FlussAdmin.listOffsets, metadata is refreshed before requests are constructed. On a subsequent caller retry, the refreshed TabletServer endpoint can therefore already be available. With UID-only connection caching, however, that updated endpoint is still masked by the old reachable connection.

   This PR fixes that transport-layer condition directly and intentionally does not introduce another retry policy.

   Tests

   Passed:

   • .\mvnw.cmd test -Dtest=NettyClientTest -pl fluss-rpc

     • 9 tests
     • 0 failures
     • 0 errors

   • .\mvnw.cmd verify -pl fluss-rpc -Dtest="!ApiErrorTest"

     • 89 tests
     • 0 failures
     • 0 errors
     • 2 skipped

   • Checkstyle: 0 violations

   • git diff --check

   The complete fluss-rpc verification on Windows additionally encounters the existing ApiErrorTest#testStringifyException line-ending assertion: the test expects LF while the Windows JVM stack trace uses CRLF. This test is unrelated to the changed RPC connection behavior.

   The new regression intentionally isolates the transport condition rather than reproducing an entire rolling upgrade: two reachable TabletServers use the same UID with different endpoints, and the test verifies that a request supplied with the updated endpoint reaches that endpoint rather than the previously cached one.

   API and Format

   No public API, RPC protocol or storage-format changes.

   Documentation

   No documentation changes. This fixes existing RPC connection behavior when a server endpoint changes.

   Generative AI disclosure

   Yes — ChatGPT (GPT-5.6 Sol) assisted with root-cause analysis, implementation review and test planning.
6. BackendArchitectX commented on Sep 9, 2026

   

   BackendArchitectX

   on Sep 9, 2026

   Contributor

   More actions

   Thanks @gyang94. I've opened PR #4263 with the endpoint-aware connection fix and deterministic regression coverage.

   I also documented the relationship with #4216 and the difference in connection handling.

   Thanks for offering to take a look. Any feedback would be appreciated.
7. BackendArchitectX commented on Sep 10, 2026

   

   BackendArchitectX

   on Sep 10, 2026

   Contributor

   More actions

   @loserwang1024, @gyang94 mentioned that a committer is needed to assign #4256.

   I?ve reproduced the stale-endpoint case and opened #4263 with the endpoint-aware NettyClient fix and deterministic regression coverage. I also coordinated with @litiliu on the overlap with #4216, and we agreed that #4263 should own the transport-level endpoint-change handling.

   Could you assign #4256 to me when convenient? I?ll continue driving #4263 through review and resolution.

   Thanks @gyang94 and @litiliu for the coordination.
8. polyzos commented on Oct 27, 2026

   

   polyzos

   on Oct 27, 2026

   Contributor

   More actions

   Adding a server-to-server data point from 1.0.0 (upstream Helm chart, which advertises the INTERNAL listener as the raw pod IP, see #2076).

   After a tablet server restarted with a new pod IP (re-registered in ZooKeeper within ~30 s with the new INTERNAL address), the other tablet servers' replica fetchers kept making new connection attempts to the old IP for ~10 minutes: Connection refused and connection timed out after 15000 ms: /<old-ip>:9123, logged as "Error in response from leader server N". It happened on every restart in the roll, and it kept the restarted server's buckets out of the ISR for those ~10 minutes, which in turn held back the chart's cluster-health readiness gate.

   These were fresh dials, not a lingering open connection, so it looks like the fetcher keeps the leader ServerNode it was created with rather than picking up the new endpoint. Could you confirm whether #4263 covers the server-side ReplicaFetcherThread / coordinator channels as well as the client?

   Reuse risk: on EKS a new pod received a restarted tablet server's old IP about 30 minutes later, so pod-IP reuse is real. Since the handshake only checks the server type, it might be worth also returning server id and cluster id in ApiVersionsResponse and verifying them on connect, so a reused IP can never be accepted as the wrong server.
9. polyzos commented on Oct 27, 2026

   

   polyzos

   on Oct 27, 2026

   Contributor

   More actions

   is there any help needed here?
10. gyang94 commented on Oct 27, 2026

    

    gyang94

    on Oct 27, 2026

    Last edited by gyang94

    ContributorAuthor

    More actions

    Hi @polyzos , there is a PR #4263 for this issue.
    @loserwang1024 and I have reviewed the changes and looks good. Now it is waiting for a merge from maintainers. You can take a look.