Repository navigation
Conversation
A server that joins the cluster takes its config from the join request. That is the leader's committed config, which does not contain the joiner. The joiner's log is then synced from the start, so it holds config entries older than its current config, followed by the config that adds it. If the joiner is elected before its state machine applies those entries, become_leader() scans the uncommitted range for the newest config, but the scan stopped at the first entry older than the current config. It never reached the config that adds the joiner and re-appended the join config instead, so the new leader removed itself from the cluster: the followers dropped it as a peer, and requests they forward to it failed. Skip older config entries instead of stopping at them. A config that force recovery sets ahead of the log is still preferred, because every entry in the scanned range is older than it. Also set uncommitted_config_ to the appended config, as every other leader-side config change does. Otherwise a priority change handled before the new leader applies that config is built from its previous config, which on a new joiner still lacks the joiner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A server that joins takes its config from the join request. That is the leader's committed config, and it does not contain the joiner. The joiner's log is then synced to it (from index 1 for a new server), so it holds config entries older than that config, followed by the entry that adds the joiner.
If the joiner is elected before its state machine applies those entries,
become_leader()scans the uncommitted range for the newest config, but it stops at the first entry older than the current config (the check from #37). ClickHouse Keeper'srcfgdoestransfer_leadershipright after adding servers, so it can hit this. The scan never reaches the entry that adds the joiner and re-appends the join config. Once that commits:server 7 is removed from cluster);cannot find leader peer).Seen in ClickHouse CI,
test_keeper_4lw_reconfigurationon amd_tsan: 1, 2, 3.Changes:
next_slotstill wins, because every entry in the scanned range is older than it.uncommitted_config_to the appended config, as the membership and priority changes do. Otherwise a priority change handled before the new leader applies that config is built from its previous config, which on a joiner still lacks the joiner.Of the other config changes, add and remove also start from
uncommitted_config_.flip_learner_flag(),set_user_ctx()andupdate_srv_config()start fromget_config(); ClickHouse Keeper calls none of them (nor setsuse_new_joiner_type_), so they are unchanged here.The new test
new_joiner_elected_before_sm_catches_up_keeps_itself_in_config_testpauses the joiner's state machine, elects the joiner withyield_leadership, then forwards aset_priority_v2to it: