[fix][broker] Prevent incorrect isolation fallback during cluster updates - #26798
Denovo1998 wants to merge 1 commit into
Conversation
|
Thanks for covering the read of a partially updated topology. There appears to be a remaining gap after the isolation exclusions are calculated: For an ensemble of two, if only one primary is writable during exclusion calculation, an ungrouped bookie is permitted as fallback. A second primary can join before selection, leaving the ungrouped bookie eligible even though the updated primary group is sufficient. The reverse transition can cause an avoidable |
Follow-up to the concurrency concern raised in #26701.
Motivation
IsolatedBookieEnsemblePlacementPolicy#getExcludedBookiesWithIsolationGroupsreadsknownBookieswithout holding BookKeeper's topology lock, whileonClusterChangedmodifies thisHashMapunder the write lock.A placement request can observe an intermediate topology after a departing primary bookie has been removed but before its replacement has been added. This can incorrectly enable the ungrouped fallback even though the completed cluster update leaves enough writable primary bookies, affecting both
newEnsembleandreplaceBookie.Modifications
rwLock.readLock()throughout the isolation exclusion calculation so availability counts and exclusions use one consistent topology view.finally, including early-return and exception paths.IsolatedBookieEnsemblePlacementPolicyTestwith deterministic regression coverage for both ensemble creation and bookie replacement. The tests use a real in-memory metadata store and pause address resolution during the actualonClusterChangedpath, without mocks or direct manipulation of internal state.Verifying this change
This change added tests and can be verified as follows:
IsolatedBookieEnsemblePlacementPolicyTestpass with retries disabled../gradlew :pulsar-broker-common:test --tests "IsolatedBookieEnsemblePlacementPolicyTest" -PtestRetryCount=0 -PtestFailFast=false -PtestMaxParallelForks=1 quickCheckgit diff --checkDoes this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Isolation exclusion calculations now acquire the read lock corresponding to the write lock used by cluster membership updates.