Skip to content

Avoid killing too many machines if one region is being shared between the remote primary and a satellite #9727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sfc-gh-abeamon merged 2 commits into from
Mar 17, 2023
Merged

Avoid killing too many machines if one region is being shared between the remote primary and a satellite #9727

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
75b8148
If one region is being shared between the remote primary and a satell…
sfc-gh-abeamon Mar 16, 2023
6818ce9
Fix check that excludes satellites from consideration to consider sat…
sfc-gh-abeamon Mar 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion fdbserver/tester.actor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1782,8 +1782,45 @@ ACTOR Future<Void> initializeSimConfig(Database db) {
g_simulator->storagePolicy = dbConfig.storagePolicy;
g_simulator->tLogPolicy = dbConfig.tLogPolicy;
g_simulator->tLogWriteAntiQuorum = dbConfig.tLogWriteAntiQuorum;
g_simulator->remoteTLogPolicy = dbConfig.getRemoteTLogPolicy();
g_simulator->usableRegions = dbConfig.usableRegions;

// If the same region is being shared between the remote and a satellite, then our simulated policy checking
// may fail to account for the total number of needed machines when deciding what can be killed. To work
// around this, we increase the required transaction logs in the remote policy to include the number of
// satellite logs that may get recruited there
bool foundSharedDcId = false;
std::set<Key> dcIds;
int maxSatelliteReplication = 0;
for (auto const& r : dbConfig.regions) {
if (!dcIds.insert(r.dcId).second) {
foundSharedDcId = true;
}
if (!r.satellites.empty() && r.satelliteTLogReplicationFactor > 0 && r.satelliteTLogUsableDcs > 0) {
for (auto const& s : r.satellites) {
if (!dcIds.insert(s.dcId).second) {
foundSharedDcId = true;
}
}

maxSatelliteReplication =
std::max(maxSatelliteReplication, r.satelliteTLogReplicationFactor / r.satelliteTLogUsableDcs);
}
}

if (foundSharedDcId) {
int totalRequired = std::max(dbConfig.tLogReplicationFactor, dbConfig.remoteTLogReplicationFactor) +
maxSatelliteReplication;
g_simulator->remoteTLogPolicy = Reference<IReplicationPolicy>(
new PolicyAcross(totalRequired, "zoneid", Reference<IReplicationPolicy>(new PolicyOne())));
TraceEvent("ChangingSimTLogPolicyForSharedRemote")
.detail("TotalRequired", totalRequired)
.detail("MaxSatelliteReplication", maxSatelliteReplication)
.detail("ActualPolicy", dbConfig.getRemoteTLogPolicy()->info())
.detail("SimulatorPolicy", g_simulator->remoteTLogPolicy->info());
} else {
g_simulator->remoteTLogPolicy = dbConfig.getRemoteTLogPolicy();
}

return Void();
} catch (Error& e) {
wait(tr.onError(e));
Expand Down