Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
174 commits
Select commit Hold shift + click to select a range
7121350
Fix transaction_too_old error when version vector is enabled
jzhou77 Nov 5, 2022
3371992
Fix assertions w.r.t. VV
jzhou77 Nov 5, 2022
c3f25a2
Avoid using oldest version as read version for VV
jzhou77 Nov 7, 2022
18add2a
Disable a debugging trace event
jzhou77 Nov 7, 2022
339543a
Cherry pick 8630
dlambrig Nov 8, 2022
4173acf
Address review comments
jzhou77 Nov 8, 2022
b4bd84b
enable AVX and update version for 7.1.25 release
foundationdb-ci Nov 7, 2022
6cb7ac4
skip proxy when fetching kubectl
neethuhaneesha Nov 8, 2022
3a4641a
Merge pull request #8710 from jzhou77/release-7.1
jzhou77 Nov 8, 2022
1439943
add generated.go
dlambrig Nov 9, 2022
cf0aed4
update version after 7.1.25 release
foundationdb-ci Nov 9, 2022
e58d388
Add changes to generated.go from PR8761, and remove change to Configu…
dlambrig Nov 9, 2022
cf8b9fa
Merge remote-tracking branch 'origin/release-7.1' into r71
dlambrig Nov 9, 2022
91f452e
Update generated.go
dlambrig Nov 9, 2022
3b52fc4
Update generated.go with 8761
dlambrig Nov 9, 2022
288e922
Merge pull request #8740 from apple/r71
jzhou77 Nov 9, 2022
347bd92
Rocksdb stats level knob. (#8713)
neethuhaneesha Nov 11, 2022
c2321bf
Adding counters for singlekey clear requests (#8792)
neethuhaneesha Nov 12, 2022
32a9c5a
add bytelimit for prefetch
flowguru Nov 11, 2022
fadcb08
debug seg fault
flowguru Nov 14, 2022
f6dc3ba
Revert "debug seg fault"
flowguru Nov 15, 2022
f9338d6
[release-7.1] Add SS read range bytes metrics. (#8697) (#8724)
xy-54321 Nov 16, 2022
b0f84b8
Merge pull request #8802 from hfu94/byte71
jzhou77 Nov 16, 2022
074e6f7
Rocksdb suggest compact range checks
neethuhaneesha Nov 16, 2022
5839b6f
Merge pull request #8862 from neethuhaneesha/suggest-compacts-7.1
jzhou77 Nov 18, 2022
a716127
RocksDB 7.7.3 version upgrade
neethuhaneesha Nov 17, 2022
76df95d
Merge pull request #8880 from neethuhaneesha/rocksdb_version-7.1
jzhou77 Nov 19, 2022
5a12979
Fix backup worker assertion failure
jzhou77 Nov 19, 2022
ae6ae7c
Merge pull request #8887 from jzhou77/release-7.1
jzhou77 Nov 22, 2022
5dbbbe9
Increase buggified lock bytes for backup workers
jzhou77 Nov 28, 2022
02601a7
Merge pull request #8928 from jzhou77/release-7.1
jzhou77 Nov 28, 2022
45e88e1
Send error when LogRouterPeekPopped happens
jzhou77 Nov 16, 2022
719927d
Add more debug events
jzhou77 Nov 17, 2022
8ec6ae1
Add DebugTrace.h to 7.1 branch
xumengpanda Dec 2, 2022
2ccfa55
Merge pull request #8966 from xumengpanda/release-7.1
jzhou77 Dec 5, 2022
c764f80
Fix the bug of variable int32 overflow.
bryant507 Nov 29, 2022
55d2fd3
Fix -Wformat warning
sfc-gh-tclinkenbeard Nov 29, 2022
89a038c
Merge pull request #8996 from jzhou77/release-7.1
jzhou77 Dec 7, 2022
fc6a77f
Add determinstic in gray failure degraded server selection
halfprice Dec 2, 2022
add49dc
Merge pull request #9001 from halfprice/zhewu/cherry-pick-determinsm
jzhou77 Dec 8, 2022
5c644f2
format source code after switch to clang 15
foundationdb-ci Dec 8, 2022
af99d3b
Fix clang 15 compiling errors
jzhou77 Dec 9, 2022
711fb7f
Fix gcc 11 compiling errors
jzhou77 Dec 9, 2022
e362a1f
Fix more warnings
jzhou77 Dec 9, 2022
5984912
Merge pull request #9018 from jzhou77/release-7.1
jzhou77 Dec 10, 2022
dae0e0e
Moving rocksdb read iterator destruction from commit path to actor. (…
neethuhaneesha Dec 13, 2022
0f21fc6
Release 7.1: Cherry pick pull request #9033 (#9037)
sbodagala Dec 14, 2022
250a67a
Fix:Exclusion stuck because DD cannot build new teams
xumengpanda Nov 10, 2022
89854c4
Resolve review comment: No functional change
xumengpanda Dec 13, 2022
2b04c18
Merge branch 'release-7.1' into release-7.1
xumengpanda Dec 19, 2022
a788c09
Merge pull request #9035 from xumengpanda/release-7.1
jzhou77 Dec 20, 2022
3012c71
Add back samples for (non)empty peeks stats [release-7.1] (#9074)
jzhou77 Dec 21, 2022
f5163b3
fix health monitor last logged time
halfprice Dec 22, 2022
4d9493e
Backport RocksDB cmake file to 7.1 (#9093)
xis19 Jan 5, 2023
1eaf95a
add some rocksdb compile options that can be passed in at build time
ammolitor Jan 6, 2023
ee7c9d4
Disconnection to satellite TLog should trigger recovery in gray failu…
halfprice Jan 6, 2023
fd6a5c6
Merge pull request #9107 from halfprice/zhewu/satellite-disconnection…
jzhou77 Jan 9, 2023
b2cf133
Merge pull request #9102 from ammolitor/add_rocksdb_compile_options
jzhou77 Jan 9, 2023
1f1390c
Upgrade sphinx and document test harness and code probes
sfc-gh-mpilman Sep 20, 2022
0566533
Apply suggestions from code review
sfc-gh-mpilman Sep 21, 2022
72e1ca6
clarify how code probes are reported
sfc-gh-mpilman Sep 21, 2022
5218535
clarify statistics of TestHarness
sfc-gh-mpilman Sep 21, 2022
ddd85ef
Bump setuptools from 65.3.0 to 65.5.1 in /documentation/sphinx
dependabot[bot] Dec 27, 2022
0e0f567
Merge pull request #9172 from jzhou77/release-7.1
jzhou77 Jan 17, 2023
91f6252
Add event for txn server initialization and a warning for TLog slow c…
halfprice Jan 11, 2023
9268e19
Change TLog pull async data warning timeout
halfprice Jan 12, 2023
89a4555
Adding rocksDB control compaction on deletion knobs. (#9165)
neethuhaneesha Jan 18, 2023
3df4a40
Add 7.1.26, 7.1.27 release notes (#9186)
jzhou77 Jan 20, 2023
83b235d
Added metrics for read range operations.
neethuhaneesha Jan 21, 2023
56ee545
Merge pull request #9211 from neethuhaneesha/SystemMetrics-7.1
jzhou77 Jan 24, 2023
2a0ac99
Log PingLatency when there is no ping latency samples, but ping attempts
halfprice Jan 21, 2023
d4b1ed0
Merge pull request #9223 from halfprice/zhewu/log-ping-latency-7.1
jzhou77 Jan 24, 2023
d59629e
Changing histogram type. (#9227)
neethuhaneesha Jan 25, 2023
acd663e
Release 7.1: Cherry pick pull request #9225 (#9252)
sbodagala Jan 27, 2023
2d04275
disable AVX for 7.1.26 release
foundationdb-ci Jan 27, 2023
dc4bd7e
enable AVX and update version for 7.1.27 release
foundationdb-ci Jan 27, 2023
788f36d
update version after 7.1.27 release
foundationdb-ci Jan 31, 2023
8b3a5ab
Increase buggified lock bytes for backup workers to at least 256 MB.
sfc-gh-dadkins Jan 20, 2023
668e142
Reduce logging level for verbose events
jzhou77 Feb 1, 2023
98275ef
Merge pull request #9280 from jzhou77/release-7.1
jzhou77 Feb 1, 2023
1f5da87
Fix typo in fdb.options
dlambrig Feb 2, 2023
ff2531d
update bindings/go/src/fdb/generated.go
dlambrig Feb 3, 2023
37fe673
Merge branch 'release-7.1' into vexilographerfix
dlambrig Feb 6, 2023
c1c432d
Merge pull request #9294 from apple/vexilographerfix
jzhou77 Feb 7, 2023
b0f78ef
Fix getMappedRange metrics(release-7.1) (#9331)
flowguru Feb 13, 2023
f99915a
Enable rocksdb in simulation in 7.1. Exclude FuzzApi and HighContenti…
neethuhaneesha Feb 15, 2023
55c1343
Fix IDE build and warnings
jzhou77 Feb 16, 2023
5657e3c
Rocksdb knob changes. (#9393)
neethuhaneesha Feb 17, 2023
950c047
Fix compiler warnings
jzhou77 Feb 22, 2023
8ffacc7
Add exclude to fdbcli's configure command
jzhou77 Feb 16, 2023
550be07
Allow a comma separated list of excluded addresses
jzhou77 Feb 17, 2023
e6a64e0
Add ClogTlog workload
jzhou77 Feb 21, 2023
5b345b2
Update clogTlog workload to be single region
jzhou77 Feb 21, 2023
314a4d1
Exclude failed tlog if recovery stuck more than 30s
jzhou77 Feb 23, 2023
51375c5
Change to only clog once for a particular tlog
jzhou77 Feb 23, 2023
ac58b84
Move ClogTlog.toml to rare
jzhou77 Feb 23, 2023
5868572
Fix rare test failures
jzhou77 Feb 23, 2023
bb3bb78
Address review comments
jzhou77 Feb 24, 2023
2d77a6a
Merge branch 'release-7.1' into release-7.1
jzhou77 Feb 25, 2023
e545dc3
Allow fdbdecode to read filters from a file
jzhou77 Feb 25, 2023
f72693d
Fix filter delimiter and print sub versions
jzhou77 Feb 25, 2023
3dc8771
Merge pull request #9404 from jzhou77/release-7.1
jzhou77 Feb 25, 2023
745e86e
Use KeyRangeMap for better matching performance
jzhou77 Feb 25, 2023
6b70e5f
fdbdecode: read backup range files
vishesh Feb 25, 2023
2be2644
add filtering
vishesh Feb 25, 2023
c9f13aa
Allow fdbdecode to read filters from a file
jzhou77 Feb 25, 2023
f41f08d
Fix filter delimiter and print sub versions
jzhou77 Feb 25, 2023
1626d63
Use KeyRangeMap for better matching performance
jzhou77 Feb 25, 2023
41a92e9
Disable filter validate by default
jzhou77 Feb 25, 2023
2c8f9da
Use RangeMap for backup agent filtering
jzhou77 Feb 26, 2023
7eb805d
Refactor code
jzhou77 Feb 26, 2023
91bc83a
Allow fdbbackup, fdbrestore to read keyranges from a file
jzhou77 Feb 26, 2023
e652abf
Merge remote-tracking branch 'jzhou/restore' into restore
vishesh Feb 26, 2023
794f371
Use the RangeMapFilters
vishesh Feb 26, 2023
c71243a
add command line option
vishesh Feb 26, 2023
e900283
Clang-format
vishesh Feb 26, 2023
39f389e
Merge pull request #11 from vishesh/restore
jzhou77 Feb 26, 2023
3fb864d
Merge pull request #9483 from jzhou77/restore
jzhou77 Feb 27, 2023
481d4ed
Fix -t flag bug for fdbdecode (#9489)
jzhou77 Feb 27, 2023
6593fa7
Fix fdbbackup query returning earliest version
halfprice Feb 26, 2023
dcdee8c
Query backup size from a specific snapshot
halfprice Feb 26, 2023
20c663b
clean format
halfprice Feb 26, 2023
62c3bf2
Explicitly using min and max restorable version from backup descripti…
halfprice Feb 26, 2023
f281be0
fix clang build error
halfprice Feb 26, 2023
e86adab
Add more comments in fdbbackup query command, and address comments
halfprice Feb 27, 2023
de6496f
Change PTreeImpl::insert to overwrite existing entries (#9138)
sfc-gh-anoyes Jan 18, 2023
1ae5b1f
Avoid repeated search in VersionedMap::erase(iterator) (#9143)
sfc-gh-anoyes Jan 27, 2023
038ee58
Merge branch 'release-7.1' into zhewu/backup-size-estimate-7.1
halfprice Feb 27, 2023
5737512
Use KeyspaceSnapshotFile to filter range files
jzhou77 Feb 28, 2023
88d0f12
Merge pull request #9508 from jzhou77/release-7.1
jzhou77 Feb 28, 2023
e3397c2
Merge pull request #9506 from halfprice/zhewu/backup-size-estimate-7.1
jzhou77 Feb 28, 2023
c12e0f1
Change mutation and KV logging to SevInfo
jzhou77 Feb 28, 2023
23702f1
Output in HEX format for easy regex matching
jzhou77 Feb 28, 2023
f89610e
Merge branch 'release-7.1' into restore
jzhou77 Feb 28, 2023
bf9c642
Merge pull request #9511 from jzhou77/restore
jzhou77 Mar 1, 2023
8559e5c
Refactor decoder to read file as a whole once
jzhou77 Mar 3, 2023
b59b4ab
Add more trace events
jzhou77 Mar 3, 2023
ac51a07
Merge pull request #9560 from jzhou77/restore
jzhou77 Mar 6, 2023
02eb4fc
Allow log router to detect slow peeks and to switch DC for peeking [r…
jzhou77 Mar 10, 2023
13a98ab
Fix issue where the versions on seed storage servers decreased
sfc-gh-ljoswiak Dec 2, 2022
dd379cc
Added 7.1.28 and 7.1.29 release notes
jzhou77 Mar 13, 2023
c7b524c
Reduce running time for ClogTlog
jzhou77 Mar 8, 2023
4688bb2
Merge pull request #9668 from jzhou77/release-7.1
jzhou77 Mar 13, 2023
c52b18f
Remove profile code from SpecialKeySpace workload
jzhou77 Mar 13, 2023
cb90dd3
Merge branch 'release-7.1' into 7.1-fix-specialkeyspace
xis19 Mar 14, 2023
a60d512
disable AVX for 7.1.28 release
foundationdb-ci Mar 14, 2023
1b2517a
enable AVX and update version for 7.1.29 release
foundationdb-ci Mar 14, 2023
2443616
Merge branch 'release-7.1' into 7.1-fix-specialkeyspace
xis19 Mar 14, 2023
7d3044d
Merge pull request #9681 from jzhou77/7.1-fix-specialkeyspace
jzhou77 Mar 15, 2023
c3ad9b5
update version after 7.1.29 release
foundationdb-ci Mar 15, 2023
5815d89
Update info trigger new DB info update immediately
halfprice Mar 22, 2023
e86a9ab
Merge pull request #9778 from jzhou77/release-7.1
jzhou77 Mar 23, 2023
205f58b
Backport exclusion fix #9468 (#9789)
johscheuer Mar 23, 2023
a121a93
Fix finishedQueries metric, add metrics reporting in GetMappedRange …
flowguru Mar 23, 2023
4537535
Adding rocksdb bloom filter knobs. (#9770)
neethuhaneesha Mar 23, 2023
0bfab1f
[Release 7.1] Do not update exclude/failed system metadata in exclude…
halfprice Mar 27, 2023
50816eb
Parameterized queue length in GetMappedRange test (#9808)
flowguru Mar 28, 2023
3e641a8
Add 7.1.30, 7.1.31 release notes (#9822)
jzhou77 Mar 29, 2023
21b1cdf
Don't stop iterating over all storage processes in exclusion check (#…
johscheuer Apr 2, 2023
d4ffd35
checkSafeExclusion should always create new ExclusionSafetyCheckReque…
halfprice Apr 2, 2023
4808747
RocksDB 7.10.2 version upgrade (#9829)
neethuhaneesha Apr 3, 2023
6af1ee6
Changing single key deletions to delete based on number of deletes in…
neethuhaneesha Mar 31, 2023
d4e12f6
Merge pull request #9882 from neethuhaneesha/bytesample-7.1
jzhou77 Apr 4, 2023
f4ee02a
Implement check if locality is already excluded in exclude locality c…
halfprice Apr 4, 2023
451a8ae
Merge pull request #9814 from sbodagala/main (#9883)
sbodagala Apr 4, 2023
d0cb8cd
Update 7.1.30 release notes
jzhou77 Apr 3, 2023
f3ae0c7
Remove printable() from TSS trace events
jzhou77 Apr 4, 2023
af31ee2
Fix releast notes
jzhou77 Apr 4, 2023
068bc8b
Merge branch 'release-7.1' into release-7.1
jzhou77 Apr 4, 2023
b87ce2f
Merge pull request #9886 from jzhou77/release-7.1
jzhou77 Apr 4, 2023
c6e4a3d
Fixed stuck data movement when a server is removed [release-7.1] (#9904)
jzhou77 Apr 5, 2023
bfa0673
disable AVX for 7.1.30 release
foundationdb-ci Apr 6, 2023
94e42d5
enable AVX and update version for 7.1.31 release
foundationdb-ci Apr 6, 2023
3c27831
Merge remote-tracking branch 'upstream/release-7.1' into ow-fork-7.1
Apr 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ else()
endif()

project(foundationdb
VERSION 7.1.29
VERSION 7.1.31
DESCRIPTION "FoundationDB is a scalable, fault-tolerant, ordered key-value store with full ACID transactions."
HOMEPAGE_URL "http://www.foundationdb.org/"
LANGUAGES C CXX ASM)
Expand Down
6 changes: 3 additions & 3 deletions cmake/CompileRocksDB.cmake
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# FindRocksDB

find_package(RocksDB 7.7.3)
find_package(RocksDB 7.10.2)

include(ExternalProject)

Expand Down Expand Up @@ -54,8 +54,8 @@ if(ROCKSDB_FOUND)
${BINARY_DIR}/librocksdb.a)
else()
ExternalProject_Add(rocksdb
URL https://github.com/facebook/rocksdb/archive/refs/tags/v7.7.3.tar.gz
URL_HASH SHA256=b8ac9784a342b2e314c821f6d701148912215666ac5e9bdbccd93cf3767cb611
URL https://github.com/facebook/rocksdb/archive/refs/tags/v7.10.2.tar.gz
URL_HASH SHA256=4619ae7308cd3d11cdd36f0bfad3fb03a1ad399ca333f192b77b6b95b08e2f78
CMAKE_ARGS ${RocksDB_CMAKE_ARGS}
BUILD_BYPRODUCTS <BINARY_DIR>/librocksdb.a
INSTALL_COMMAND ""
Expand Down
17 changes: 17 additions & 0 deletions documentation/sphinx/source/release-notes/release-notes-710.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@
Release Notes
#############

7.1.31
======
* Same as 7.1.30 release with AVX enabled.

7.1.30
======
* Released with AVX disabled.
* Fixed storage server finishedQueries metric when using getMappedRange. `(PR #9785) <https://github.com/apple/foundationdb/pull/9785>`_
* Fixed unnecessary transaction system recovery when excluding the servers that are already excluded/failed. `(PR #9809) <https://github.com/apple/foundationdb/pull/9809>`_ and `(PR #9878) <https://github.com/apple/foundationdb/pull/9878>`_
* Fixed the exclusion of stateless processes by skipping the free capacity check. `(PR #9789) <https://github.com/apple/foundationdb/pull/9789>`_ and `(PR #9769) <https://github.com/apple/foundationdb/pull/9769>`_
* Fixed an issue where the new worker cannot get ServerDBInfo update. `(PR #9778) <https://github.com/apple/foundationdb/pull/9778>`_
* Added RocksDB bloom filter knobs. `(PR #9770) <https://github.com/apple/foundationdb/pull/9770>`_
* Upgraded RocksDB to version 7.10.2. `(PR #9829) <https://github.com/apple/foundationdb/pull/9829>`_
* Fixed an issue where ExclusionSafetyCheckRequest could be blocked forever. `(PR #9871) <https://github.com/apple/foundationdb/pull/9871>`_
* Fixed fdbserver not able to join the cluster if the majority of coordinators in its connection string have failed. `(PR #9883) <https://github.com/apple/foundationdb/pull/9883>`_
* Fixed stuck data movement when moving to removed a storage server. `(PR #9904) <https://github.com/apple/foundationdb/pull/9904>`_

7.1.29
======
* Same as 7.1.28 release with AVX enabled.
Expand Down
140 changes: 100 additions & 40 deletions fdbclient/ManagementAPI.actor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1124,7 +1124,7 @@ struct AutoQuorumChange final : IQuorumChange {
desiredCount = redundancy * 2 - 1;
}

std::vector<AddressExclusion> excl = wait(getExcludedServers(tr));
std::vector<AddressExclusion> excl = wait(getAllExcludedServers(tr));
state std::set<AddressExclusion> excluded(excl.begin(), excl.end());

std::vector<ProcessData> _workers = wait(getWorkers(tr));
Expand Down Expand Up @@ -1273,23 +1273,37 @@ Reference<IQuorumChange> autoQuorumChange(int desired) {
return Reference<IQuorumChange>(new AutoQuorumChange(desired));
}

void excludeServers(Transaction& tr, std::vector<AddressExclusion>& servers, bool failed) {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
tr.setOption(FDBTransactionOptions::USE_PROVISIONAL_PROXIES);
std::string excludeVersionKey = deterministicRandom()->randomUniqueID().toString();
auto serversVersionKey = failed ? failedServersVersionKey : excludedServersVersionKey;
tr.addReadConflictRange(singleKeyRange(serversVersionKey)); // To conflict with parallel includeServers
tr.set(serversVersionKey, excludeVersionKey);
ACTOR Future<Void> excludeServers(Transaction* tr, std::vector<AddressExclusion> servers, bool failed) {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
tr->setOption(FDBTransactionOptions::USE_PROVISIONAL_PROXIES);
std::vector<AddressExclusion> excl = wait(failed ? getExcludedFailedServerList(tr) : getExcludedServerList(tr));
std::set<AddressExclusion> exclusions(excl.begin(), excl.end());
bool containNewExclusion = false;
for (auto& s : servers) {
if (exclusions.find(s) != exclusions.end()) {
continue;
}
containNewExclusion = true;
if (failed) {
tr.set(encodeFailedServersKey(s), StringRef());
tr->set(encodeFailedServersKey(s), StringRef());
} else {
tr.set(encodeExcludedServersKey(s), StringRef());
tr->set(encodeExcludedServersKey(s), StringRef());
}
}
TraceEvent("ExcludeServersCommit").detail("Servers", describe(servers)).detail("ExcludeFailed", failed);

if (containNewExclusion) {
std::string excludeVersionKey = deterministicRandom()->randomUniqueID().toString();
auto serversVersionKey = failed ? failedServersVersionKey : excludedServersVersionKey;
tr->addReadConflictRange(singleKeyRange(serversVersionKey)); // To conflict with parallel includeServers
tr->set(serversVersionKey, excludeVersionKey);
}
TraceEvent("ExcludeServersCommit")
.detail("Servers", describe(servers))
.detail("ExcludeFailed", failed)
.detail("ExclusionUpdated", containNewExclusion);
return Void();
}

ACTOR Future<Void> excludeServers(Database cx, std::vector<AddressExclusion> servers, bool failed) {
Expand Down Expand Up @@ -1321,7 +1335,7 @@ ACTOR Future<Void> excludeServers(Database cx, std::vector<AddressExclusion> ser
state Transaction tr(cx);
loop {
try {
excludeServers(tr, servers, failed);
wait(excludeServers(&tr, servers, failed));
wait(tr.commit());
return Void();
} catch (Error& e) {
Expand All @@ -1333,23 +1347,36 @@ ACTOR Future<Void> excludeServers(Database cx, std::vector<AddressExclusion> ser
}

// excludes localities by setting the keys in api version below 7.0
void excludeLocalities(Transaction& tr, std::unordered_set<std::string> localities, bool failed) {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
tr.setOption(FDBTransactionOptions::USE_PROVISIONAL_PROXIES);
std::string excludeVersionKey = deterministicRandom()->randomUniqueID().toString();
auto localityVersionKey = failed ? failedLocalityVersionKey : excludedLocalityVersionKey;
tr.addReadConflictRange(singleKeyRange(localityVersionKey)); // To conflict with parallel includeLocalities
tr.set(localityVersionKey, excludeVersionKey);
ACTOR Future<Void> excludeLocalities(Transaction* tr, std::unordered_set<std::string> localities, bool failed) {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
tr->setOption(FDBTransactionOptions::USE_PROVISIONAL_PROXIES);
std::vector<std::string> excl = wait(failed ? getExcludedFailedLocalityList(tr) : getExcludedLocalityList(tr));
std::set<std::string> exclusion(excl.begin(), excl.end());
bool containNewExclusion = false;
for (const auto& l : localities) {
if (exclusion.find(l) != exclusion.end()) {
continue;
}
containNewExclusion = true;
if (failed) {
tr.set(encodeFailedLocalityKey(l), StringRef());
tr->set(encodeFailedLocalityKey(l), StringRef());
} else {
tr.set(encodeExcludedLocalityKey(l), StringRef());
tr->set(encodeExcludedLocalityKey(l), StringRef());
}
}
TraceEvent("ExcludeLocalitiesCommit").detail("Localities", describe(localities)).detail("ExcludeFailed", failed);
if (containNewExclusion) {
std::string excludeVersionKey = deterministicRandom()->randomUniqueID().toString();
auto localityVersionKey = failed ? failedLocalityVersionKey : excludedLocalityVersionKey;
tr->addReadConflictRange(singleKeyRange(localityVersionKey)); // To conflict with parallel includeLocalities
tr->set(localityVersionKey, excludeVersionKey);
}
TraceEvent("ExcludeLocalitiesCommit")
.detail("Localities", describe(localities))
.detail("ExcludeFailed", failed)
.detail("ExclusionUpdated", containNewExclusion);
return Void();
}

// Exclude the servers matching the given set of localities from use as state servers.
Expand Down Expand Up @@ -1384,7 +1411,7 @@ ACTOR Future<Void> excludeLocalities(Database cx, std::unordered_set<std::string
state Transaction tr(cx);
loop {
try {
excludeLocalities(tr, localities, failed);
wait(excludeLocalities(&tr, localities, failed));
wait(tr.commit());
return Void();
} catch (Error& e) {
Expand Down Expand Up @@ -1617,19 +1644,26 @@ ACTOR Future<Void> setClass(Database cx, AddressExclusion server, ProcessClass p
}
}

ACTOR Future<std::vector<AddressExclusion>> getExcludedServers(Transaction* tr) {
ACTOR Future<std::vector<AddressExclusion>> getExcludedServerList(Transaction* tr) {
state RangeResult r = wait(tr->getRange(excludedServersKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r.more && r.size() < CLIENT_KNOBS->TOO_MANY);
state RangeResult r2 = wait(tr->getRange(failedServersKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r2.more && r2.size() < CLIENT_KNOBS->TOO_MANY);

std::vector<AddressExclusion> exclusions;
for (auto i = r.begin(); i != r.end(); ++i) {
auto a = decodeExcludedServersKey(i->key);
if (a.isValid())
exclusions.push_back(a);
}
for (auto i = r2.begin(); i != r2.end(); ++i) {
uniquify(exclusions);
return exclusions;
}

ACTOR Future<std::vector<AddressExclusion>> getExcludedFailedServerList(Transaction* tr) {
state RangeResult r = wait(tr->getRange(failedServersKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r.more && r.size() < CLIENT_KNOBS->TOO_MANY);

std::vector<AddressExclusion> exclusions;
for (auto i = r.begin(); i != r.end(); ++i) {
auto a = decodeFailedServersKey(i->key);
if (a.isValid())
exclusions.push_back(a);
Expand All @@ -1638,50 +1672,76 @@ ACTOR Future<std::vector<AddressExclusion>> getExcludedServers(Transaction* tr)
return exclusions;
}

ACTOR Future<std::vector<AddressExclusion>> getExcludedServers(Database cx) {
ACTOR Future<std::vector<AddressExclusion>> getAllExcludedServers(Transaction* tr) {
state std::vector<AddressExclusion> exclusions;
std::vector<AddressExclusion> excludedServers = wait(getExcludedServerList(tr));
exclusions.insert(exclusions.end(), excludedServers.begin(), excludedServers.end());
std::vector<AddressExclusion> excludedFailed = wait(getExcludedFailedServerList(tr));
exclusions.insert(exclusions.end(), excludedFailed.begin(), excludedFailed.end());
uniquify(exclusions);
return exclusions;
}

ACTOR Future<std::vector<AddressExclusion>> getAllExcludedServers(Database cx) {
state Transaction tr(cx);
loop {
try {
tr.setOption(FDBTransactionOptions::READ_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE); // necessary?
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
std::vector<AddressExclusion> exclusions = wait(getExcludedServers(&tr));
std::vector<AddressExclusion> exclusions = wait(getAllExcludedServers(&tr));
return exclusions;
} catch (Error& e) {
wait(tr.onError(e));
}
}
}

// Get the current list of excluded localities by reading the keys.
ACTOR Future<std::vector<std::string>> getExcludedLocalities(Transaction* tr) {
ACTOR Future<std::vector<std::string>> getExcludedLocalityList(Transaction* tr) {
state RangeResult r = wait(tr->getRange(excludedLocalityKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r.more && r.size() < CLIENT_KNOBS->TOO_MANY);
state RangeResult r2 = wait(tr->getRange(failedLocalityKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r2.more && r2.size() < CLIENT_KNOBS->TOO_MANY);

std::vector<std::string> excludedLocalities;
for (const auto& i : r) {
auto a = decodeExcludedLocalityKey(i.key);
excludedLocalities.push_back(a);
}
for (const auto& i : r2) {
uniquify(excludedLocalities);
return excludedLocalities;
}

ACTOR Future<std::vector<std::string>> getExcludedFailedLocalityList(Transaction* tr) {
state RangeResult r = wait(tr->getRange(failedLocalityKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!r.more && r.size() < CLIENT_KNOBS->TOO_MANY);

std::vector<std::string> excludedLocalities;
for (const auto& i : r) {
auto a = decodeFailedLocalityKey(i.key);
excludedLocalities.push_back(a);
}
uniquify(excludedLocalities);
return excludedLocalities;
}

ACTOR Future<std::vector<std::string>> getAllExcludedLocalities(Transaction* tr) {
state std::vector<std::string> exclusions;
std::vector<std::string> excludedLocalities = wait(getExcludedLocalityList(tr));
exclusions.insert(exclusions.end(), excludedLocalities.begin(), excludedLocalities.end());
std::vector<std::string> failedLocalities = wait(getExcludedFailedLocalityList(tr));
exclusions.insert(exclusions.end(), failedLocalities.begin(), failedLocalities.end());
uniquify(exclusions);
return exclusions;
}

// Get the list of excluded localities by reading the keys.
ACTOR Future<std::vector<std::string>> getExcludedLocalities(Database cx) {
ACTOR Future<std::vector<std::string>> getAllExcludedLocalities(Database cx) {
state Transaction tr(cx);
loop {
try {
tr.setOption(FDBTransactionOptions::READ_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
std::vector<std::string> exclusions = wait(getExcludedLocalities(&tr));
std::vector<std::string> exclusions = wait(getAllExcludedLocalities(&tr));
return exclusions;
} catch (Error& e) {
wait(tr.onError(e));
Expand Down
26 changes: 19 additions & 7 deletions fdbclient/ManagementAPI.actor.h
Original file line number Diff line number Diff line change
Expand Up @@ -65,12 +65,12 @@ Reference<IQuorumChange> nameQuorumChange(std::string const& name, Reference<IQu
// necessarily waiting for the servers to be evacuated. A NetworkAddress with a port of 0 means all servers on the
// given IP.
ACTOR Future<Void> excludeServers(Database cx, std::vector<AddressExclusion> servers, bool failed = false);
void excludeServers(Transaction& tr, std::vector<AddressExclusion>& servers, bool failed = false);
ACTOR Future<Void> excludeServers(Transaction* tr, std::vector<AddressExclusion> servers, bool failed = false);

// Exclude the servers matching the given set of localities from use as state servers. Returns as soon as the change
// is durable, without necessarily waiting for the servers to be evacuated.
ACTOR Future<Void> excludeLocalities(Database cx, std::unordered_set<std::string> localities, bool failed = false);
void excludeLocalities(Transaction& tr, std::unordered_set<std::string> localities, bool failed = false);
ACTOR Future<Void> excludeLocalities(Transaction* tr, std::unordered_set<std::string> localities, bool failed = false);

// Remove the given servers from the exclusion list. A NetworkAddress with a port of 0 means all servers on the given
// IP. A NetworkAddress() means all servers (don't exclude anything)
Expand All @@ -86,13 +86,25 @@ ACTOR Future<Void> includeLocalities(Database cx,
// the given IP.
ACTOR Future<Void> setClass(Database cx, AddressExclusion server, ProcessClass processClass);

// Get the current list of excluded servers
ACTOR Future<std::vector<AddressExclusion>> getExcludedServers(Database cx);
ACTOR Future<std::vector<AddressExclusion>> getExcludedServers(Transaction* tr);
// Get the current list of excluded servers including both "exclude" and "failed".
ACTOR Future<std::vector<AddressExclusion>> getAllExcludedServers(Database cx);
ACTOR Future<std::vector<AddressExclusion>> getAllExcludedServers(Transaction* tr);

// Get the current list of excluded servers.
ACTOR Future<std::vector<AddressExclusion>> getExcludedServerList(Transaction* tr);

// Get the current list of failed servers.
ACTOR Future<std::vector<AddressExclusion>> getExcludedFailedServerList(Transaction* tr);

// Get the current list of excluded localities
ACTOR Future<std::vector<std::string>> getExcludedLocalities(Database cx);
ACTOR Future<std::vector<std::string>> getExcludedLocalities(Transaction* tr);
ACTOR Future<std::vector<std::string>> getAllExcludedLocalities(Database cx);
ACTOR Future<std::vector<std::string>> getAllExcludedLocalities(Transaction* tr);

// Get the current list of excluded localities.
ACTOR Future<std::vector<std::string>> getExcludedLocalityList(Transaction* tr);

// Get the current list of failed localities.
ACTOR Future<std::vector<std::string>> getExcludedFailedLocalityList(Transaction* tr);

std::set<AddressExclusion> getAddressesByLocality(const std::vector<ProcessData>& workers, const std::string& locality);

Expand Down
3 changes: 1 addition & 2 deletions fdbclient/NativeAPI.actor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8264,7 +8264,6 @@ ACTOR Future<bool> checkSafeExclusions(Database cx, std::vector<AddressExclusion
TraceEvent("ExclusionSafetyCheckBegin")
.detail("NumExclusion", exclusions.size())
.detail("Exclusions", describe(exclusions));
state ExclusionSafetyCheckRequest req(exclusions);
state bool ddCheck;
try {
loop {
Expand All @@ -8273,7 +8272,7 @@ ACTOR Future<bool> checkSafeExclusions(Database cx, std::vector<AddressExclusion
when(ExclusionSafetyCheckReply _ddCheck =
wait(basicLoadBalance(cx->getCommitProxies(UseProvisionalProxies::False),
&CommitProxyInterface::exclusionSafetyCheckReq,
req,
ExclusionSafetyCheckRequest(exclusions),
cx->taskID))) {
ddCheck = _ddCheck.safe;
break;
Expand Down
6 changes: 6 additions & 0 deletions fdbclient/ServerKnobs.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,10 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
init( ROCKSDB_UNSAFE_AUTO_FSYNC, false );
init( ROCKSDB_PERIODIC_COMPACTION_SECONDS, 0 );
init( ROCKSDB_PREFIX_LEN, 0 );
init( ROCKSDB_MEMTABLE_PREFIX_BLOOM_SIZE_RATIO, 0.1 );
init( ROCKSDB_BLOOM_BITS_PER_KEY, 10 );
init( ROCKSDB_BLOOM_WHOLE_KEY_FILTERING, false );
init( ROCKSDB_MAX_AUTO_READAHEAD_SIZE, 0 );
// If rocksdb block cache size is 0, the default 8MB is used.
int64_t blockCacheSize = isSimulated ? 16 * 1024 * 1024 : 2147483648 /* 2GB */;
init( ROCKSDB_BLOCK_CACHE_SIZE, blockCacheSize );
Expand Down Expand Up @@ -417,7 +421,9 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi
// These knobs have contrary functionality.
init( ROCKSDB_SINGLEKEY_DELETES_ON_CLEARRANGE, true );
init( ROCKSDB_SINGLEKEY_DELETES_BYTES_LIMIT, 10000 ); // 10KB
init( ROCKSDB_SINGLEKEY_DELETES_MAX, 200 ); // Max rocksdb::delete calls in a transaction
init( ROCKSDB_ENABLE_CLEAR_RANGE_EAGER_READS, false );
init( ROCKSDB_FORCE_DELETERANGE_FOR_CLEARRANGE, false );
// ROCKSDB_STATS_LEVEL=1 indicates rocksdb::StatsLevel::kExceptHistogramOrTimers
// Refer StatsLevel: https://github.com/facebook/rocksdb/blob/main/include/rocksdb/statistics.h#L594
init( ROCKSDB_STATS_LEVEL, 1 ); if( randomize && BUGGIFY ) ROCKSDB_STATS_LEVEL = deterministicRandom()->randomInt(0, 6);
Expand Down
6 changes: 6 additions & 0 deletions fdbclient/ServerKnobs.h
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,10 @@ class ServerKnobs : public KnobsImpl<ServerKnobs> {
bool ROCKSDB_UNSAFE_AUTO_FSYNC;
int64_t ROCKSDB_PERIODIC_COMPACTION_SECONDS;
int ROCKSDB_PREFIX_LEN;
double ROCKSDB_MEMTABLE_PREFIX_BLOOM_SIZE_RATIO;
int ROCKSDB_BLOOM_BITS_PER_KEY;
bool ROCKSDB_BLOOM_WHOLE_KEY_FILTERING;
int ROCKSDB_MAX_AUTO_READAHEAD_SIZE;
int64_t ROCKSDB_BLOCK_CACHE_SIZE;
double ROCKSDB_METRICS_DELAY;
double ROCKSDB_READ_VALUE_TIMEOUT;
Expand Down Expand Up @@ -338,7 +342,9 @@ class ServerKnobs : public KnobsImpl<ServerKnobs> {
bool ROCKSDB_DISABLE_WAL_EXPERIMENTAL;
bool ROCKSDB_SINGLEKEY_DELETES_ON_CLEARRANGE;
int64_t ROCKSDB_SINGLEKEY_DELETES_BYTES_LIMIT;
int ROCKSDB_SINGLEKEY_DELETES_MAX;
bool ROCKSDB_ENABLE_CLEAR_RANGE_EAGER_READS;
bool ROCKSDB_FORCE_DELETERANGE_FOR_CLEARRANGE;
bool ROCKSDB_ENABLE_COMPACT_ON_DELETION;
int64_t ROCKSDB_CDCF_SLIDING_WINDOW_SIZE; // CDCF: CompactOnDeletionCollectorFactory
int64_t ROCKSDB_CDCF_DELETION_TRIGGER; // CDCF: CompactOnDeletionCollectorFactory
Expand Down
Loading