Add logging to LogRouter's waitForVersion function (cherry pick #11359)#11370
Merged
Conversation
* Add logging to waitForVersion * Respond to review comments. --------- Co-authored-by: Dan Lambright <hlambright@apple.com>
Contributor
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Contributor
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Contributor
Result of foundationdb-pr-clang on Linux CentOS 7
|
Contributor
Result of foundationdb-pr on Linux CentOS 7
|
Contributor
Result of foundationdb-pr-cluster-tests on Linux CentOS 7
|
jzhou77
approved these changes
May 7, 2024
sbodagala
approved these changes
May 8, 2024
MarkSh1
added a commit
to owtech/foundationdb
that referenced
this pull request
Apr 17, 2025
* Fix an invalid memory access Found by Valgrind * Fix guess region from s3 URL * Reducing canCommit wait to avoid bigger spikes on pendinng compaction bytes (apple#10945) * fix: Non-storage processes were not being checked for locality exclusions fix: Data distribution when not detect a newly added process was locality excluded fix: RemoveServerSafely did not wait for processes to be excluded before killing them when excluding localities * fix: do not allow locality based excludes if they cannot exclude the required addresses * fixup! Fix build failures * Fix strtol return type and error checking Added an overflow test as well. * fix-redistribute-cli * auto hot storage server detect * address comments * Fix trackExcludedServers to use ExclusionTracker * Add 7.1.42, 7.1.43 release notes (apple#10981) * add write traffic to ddMetricsGetRange * Fix unknown key in DataDistributionMetrics test * Compaction rate limiter changes. * enable DataDistributionMetrics test * Perpetual wiggle option to have multiple SS in rebalance state during wiggling. * Fix exclude status of machines in status json (apple#10996) Previously, `status json` would report a machine as excluded when any of its processes were excluded. But a machine should only be reported as excluded when all of its processes are excluded. I tested this change by running a small, three process test cluster. I excluded one machine, and verified `status json` reported the overall machine as not excluded. Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com> * Fix an assertion failure in tests Restore the default value for knob peek_tracker_expiration_time. Failed seed: -f ./tests/slow/MoveKeysSideband.toml -s 669512178 -b on commit: a3dbd4b * Fix RemoveServersSafely workload timeout error This workload can have timeout error when using locality-based exclusion. The sequence is: 1. RemoveServerSafely workload exclude locality by processid 2. Attrition reboots the target process, thus changing the processid, because processid is generated for each worker process at fdbd() 3. RemoveServerSafely waits for the process exclusion, which never succeed 4. Timeout The fix monitors processid locality changes and reissue the exclusion with the correct locality. To reproduce: seed: -f ./tests/fast/SwizzledRollbackSideband.toml -s 879108103 -b on commit: a3dbd4b release-7.1 * Fix force flag for exclusion * Correct the check for a machine being excluded * Make sure the get all excluded servers method returns servers that are excluded based on locality * Issue get worker request concurrently * Make sure server list is validated against the excluded localites * Add missing fdbcli util method * Add missing fdbcli util method * Update 7.1.42 release notes (apple#11029) * Allow the exclusions of localities that are not matching any process * disable AVX for 7.1.42 release * enable AVX and update version for 7.1.43 release * update version after 7.1.43 release * fix-consistency-check-issues * fix * allow real time cancel * allow real time cancel * add trace length * Merge pull request apple#11091 from sbodagala/main Address issues in code related to "fdbcli checkall" command * print ConsistencyCheck_ShardWrongPlaceSkipped * refine trace event * handle a corner case * disable AVX for 7.1.44 release * enable AVX and update version for 7.1.45 release * Added logs for invalid mutations. * update version after 7.1.45 release * fmt. * Add key ranges as filters for fdbdecode In the filter file, now we can use key ranges like "\x01\x05-\x01\x06" as the filter, in addition to prefix filters. * retriable consistency checker * repair DC when retry too many times * Range-Customizable Consistency Checker * disable AVX for 7.1.46 release * enable AVX and update version for 7.1.47 release * update version after 7.1.47 release * Merge pull request apple#11098 from sbodagala/main Bump up the number of keys that "fdbcli checkall" can fetch per GetKeyValueRequest call * improved-getKeyLocationsForRangeList-and-one-time-run * urgent mode * fmt * skip entire shard if any subkeys failed to check * turn on CONSISTENCY_CHECK_URGENT_MODE * input-multiple-ranges-to-check * comment change * fix user input range * fix knob val * fix NumShardToCheckThisEpoch counter * disable AVX for 7.1.48 release * enable AVX and update version for 7.1.49 release * update version after 7.1.49 release * stateful consistency checker * add to simulation test * bypass consistencycheck command * fix bugs * fix bugs * introduce random failure of consistency checker * simplifying consistency checker urgent workload * more trace event * better logic of runConsistencyCheckerUrgentCore * better spec * address comments * addressed comments * make consistency checker operations serializable * Fix checkall when shard is large the begin key has to be updated for checkall command when a shard is large, this PR makes this change. * report error in dispatchMonitorUrgentConsistencyCheckWorkload * nit * Change comment * nit * code clean * fix bugs * add trace of read range failed * guarantee complete check without persisting metadata * code cleanup and nits * tester should notify checker if it gives up retrying * tolerate when CC returns stale testers interface * nits * fmt * disable AVX for 7.1.50 release * enable AVX and update version for 7.1.51 release * update version after 7.1.51 release * consistency checker urgent should tolerate busy restart loop of the checker * fix a simulation failure due to setup error * disable AVX for 7.1.52 release * enable AVX and update version for 7.1.53 release * update version after 7.1.53 release * fix check all command * fix-checkall-false-alarm-bug * address comments * address comments and nits * fix-check-all-block-issue * address comments * fix tester stale interface issue at CC * Allow AsyncFileWriteChecker history.timestamp to have better precision * Update value to be hex output So we can use the output easily for fdbcli and analyzing scripts. * Use hex for checkall command output * Enable DD verbose tracing by default. * Fix complie issue for ALLOC_INSTRUMENTATION (apple#11164) * Upgrade rocksdb to 8.6.7 --------- Co-authored-by: Yao Xiao <yxiao6@apple.com> Upgrade rocksdb to 8.6.7 Upgrade rocksdb to 8.6.7 * add sha to opentest4j download * Fix the compile warning in KeyValueStoreRocksDB.actor.cpp * Make FoundationDB Compilable Again * Removing the type check unit test (apple#11168) * disable AVX for 7.1.54 release * enable AVX and update version for 7.1.55 release * Add 7.1.44 to 7.1.55 release notes (apple#11183) * update version after 7.1.55 release * fix find toml in release-7.1 to avoid macos pr failures * Upgrade to RocksDB 8.10.0. * [RELEASE-7.1] Abort process when abnormal shutdown is initiated to allow coredumps to be generated (apple#11191) * Abort process when abnormal shutdown is initiated to allow coredumps to be generated * Update link to abort function * Add knob to allow fdbserver to abort under abnormal behaviour * Fix single consistency checker (apple#11205) * reproduce the issue * revert single consistency checker back * fix sev error in distributed consistency checker (apple#11203) * Allow to disable the removal of maintenance mode when a SS outside of the maintenance zone fails (apple#11207) * disable AVX for 7.1.56 release * enable AVX and update version for 7.1.57 release * update version after 7.1.57 release * Add 7.1.56 to 7.1.57 release notes [release-7.1] (apple#11209) * cherrypick pr 11229 (apple#11230) * Update doctest to version 2.4.8 * Fix a DR corruption bug The clear in status call can cause corruptions. To reproduce with clang build: seed: -f ./tests/slow/ApiCorrectnessSwitchover.toml -s 223736449 -b on commit: f8eca6a Apply the ./tests/slow/ApiCorrectnessSwitchover.toml file in this commit. * Fix clang compiling errors * disable AVX for 7.1.58 release * enable AVX and update version for 7.1.59 release * update version after 7.1.59 release * align script with main branch * cherry picked https://github.com/DataDog/foundationdb/commits/griffin/go-tenant-impl/ * 720 -> 710 * tenant/map -> tenant_map * Address some PR review comments. * update copyright notice. * update fdbkubernetesmonitor to match main, and add fdb-kubernetes-monitor to the standard build flow * Cherry pick apple#11309 Raise visibility of gray failure actions (apple#11314) * Cherry pick apple#11308 Raise visibility of gray failure actions * format change --------- Co-authored-by: Dan Lambright <hlambright@apple.com> * update go bindings build to play nice with golang 1.22 * Increase CommitProxyTerminated severity for failed_to_progress errors. (apple#11316) For better visibility. * Throw errors in getConsistentReadVersion (apple#11352) In the current code, errors are retried in getConsistentReadVersion, so it's possible that the client has cancelled the GRV request, but readVersionBatcher continue retrying, which can lead to many clients DDoS GRV proxies, especially when the database has become unavailable for a while and clients are issuing many GRV requests. * cherry-pick-improve-dcc (apple#11351) * Add logging to LogRouter's waitForVersion function (apple#11359) (apple#11370) * Add logging to waitForVersion * Respond to review comments. --------- Co-authored-by: Dan Lambright <hlambright@apple.com> * [Release 7.1] Disable TestOpenNotExistTenant test (apple#11380) * Disable fdb_test.go:TestOpenNotExistTenant This is not supported yet. * fixup! Fix the comments * Fix an assertion failure when waiting for recovery CC's checkBetterSingletons() calls getUsedIds() that asserts proxy interfaces are present. However, when a GRV/commit proxy failed, before CC starts a new recovery, the proxy's processId becomes empty, thus triggering the failure. The fix is to cancel the caller while waiting for recovery. To reproduce 7.1 commit 725a08a clang build: ./fdbserver.6.0.15 -r simulation -f ./tests/restarting/from_5.2.0_until_6.3.0/ClientTransactionProfilingCorrectness-1.txt -s 900000399 -b on -f ./tests/restarting/from_5.2.0_until_6.3.0/ClientTransactionProfilingCorrectness-2.txt --restarting -s 900000400 -b on * Add default timeout for fdbbackup * Correct format * Correct imports for release-7.1 * Make it so can run the ycsb container standalone without need of a k8s context (useful for running ycsb against bare metal cluster). * packaging/docker/Dockerfile * packaging/docker/Dockerfile.eks Make the ycsb target inherit from foundationdb-base so we pick up libfdb_c.so. Add in a version of run_ycsb.sh that doesn't presume k8s. Use 'entrypoint' rather than 'cmd' so can override on 'docker run'. * packaging/docker/run_ycsb_standalone.sh Version of run_ycsb.sh w/o the presumption of k8s. * disable AVX for 7.1.60 release * enable AVX and update version for 7.1.61 release * update version after 7.1.61 release * Add 7.1.58 to 7.1.61 release notes (apple#11405) * Correct the path where the fdb-kubernetes-monitor copies the binary into when running in sidecar mode * Revert variable renaming * Improve checkall tool (apple#11440) * improve checkall * fmt * simplify * nit * simplify * nit * Add exponential backoff mechanism for restarting fdbserver processes in the monitor (apple#11452) * Remove some steps are only needed once * fdbkubernetesmonitor: Update the pod annotation if the cluster file changes * Initial add of custom prometheus metrics * Add support in the fdb-kubernetes-monitor to read node labels * Restructure approach to be compatible with operator setup * Correct name of the flag * Add unit tests for PodClient * only setup node informer if node watch feature is enabeld * Add additional tests for PodClient * Update format * Add client metric test(release-7.1) (apple#11466) * Add client metric test This restarting test start with 7.1 version and setup the sample rate for transaction log, then it test with 7.3 version and verify transaction log are still being written. This test use latency as the sample transaction log metric. Now only code is added, but it is not run by simulation. It will be run as part of the restarting test in simulation in 7.3 * add toml This test should also be run in 7.1 but not as restarting test, to validate its correctness. * Consolidate test code * Fix format issue * Add a mechanism to allow to specifiy the command line flags over env variables (apple#11462) * add TLS ability to fdb kubernetes monitor * review fixes * move certloader to internal folder * fdbkubernetesmonitor: Retry update pod annotations in case of an error (apple#11458) * Add support for the isolate process group annotation to shutdown fdbserver processes (apple#11464) * Add support for the isolate process group annotation to shutdown fdbserver processes * fdb-kubernetes-monitor: Ensure that the annotations are updated when the correct configuration is already loaded * patch image for building after EOL * cherrypick Add consistency-check-urgent-mode to tester process class (apple#11503) * lower the sample rate of client metric in simulation (apple#11524) * disable AVX for 7.1.62 release * enable AVX and update version for 7.1.63 release * [RELEASE-7.1] Change the default base image to RockyLinux9 (apple#11550) * Change the default base image to RockyLinux9 * Update the sidecar install commands * Update the python verson used for the sidecar * update version after 7.1.63 release * Add more trace events for exclude command Use PRIORITY_SYSTEM_IMMEDIATE for excludeServersAndLocalities() call. * Fix a cluster controller crash in status server The logSystemConfig could be changed after the yield(), thus the iterator can points to invalid memory. To reproduce with gcc build at commit 3e90644: -f tests/slow/DDBalanceAndRemoveStatus.toml -s 3134372275 -b off * Fix fdbbackup crashes [release-7.1] (apple#11591) * Capture default database options in fdbbackup in a local variable. * Add debug logging to help validate the transaction options set in fdbbackup. * Reformat changes. * Rename fields in new trace events to match formatting standards. * Fix references to the wrong DB in the backup actor initialization. (apple#11588) * Fix references to the wrong DB in the backup actor initialization. * Fix variable name. * Apply clang-format updates. --------- Co-authored-by: John Brownlee <john_brownlee@apple.com> * Fix TracedTooManyLines in simulation (apple#11659) * disable AVX for 7.1.64 release * enable AVX and update version for 7.1.65 release * fixed boost download url --------- Co-authored-by: Hao Fu <77984096+hfu94@users.noreply.github.com> Co-authored-by: Jingyu Zhou <jingyu_zhou@apple.com> Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com> Co-authored-by: w41ter <w41ter.l@gmail.com> Co-authored-by: neethuhaneesha <nbingi@apple.com> Co-authored-by: Evan Tschannen <evan.tschannen@snowflake.com> Co-authored-by: Xiaoge Su <magichp@gmail.com> Co-authored-by: Zhe Wang <zhe.wang@wustl.edu> Co-authored-by: Johannes Scheuermann <johscheuer@users.noreply.github.com> Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com> Co-authored-by: Johannes M. Scheuermann <jscheuermann@apple.com> Co-authored-by: FoundationDB CI <foundationdb_ci@apple.com> Co-authored-by: He Liu <heliu05023@gmail.com> Co-authored-by: hao fu <hfu5@apple.com> Co-authored-by: Aaron Molitor <ammolitor@users.noreply.github.com> Co-authored-by: Aaron Molitor <amolitor@apple.com> Co-authored-by: Yao Xiao <yxiao6@apple.com> Co-authored-by: Griffin Dunn <griffin.dunn@datadoghq.com> Co-authored-by: Matthew Newhook <matthew@customer.io> Co-authored-by: Dan Lambright <dlambrig@gmail.com> Co-authored-by: Dan Lambright <hlambright@apple.com> Co-authored-by: stack <stack@apache.org> Co-authored-by: Nicole Morales <nicole_morales@apple.com> Co-authored-by: John Brownlee <john_brownlee@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add logging to waitForVersion
Respond to review comments.
20240506-225105-dlambrig-26746fe25a2b23c8