Enhance fdbbackup query command to estimate data processing from a specific snapshot to a target version by halfprice · Pull Request #9485 · apple/foundationdb

halfprice · 2023-02-26T01:40:51Z

Previously, it always uses the closest snapshot from the restore version. Now, it can uses --query-restore-snapshot-version to set which snapshot version to use. It will pick the latest snapshot version prior to that version.
Also fix --query-restore-version == -2, which doesn't produce minimum restore version (rather, it produces just the latest snapshot).
Introduce earliestVersion constant.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2023-02-26T01:45:26Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 8a88df0
Duration 0:04:23
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T01:45:29Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 8a88df0
Duration 0:04:29
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T01:45:30Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 8a88df0
Duration 0:04:27
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T01:46:50Z

Result of foundationdb-pr-macos on macOS Monterey 12.x

Commit ID: 8a88df0
Duration 0:05:48
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T01:47:01Z

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Commit ID: 8a88df0
Duration 0:05:57
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T01:48:04Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 8a88df0
Duration 0:07:02
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-02-26T02:25:01Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 92a90c7
Duration 0:10:48
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdb-windows-ci · 2023-02-26T02:39:00Z

Doxense CI Report for Windows 10

Commit ID: 8a88df0
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

fdb-windows-ci · 2023-02-26T02:47:33Z

Doxense CI Report for Windows 10

Commit ID: 92a90c7
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-02-26T02:49:29Z

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Commit ID: 92a90c7
Duration 0:35:19
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T02:54:09Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 92a90c7
Duration 0:39:58
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-02-26T03:02:59Z

Result of foundationdb-pr-macos on macOS Monterey 12.x

Commit ID: 92a90c7
Duration 0:48:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T03:04:46Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 92a90c7
Duration 0:50:35
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T03:09:03Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 92a90c7
Duration 0:54:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2023-02-26T04:40:31Z

+		if (targetVersion == earliestVersion) {
+			i = 0;


This doesn't work with the logic below, i.e., loop at line 933. There we do i--, so if 0 is not desirable, we won't try next ones like 1, 2, ... So the change needed is to change the loop to be i++ if earliestVersion is the target version.

Both way may not work because the snapshot at head or tail may be incomplete.

So I changed the behavior to explicitly selection the restore version before entering this command based on backup description. PTAL.

…on in query command in stead of going throw snapshots

foundationdb-ci · 2023-02-26T20:35:25Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: ffa3467
Duration 0:10:06
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T20:52:52Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 4fd3a90
Duration 0:13:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdb-windows-ci · 2023-02-26T20:53:37Z

Doxense CI Report for Windows 10

Commit ID: 4fd3a90
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

foundationdb-ci · 2023-02-26T20:59:38Z

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Commit ID: ffa3467
Duration 0:34:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:01:18Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: ffa3467
Duration 0:36:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:04:46Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: ffa3467
Duration 0:39:26
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-02-26T21:10:27Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: ffa3467
Duration 0:45:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:14:38Z

Result of foundationdb-pr-macos on macOS Monterey 12.x

Commit ID: ffa3467
Duration 0:49:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:25:30Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 4fd3a90
Duration 0:46:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:25:54Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 4fd3a90
Duration 0:46:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-26T21:27:50Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 4fd3a90
Duration 0:48:41
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77 · 2023-02-27T16:45:51Z

 	TraceEvent("BackupQueryFailure").detail("OperationId", operationId).detail("Reason", errorMessage);
 }

+std::pair<Version, Version> getMaxMinRestorableVersions(const BackupDescription& desc, bool mayOnlyApplyMutationLog) {


I don't understand why mayOnlyApplyMutationLog is !keyRangesFilter.empty().

I don't understand that either. The code simply preserves previous behavior. Looks like if backup doesn't have a restorable version, and it has KeyRangeFilter set, it just tries to use the mutation log only.

I don't see this logis is used anywhere else. I can also remove it.

I'm fine either way. If we keep this logic, maybe add a comment on this.

I added a comment. This option (hopefully) won't be used in our case.

jzhou77 · 2023-02-27T17:32:16Z

+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter)));
+		} else {
+			// We only need to know all the log files from snapshotVersion to restoreVersion.
+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter, /*logOnly=*/true, snapshotVersion)));


how different is this from 2801? can we consolidate these two calls?

the 2801 code block is to calculate backup data "up to the user-configured snapshot version".
this code back adds up the backup data from snapshot version to restorable version.

virtual Future<Optional> getRestoreSet(Version targetVersion,
VectorRef keyRangesFilter = {},
bool logsOnly = false,
Version beginVersion = -1) = 0;

I restructured the code to make it more readable. PTAL.

xumengpanda · 2023-02-27T17:38:19Z

 	       "                 version approximately NUM_DAYS days worth of versions prior to the latest log version in "
 	       "the backup.\n");
+	printf("  --query-restore-snapshot-version VERSION\n"
+	       "                 For query operations, set snapshot version for restoring a backup. Set -1 for maximum\n"


" For query operations, set snapshot version for restoring a backup. " ->
" For query operations, set the snapshot version, inclusive, used to restore a backup. "

for better readability

" Set -1 for maximumrestorable version (default) and -3 for minimum restorable version."
Change to
"
Set -1 to use the latest valid snapshot,
Set -3 to use the oldest valid snapshot.
"

xumengpanda · 2023-02-27T17:49:46Z

+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter)));
+		} else {
+			// We only need to know all the log files from snapshotVersion to restoreVersion.
+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter, /*logOnly=*/true, snapshotVersion)));


the 2801 code block is to calculate backup data "up to the user-configured snapshot version".
this code back adds up the backup data from snapshot version to restorable version.

xumengpanda · 2023-02-27T17:49:57Z

+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter)));
+		} else {
+			// We only need to know all the log files from snapshotVersion to restoreVersion.
+			wait(store(fileSet, bc->getRestoreSet(restoreVersion, keyRangesFilter, /*logOnly=*/true, snapshotVersion)));


virtual Future<Optional> getRestoreSet(Version targetVersion,
VectorRef keyRangesFilter = {},
bool logsOnly = false,
Version beginVersion = -1) = 0;

foundationdb-ci · 2023-02-27T18:36:17Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: c5c434e
Duration 0:35:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-27T18:50:34Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: c5c434e
Duration 0:49:30
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2023-02-27T18:51:36Z

Result of foundationdb-pr-macos on macOS Monterey 12.x

Commit ID: c5c434e
Duration 0:50:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-27T18:52:10Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: c5c434e
Duration 0:51:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2023-02-27T18:56:28Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: c5c434e
Duration 0:55:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

fdb-windows-ci · 2023-02-27T21:38:11Z

Doxense CI Report for Windows 10

Commit ID: c5c434e
Result: ✔️ SUCCEEDED
Build Logs (available for 30 days)

halfprice added 2 commits February 25, 2023 16:44

Fix fdbbackup query returning earliest version

a94dd3a

Query backup size from a specific snapshot

8a88df0

clean format

92a90c7

jzhou77 requested changes Feb 26, 2023

View reviewed changes

Explicitly using min and max restorable version from backup descripti…

ffa3467

…on in query command in stead of going throw snapshots

fix clang build error

4fd3a90

halfprice marked this pull request as ready for review February 26, 2023 20:39

halfprice requested review from jzhou77 and xumengpanda February 26, 2023 20:39

jzhou77 reviewed Feb 27, 2023

View reviewed changes

halfprice requested a review from jzhou77 February 27, 2023 17:20

jzhou77 reviewed Feb 27, 2023

View reviewed changes

xumengpanda reviewed Feb 27, 2023

View reviewed changes

Add more comments in fdbbackup query command, and address comments

c5c434e

halfprice requested review from jzhou77 and xumengpanda February 27, 2023 18:02

jzhou77 approved these changes Feb 27, 2023

View reviewed changes

halfprice merged commit 7304e5c into apple:main Feb 27, 2023

halfprice deleted the zhewu/backup-size-estimate branch February 27, 2023 21:35

This was referenced Feb 27, 2023

[Release 7.2] Enhance fdbbackup query command to estimate data processing from a specific snapshot to a target version #9504

Merged

[Release 7.1] Enhance fdbbackup query command to estimate data processing from a specific snapshot to a target version #9506

Merged

Uh oh!

Conversation

halfprice commented Feb 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

fdb-windows-ci commented Feb 26, 2023

Doxense CI Report for Windows 10

Uh oh!

fdb-windows-ci commented Feb 26, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

fdb-windows-ci commented Feb 26, 2023

Doxense CI Report for Windows 10

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-macos on macOS Monterey 12.x

Uh oh!

foundationdb-ci commented Feb 26, 2023

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

halfprice commented Feb 26, 2023 •

edited

Loading