Skip to content

Remove parallel restore feature#12903

Merged
gxglass merged 5 commits into
apple:mainfrom
gxglass:delete-parallel-restore
Apr 1, 2026
Merged

Remove parallel restore feature#12903
gxglass merged 5 commits into
apple:mainfrom
gxglass:delete-parallel-restore

Conversation

@gxglass

@gxglass gxglass commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator

This parallel restore feature has been slated for removal for at least a year. This PR is closely based on earlier PR #12107.

This blog post explains some of the problems with the parallel restore feature: https://medium.com/@jingyuzhou/why-foundationdb-restore-is-slow-and-what-can-be-done-about-it-e73a821fdd33

As far as large feature removal changes go, this one is very straightforward, with most relevant files and test cases simply being deleted. There is one knob rename where storageserver.actor.cpp was using a knob with FASTRESTORE in the name. Other than that, changes to shared files mainly involve removing fastrestore-specific CLI and role support.

In progress:
20260330-222511-gglass-5ee0142213471b70 compressed=True data_size=35343375 duration=4611964 ended=100000 fail=1 fail_fast=1000 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:58:23 sanity=False started=100000 stopped=20260330-232334 submitted=20260330-222511 timeout=5400 username=gglass

The one failure was in SwizzledCycleTest.toml with too many lines of output and a timeout. I kind of suspect it's unrelated but haven't looked further.

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: d4543b8
  • Duration 0:23:07
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: d4543b8
  • Duration 0:35:25
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: d4543b8
  • Duration 0:35:31
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: d4543b8
  • Duration 0:41:38
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: d4543b8
  • Duration 0:43:23
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: d4543b8
  • Duration 0:47:06
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: d4543b8
  • Duration 1:59:34
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@gxglass

gxglass commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator Author

This seems to be failing ctests. Moving to Draft

10/52 Test #10: BackupTests ....................................... Passed 0.00 sec
Start 11: dir_backup_tests
11/52 Test #11: dir_backup_tests ..................................***Failed 39.78 sec
Start 12: s3_backup_tests
12/52 Test #12: s3_backup_tests ...................................***Failed 78.23 sec
Start 13: s3_backup_bulkdump_bulkload_tests
13/52 Test #13: s3_backup_bulkdump_bulkload_tests .................***Failed 145.53 sec
Start 14: test_venv_setup
14/52 Test #14: test_venv_setup ................................... Passed 6.47 sec
Start 15: multiversion_client/unit_tests

@gxglass gxglass marked this pull request as draft March 31, 2026 01:00
@gxglass

gxglass commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator Author

Late adopter I realize but below is what the local AI says about this PR.

It is not able to read the github comment since that overrides the warning about the old "compiles but untested" diff comment which it (reasonably) flags as item 6. This is either by local policy or by my own misconfiguration. Other than that there are some good ideas in here. On the dead code item (1 in list) it doesn't realize that the function using this knob (weightedTotalSize) itself is dead code.

PR 12903 Review: Remove Parallel Restore Feature

Base: fdb5 local repo (branch delete-parallel-restore, main at ~PR 12887)
Scope: ~9,650 lines deleted across 50 files, ~60 lines added


What It Does

Completely removes the FastRestore/ParallelRestore subsystem — a distributed parallel backup restore implementation that was written but never shipped. Specifically deleted:

  • All fdbserver/restoreworker/ source — ~5,500 lines: RestoreWorker, RestoreController, RestoreLoader, RestoreApplier, RestoreRoleCommon, RestoreCommon, RestoreUtil, and their
    interfaces/headers
  • fdbserver/core/RestoreCoreUtil.cpp/.h — utility code for the restore workers
  • fdbbackup/backup.cpp — removes runFastRestoreTool(), the fastrestore_tool binary entry point, and ProgramExe::FASTRESTORE_TOOL
  • fdbclient/FileBackupAgent.cpp — removes parallelRestoreFinish(), submitParallelRestore(), and atomicParallelRestore() from FileBackupAgent
  • fdbclient/include/fdbclient/BackupAgent.h — removes the three parallel restore method declarations and transformRestoredDatabase()
  • fdbserver/fdbserver.actor.cpp — removes the ServerRole::Restore branch, the FastRestoreClass process dispatch path, and RestoreWorkerInterface.h include
  • fdbclient/ServerKnobs.cpp/.h — removes ~30 FASTRESTORE_* knobs
  • fdbclient/ClientKnobs.cpp — removes FASTRESTORE_ATOMICOP_WEIGHT init
  • Workloads — removes BackupAndParallelRestoreCorrectness.cpp (~800 lines), ParallelRestore.cpp (RunRestoreWorkerWorkload), and AtomicRestore.cpp fastRestore option
  • Tests — removes 10 simulation test specs (.toml/.txt)
  • Docker/CMake — removes fastrestore_tool from build targets and Docker image

Intent

This is dead-code removal. The parallel restore feature was written, had its own workers, knobs, and test suite, but was never production-ready. PR 12107 was an earlier attempt at this
removal; this is a rebased/updated version of that work.


Potential Problems

  1. FASTRESTORE_ATOMICOP_WEIGHT knob is orphaned — this is a likely compile or runtime error.

The knob is declared in ClientKnobs.h:205 and actively used in CommitTransaction.h:121:
return totalSize() * CLIENT_KNOBS->FASTRESTORE_ATOMICOP_WEIGHT;
But its init(...) call was removed from ClientKnobs.cpp. The member still exists in the struct — meaning it will be uninitialized at runtime. This is either a crash waiting to happen or a
silent correctness bug depending on the zero-initialization behavior of the struct. This needs to be removed from both ClientKnobs.h and CommitTransaction.h, or the init line needs to
stay.

  1. RestoreInterface.cpp/.h and fdbclient/RestoreInterface.h are not deleted.

RestoreInterface.h declares keys (restoreRequestDoneKey, restoreRequestTriggerKey, restoreRequestKeys) and functions (restoreRequestTriggerValue, restoreRequestKeyFor) that are now dead
code. FileBackupAgent.cpp still #includes RestoreInterface.h (line 43) but no longer uses any of its symbols. SystemData.cpp:1515 also defines restoreRequestDoneKey — which now duplicates
the definition in RestoreInterface.cpp. Both files define the same key with the same value; this is a latent ODR issue even if it compiles cleanly today.

  1. atomicRestore() in FileBackupAgent.cpp still has a live fastRestore parameter with ASSERT(false) — it's a stub, not a removal.

The UsePartitionedLog fastRestore parameter remains in the function signature. Any caller passing fastRestore=true will hit ASSERT(false) at runtime (in debug builds) or undefined
behavior (in release). The callers in AtomicRestore.cpp were cleaned up, but the dead branch and parameter should be fully removed since the feature is gone.

  1. FastRestoreClass in Locality.h/.cpp is marked // deprecated with ASSERT(false) but not removed.

The enum value stays (with a static_assert preserving its numeric value of 11), and the switch case now hits ASSERT(false). This is intentional tombstoning of the enum value to avoid
renumbering — which is correct for a wire protocol concern — but it's worth a comment clarifying that the numeric value 11 must never be reused.

  1. ProcessClass::FastRestoreClass dispatch in fdbserver.actor.cpp is removed, but existing clusters may have processes with this class.

If any running cluster (or a running simulation test from an older build) sends a worker with processClass == FastRestoreClass, the server will no longer handle it specially — it'll fall
through to the generic fdbd path instead of hitting ASSERT(false). This is likely fine in practice (the class was never used in production) but worth noting.

  1. Commit message says "Compiles but untested."

No CI results are referenced. The FASTRESTORE_ATOMICOP_WEIGHT issue suggests it may not even compile cleanly.

⏺ The fix restores a single if block that was accidentally deleted when removing the FASTRESTORE_TOOL code:

  if (!restoreSystemKeys && !restoreUserKeys && backupKeys.empty()) {
      addDefaultBackupRanges(backupKeys);
  }

  When no explicit key ranges are specified on the command line and neither --user-data nor --system-metadata flags are set, this populates backupKeys with the default backup ranges
  (essentially all user data). Without it, backupKeys stays empty and hits the ASSERT(!backupRanges.empty()) in submitBackup().
@gxglass gxglass marked this pull request as ready for review March 31, 2026 22:25
@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 2e79ba4
  • Duration 0:23:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@gxglass

gxglass commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator Author

Updated AI code review:

Item 1 — Duplicate symbol: retracted. restoreRequestDoneKey is no longer in SystemData.cpp. It only exists in RestoreInterface.cpp. No duplicate. However RestoreInterface.cpp/.h are still
compiled into fdbclient as dead code — FileBackupAgent.cpp includes the header but uses none of its symbols. Low risk, just untidy.

Item 2 — fastRestore dead parameter: retracted. Reading the actual code, atomicRestore() no longer has a fastRestore parameter at all — the PR cleanly removed it. The ASSERT(false) branch
is gone. This issue no longer exists.

Item 3 — FastRestoreClass tombstone: partially confirmed. The enum is marked // deprecated and the Locality.cpp switch hits ASSERT(false) then returns "fast_restore". Notably, the
adjacent RemovedClassPlaceholder1 shows the project's established pattern for this — they rename to RemovedClassPlaceholderN rather than keeping the original name with a comment.
FastRestoreClass could be treated consistently. Minor cosmetic issue only.

Summary: The two issues I flagged as significant (duplicate symbol, dead parameter) don't exist in the current code. The only real remaining item is the dead RestoreInterface.cpp/.h
files, and the minor inconsistency in how FastRestoreClass is tombstoned vs the RemovedClassPlaceholder1 pattern.

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 2e79ba4
  • Duration 0:42:28
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 2e79ba4
  • Duration 0:48:04
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 2e79ba4
  • Duration 0:56:27
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: c8fe400
  • Duration 0:46:14
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 3274959
  • Duration 0:46:19
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: c8fe400
  • Duration 0:51:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 3274959
  • Duration 0:49:13
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: d925ea2
  • Duration 0:43:26
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 0:45:13
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 2e79ba4
  • Duration 2:02:46
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: c8fe400
  • Duration 1:44:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 3274959
  • Duration 1:43:03
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 1:36:04
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 1:36:54
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 3274959
  • Duration 1:45:45
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: c8fe400
  • Duration 1:48:40
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 3274959
  • Duration 2:26:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 2:20:14
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: c8fe400
  • Duration 2:32:40
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@gxglass gxglass closed this Apr 1, 2026
@gxglass

gxglass commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator Author

@gxglass gxglass reopened this Apr 1, 2026
@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 0:21:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: d925ea2
  • Duration 0:34:43
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: d925ea2
  • Duration 0:42:20
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: d925ea2
  • Duration 0:47:32
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 0:59:35
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 1:00:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: d925ea2
  • Duration 1:53:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@saintstack saintstack left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed. It is great. No references to fast restore in documentation?

@gxglass

gxglass commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator Author

@neethuhaneesha @akankshamahajan15 can you confirm none of this code is needed for the backup v3 stuff you are working on?

@gxglass

gxglass commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator Author

Skimmed. It is great. No references to fast restore in documentation?

Apparently not. There weren't any doc edits in the original PR I copied this from, AI didn't tell me about any, and a quick grep doesn't show anything.

@akankshamahajan15 akankshamahajan15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@neethuhaneesha neethuhaneesha left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@gxglass gxglass merged commit bca167f into apple:main Apr 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants