Skip to content

[main] Make getTeamByServers O(1) in time (#12938)#13012

Merged
spraza merged 2 commits into
apple:mainfrom
spraza:fwd-port-getTeamByServers-main
May 19, 2026
Merged

[main] Make getTeamByServers O(1) in time (#12938)#13012
spraza merged 2 commits into
apple:mainfrom
spraza:fwd-port-getTeamByServers-main

Conversation

@spraza

@spraza spraza commented Apr 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Forward port of #12938.

Under certain workloads (large data movement, storage migration, etc.), with SHARD_ENCODE_LOCATION_METADATA enabled, DD can get indefinitely stuck at initialization time, because getTeamByServers function saturates the CPU and starves other critical DD actors to complete (symptom: txn_too_old).

This PR is a perf optimization that makes getTeamByServers O(1) instead of O(teams). Previously, for every team, we were doing expensive CPU operations.

Testing and analysis

100K results:

20260518-182541-praza-fwd-port-getTeamBySer-e4f98f2296c5aa71 compressed=True data_size=35941392 duration=4250904 ended=100000 fail=3 fail_fast=10 max_runs=100000 pass=99997 priority=100 remaining=0 runtime=1:05:02 sanity=False started=100000 stopped=20260518-193043 submitted=20260518-182541 timeout=5400 username=praza-fwd-port-getTeamByServers-main-20453d9676098857888b1d862830b25dadbbe55e

3 failures:

[1] TestFile=tests/slow/WriteDuringReadSwitchover.toml  RandomSeed=2666891645  BuggifyEnabled=1
    UnseedMismatch  Unseed=48713  ExpectedUnseed=72392  Severity=40

[2] TestFile=tests/fast/InventoryTestSomeWrites.toml  RandomSeed=389468833  BuggifyEnabled=1
    QuietDatabaseFailure  Severity=40  ErrorKind=Unset  Time=22458.953045  DateTime=2026-05-18T18:54:10Z  Type=QuietDatabaseFailure  Machine=[abcd::2:3:1:0]:1  ID=0000000000000000  Error=test_failed  ErrorDescription=Test failed  ErrorCode=1228  EventName=TracedTooManyLines  EventMessage=Traced 1000001 lines  Reasons=[no items]  ThreadID=12501697803017906063  Backtrace=/usr/local/bin/llvm-addr2line -e /var/joshua/ensembles/20260518-182541-praza-fwd-port-getTeamBySer-e4f98f2296c5aa71/bin/fdbserver -p -C -f -i 0x4f488af 0x4f48ba9 0x346c90d 0x4f1af6d 0x4f42d50 0x227dd1b 0x15dba68 0x4788237 0x4787d60 0x16028b1 0x7fa44dc06610  LogGroup=default  Roles=TL

[3] TestFile=tests/fast/SidebandWithStatus.toml  RandomSeed=508357272  BuggifyEnabled=1
    QuietDatabaseFailure  Severity=40  ErrorKind=Unset  Time=15200.705780  DateTime=2026-05-18T19:30:21Z  Type=QuietDatabaseFailure  Machine=[abcd::2:1:1:0]:1  ID=0000000000000000  Error=test_failed  ErrorDescription=Test failed  ErrorCode=1228  EventName=TracedTooManyLines  EventMessage=Traced 1000001 lines  Reasons=[no items]  ThreadID=10891599674997048062  Backtrace=/usr/local/bin/llvm-addr2line -e /var/joshua/ensembles/20260518-182541-praza-fwd-port-getTeamBySer-e4f98f2296c5aa71/bin/fdbserver -p -C -f -i 0x4f488af 0x4f48ba9 0x346c90d 0x4f1af6d 0x4f42d50 0x473e987 0x15dba68 0x4788237 0x4787d60 0x16028b1 0x7fdb47be1610  LogGroup=default  Roles=CC,CP,RV,SS,TL

Investigating.

re (1):

I don't see an obvious way I am introducing non-determinism. Let me run this test 250K times, with my change, and without my change. I'll analyze if it's my change introducing non-determinism or it's there for this test to begin with.

running tests/slow/WriteDuringReadSwitchover.toml, 250K with fail fast of 10

with my change: 20260518-195437-praza-fwd-port-getTeamBySer-0eabfa61e1710d55 compressed=True data_size=35954619 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260518-195437 timeout=5400 username=praza-fwd-port-getTeamByServers-main-2def2427c3455c2dd02f76c32bbc730c71baf4dc

without my change: 20260518-200016-praza-fwd-port-getTeamBySer-651b00370962678e compressed=True data_size=35953413 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260518-200016 timeout=5400 username=praza-fwd-port-getTeamByServers-main-7462a3ade10dc699b20e5f4e3a89117896537885

re (2):

/t/r1 $ grep -ri "Type=\"Knob\".*shard_encode_location_metadata" trace.0.0.0.0.2695399.1779132312.eP2Ofz.0.1.xml
<Event Severity="10" Time="0.000000" DateTime="2026-05-18T19:25:12Z" Type="Knob" Machine="0.0.0.0:0" ID="0000000000000000" Name="shard_encode_location_metadata" Value="0" ThreadID="12501697803017906063" LogGroup="default" />

So shard_encode_location_metadata is off. My change should have no effect including on determinism. Easiest and quickest way to confirm: revert my change and expect same failure.

250K with my change: 20260519-190248-praza-fwd-port-getTeamBySer-dae6d43bf770574a compressed=True data_size=35954495 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260519-190248 timeout=5400 username=praza-fwd-port-getTeamByServers-main-2def2427c3455c2dd02f76c32bbc730c71baf4dc

250K without my change: 20260519-191053-praza-fwd-port-getTeamBySer-4c26026f397f51a8 compressed=True data_size=35952831 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260519-191053 timeout=5400 username=praza-fwd-port-getTeamByServers-main-revert-7462a3ade10dc699b20e5f4e3a89117896537885

I believe the issue is not related to my change here. It has to do with recovery and LR peeks, probably similar to #13238. Regardless see table [1] below which confirms that the tracedtoomanylines symptom exists in the baseline (without my changes) code as well.

re (3):

/t/r1 $ grep -ri "Type=\"Knob\".*shard_encode_location_metadata" trace.0.0.0.0.2698823.1779133284.IOrbEB.0.1.xml
<Event Severity="10" Time="0.000000" DateTime="2026-05-18T19:41:24Z" Type="Knob" Machine="0.0.0.0:0" ID="0000000000000000" Name="shard_encode_location_metadata" Value="1" ThreadID="10891599674997048062" LogGroup="default" />

So could be my change.

250K with my change: 20260519-190345-praza-fwd-port-getTeamBySer-3f716bfae8fca260 compressed=True data_size=35954485 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260519-190345 timeout=5400 username=praza-fwd-port-getTeamByServers-main-2def2427c3455c2dd02f76c32bbc730c71baf4dc

250K without my change: 20260519-191040-praza-fwd-port-getTeamBySer-3dec3fa1685072e3 compressed=True data_size=35953095 fail_fast=10 max_runs=250000 priority=100 sanity=False submitted=20260519-191040 timeout=5400 username=praza-fwd-port-getTeamByServers-main-revert-7462a3ade10dc699b20e5f4e3a89117896537885

I believe the issue is not related to my change here. See table [1] below which confirms that the tracedtoomanylines symptom exists in the baseline (without my changes) code as well.

[1]:

Findings: across all 4 ensembles, every recorded failure is TracedTooManyLines in InventoryTestSomeWrites.toml or SidebandWithStatus.toml:

Ensemble Branch Test Failures captured ended / 250k
4c26026f revert (no change) InventoryTestSomeWrites 3 - all TracedTooManyLines 82,577
dae6d43b with change InventoryTestSomeWrites 3 - all TracedTooManyLines 95,044
3dec3fa1 revert (no change) SidebandWithStatus 3 - all TracedTooManyLines 75,604
3f716bfa with change SidebandWithStatus 1 - TracedTooManyLines 86,336

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@spraza spraza marked this pull request as draft April 16, 2026 23:26
@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 0:44:10
  • Result: ❌ FAILED
  • Error: Error while executing command: TIDY_DIFF=/usr/local/share/clang/clang-tidy-diff.py; if [ ! -f .clang-tidy ]; then echo "No .clang-tidy config found — skipping static analysis"; elif [ ! -f "$TIDY_DIFF" ]; then echo "WARNING: clang-tidy-diff.py not found at $TIDY_DIFF — skipping static analysis"; else echo "Running clang-tidy on changed files"; git fetch --depth=1 origin "${CODEBUILD_WEBHOOK_BASE_REF}"; git diff -U0 FETCH_HEAD..HEAD | grep -v -E "(sqlite3\.amalgamation\.c|\.actor\.cpp)" | python3 "$TIDY_DIFF" -p 1 -j ${NPROC} -path build_output -quiet -extra-arg=-Wno-unknown-warning-option; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 648cb8f
  • Duration 0:45:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 0:46:01
  • Result: ❌ FAILED
  • Error: Error while executing command: TIDY_DIFF=/usr/local/share/clang/clang-tidy-diff.py; if [ ! -f .clang-tidy ]; then echo "No .clang-tidy config found — skipping static analysis"; elif [ ! -f "$TIDY_DIFF" ]; then echo "WARNING: clang-tidy-diff.py not found at $TIDY_DIFF — skipping static analysis"; else echo "Running clang-tidy on changed files"; git fetch --depth=1 origin "${CODEBUILD_WEBHOOK_BASE_REF}"; git diff -U0 FETCH_HEAD..HEAD | grep -v -E "(sqlite3\.amalgamation\.c|\.actor\.cpp)" | python3 "$TIDY_DIFF" -p 1 -j ${NPROC} -path build_output -quiet -extra-arg=-Wno-unknown-warning-option; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 0:52:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack saintstack closed this Apr 17, 2026
@saintstack saintstack reopened this Apr 17, 2026
@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 648cb8f
  • Duration 0:42:43
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 0:54:23
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 1:09:21
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 648cb8f
  • Duration 3:00:11
  • Result: ❌ FAILED
  • Error: Build has timed out.
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@spraza
Make getTeamByServers O(1) in time (apple#12938)
20453d9 
* Make getTeamByServers O(1) in time

* address feedback
@spraza spraza force-pushed the fwd-port-getTeamByServers-main branch from 648cb8f to 20453d9 Compare May 18, 2026 18:26
@spraza spraza marked this pull request as ready for review May 18, 2026 18:27
gxglass
gxglass reviewed May 18, 2026
View reviewed changes
Comment thread fdbserver/core/include/fdbserver/core/Knobs.h Outdated
@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 20453d9
  • Duration 0:25:08
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 20453d9
  • Duration 0:32:38
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 20453d9
  • Duration 0:40:28
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 2def242
  • Duration 0:25:21
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 20453d9
  • Duration 0:45:02
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 20453d9
  • Duration 0:45:46
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 2def242
  • Duration 0:42:58
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 20453d9
  • Duration 1:02:13
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 2def242
  • Duration 0:44:51
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 20453d9
  • Duration 1:06:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 2def242
  • Duration 0:49:49
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 2def242
  • Duration 1:06:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 2def242
  • Duration 1:08:03
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 18, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 2def242
  • Duration 2:39:41
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@spraza spraza merged commit 534ac1f into apple:main May 19, 2026
7 checks passed
@spraza spraza mentioned this pull request May 19, 2026
Merged
spraza added a commit to spraza/foundationdb that referenced this pull request May 19, 2026
@spraza
Make getTeamByServers O(1) in time (apple#12938) (apple#13012)
b113d62 
* Make getTeamByServers O(1) in time (apple#12938)

* Make getTeamByServers O(1) in time

* address feedback

* address feedback
@spraza spraza changed the title Make getTeamByServers O(1) in time (#12938) [main] Make getTeamByServers O(1) in time (#12938) May 19, 2026
spraza added a commit that referenced this pull request May 21, 2026
@spraza
Make getTeamByServers O(1) in time (#12938) (#13012) (#13244)
615ae19 
* Make getTeamByServers O(1) in time (#12938)

* Make getTeamByServers O(1) in time

* address feedback

* address feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gxglass gxglass gxglass approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@spraza @foundationdb-ci @gxglass @saintstack