5s - introduce fuzzed write transaction delay knob for simulation by spraza · Pull Request #12464 · apple/foundationdb

spraza · 2025-10-15T22:19:26Z

Description

Changes:

fuzz write knob in [1, 10] range. More weight given to 5s. Same logic as 5s - fuzz read knob value in simulation #12440.
previously client and server manually set the same knob value since the value has to be the same. Now client picks a random value (see above), and server picks whatever client had picked. Note that in general we set knobs in this order in simulation: flow -> client -> server.
pass isSimulated to client knobs since I pick a random value only under simulation
change override logic in simulation for knobs with same name in client and server. Old logic: if client knob FOO exists, it's updated, but server knob FOO is not. New logic: client knob FOO is updated, and server knob FOO is also updated.
misc header cleanup. Note that I started deleting unneeded headers only from files I was touching, that then showed errors in other files which incorrectly were relying on header inclusion from elsewhere, so I had to fix these other files as well.

Testing

250K: 20251104-062454-praza-5s-milestone1-iter34--341964d848d7cf63 compressed=True data_size=37405675 duration=13264619 ended=250000 fail=1 fail_fast=10 max_runs=250000 pass=249999 priority=100 remaining=0 runtime=2:12:16 sanity=False started=250000 stopped=20251104-083710 submitted=20251104-062454 timeout=5400 username=praza-5s-milestone1-iter34-2f30599007142c33e0ffeedd9ac4c98eb6c1d8fc.

The 1 failure is: fdbserver -r simulation -f /root/src/foundationdb/tests/slow/SwizzledCycleTest.toml --buggify on --seed 195531608. TLDR: I don't think it's related to this change, see below. I am still investigating this issue though to see if it needs a separate fix.

Sev40:

<Event Severity="40" ErrorKind="BugDetected" Time="261.204304" DateTime="2025-11-05T00:06:18Z" Type="InternalError" Machine="[abcd::2:1:1:3]:1" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" FailedAssertion="seconds &gt;= -0.0001" File="/root/src/foundationdb/fdbrpc/sim2.actor.cpp" Line="1101" ThreadID="12773931573902802642" Backtrace="/usr/local/bin/llvm-addr2line -e /root/cnd_build_output/bin/fdbserver -p -C -f -i 0x5b57c9f 0x5b57f99 0x5b52184 0x5a75cec 0x53f3531 0x53dcfdd 0x2fc36f6 0x2fc3d3d 0x2fc2c06 0x1d5eaf8 0x1ef9a46 0x2fd0355 0x2fcff5e 0x1d5eaf8 0x21bc67f 0x34e428d 0x34cab3d 0x20df15c 0x52f54d4 0x52f4dcc 0x1d5eaf8 0x53fcd37 0x53fc860 0x31e0eaa 0x7f956719b5d0" LogGroup="default" Roles="LR,SS,TL" />
<Event Severity="40" ErrorKind="Unset" Time="261.204304" DateTime="2025-11-05T00:06:18Z" Type="SystemError" Machine="[abcd::2:1:1:3]:1" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="12773931573902802642" Backtrace="/usr/local/bin/llvm-addr2line -e /root/cnd_build_output/bin/fdbserver -p -C -f -i 0x5b57c9f 0x5b57f99 0x5b52184 0x5a76130 0x5a75cff 0x53f3531 0x53dcfdd 0x2fc36f6 0x2fc3d3d 0x2fc2c06 0x1d5eaf8 0x1ef9a46 0x2fd0355 0x2fcff5e 0x1d5eaf8 0x21bc67f 0x34e428d 0x34cab3d 0x20df15c 0x52f54d4 0x52f4dcc 0x1d5eaf8 0x53fcd37 0x53fc860 0x31e0eaa 0x7f956719b5d0" LogGroup="default" Roles="LR,SS,TL" />

This is happening because of this code:

foundationdb/fdbrpc/sim2.actor.cpp

Lines 1097 to 1098 in b720c6d

    
           Future<class Void> delay(double seconds, TaskPriority taskID, ProcessInfo* machine, bool ordered = false) { 
        
           	ASSERT(seconds >= -0.0001);

I verified adding a trace event that the value is less than -0.0001:

<Event Severity="10" Time="261.204304" DateTime="2025-11-05T00:06:18Z" Type="Foo1" Machine="[abcd::2:1:1:3]:1" ID="0000000000000000" Seconds="-0.00591185" ThreadID="12773931573902802642" LogGroup="default" Roles="LR,SS,TL" />

From the backtrace, this is coming from:

foundationdb/fdbserver/TLogServer.actor.cpp

Lines 2989 to 2990 in 516351d

    
           when(wait(delay(lastPullAsyncDataWarningTime + SERVER_KNOBS->TLOG_PULL_ASYNC_DATA_WARNING_TIMEOUT_SECS - 
        
                           now()))) {

I do not think it is related. To confirm, I ensured for this run, both txn timeout knobs were set to 5s:

/t/sim2 $ grep -iE "MAX_READ_TRANSACTION_LIFE_VERSIONS|MAX_WRITE_TRANSACTION_LIFE_VERSIONS" trace*
<Event Severity="10" Time="0.000000" DateTime="2025-11-05T00:06:07Z" Type="Knob" Machine="0.0.0.0:0" ID="0000000000000000" Name="max_write_transaction_life_versions" Value="5000000" Atomic="0" ThreadID="12773931573902802642" LogGroup="default" />
<Event Severity="10" Time="0.000000" DateTime="2025-11-05T00:06:07Z" Type="Knob" Machine="0.0.0.0:0" ID="0000000000000000" Name="max_read_transaction_life_versions" Value="5000000" Atomic="0" ThreadID="12773931573902802642" LogGroup="default" />
<Event Severity="10" Time="0.000000" DateTime="2025-11-05T00:06:07Z" Type="Knob" Machine="0.0.0.0:0" ID="0000000000000000" Name="max_write_transaction_life_versions" Value="5000000" Atomic="0" ThreadID="12773931573902802642" LogGroup="default" />

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2025-10-15T22:31:20Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 90abd98
Duration 0:11:42
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-15T22:44:18Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 90abd98
Duration 0:24:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-15T22:57:50Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 90abd98
Duration 0:38:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-15T22:58:45Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 90abd98
Duration 0:39:09
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/base:${FDB_VERSION}-${COMMIT_SHA} --file Dockerfile --target base .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-15T23:05:23Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 90abd98
Duration 0:45:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-15T23:13:52Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 90abd98
Duration 0:54:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-15T23:19:56Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 90abd98
Duration 1:00:21
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:16:51Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: e5e2017
Duration 0:11:23
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:31:35Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: e5e2017
Duration 0:26:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:48:55Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: e5e2017
Duration 0:43:24
Result: ❌ FAILED
Error: Error while executing command: docker build --label "org.foundationdb.version=${FDB_VERSION}" --label "org.foundationdb.build_date=${BUILD_DATE}" --label "org.foundationdb.commit=${COMMIT_SHA}" --progress plain --build-arg FDB_VERSION="${FDB_VERSION}" --build-arg FDB_LIBRARY_VERSIONS="${FDB_VERSION}" --build-arg FDB_WEBSITE="${FDB_WEBSITE}" --tag foundationdb/base:${FDB_VERSION}-${COMMIT_SHA} --file Dockerfile --target base .. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-10-29T19:50:56Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: e5e2017
Duration 0:45:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:51:32Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: e5e2017
Duration 0:46:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:52:16Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: e5e2017
Duration 0:46:46
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-10-29T19:55:42Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: e5e2017
Duration 0:50:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-03T22:21:57Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: cc54dc3
Duration 0:24:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-03T22:32:05Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: cc54dc3
Duration 0:35:01
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-03T22:42:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cc54dc3
Duration 0:45:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-03T22:54:08Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: cc54dc3
Duration 0:57:06
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-03T23:05:31Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: cc54dc3
Duration 1:08:30
Result: ❌ FAILED
Error: Error while executing command: TEST_USERNAME=fdb-pr-${CODEBUILD_BUILD_NUMBER} make -kj -C tests foundationdb-pr-tests. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-04T00:57:19Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: cc54dc3
Duration 3:00:16
Result: ❌ FAILED
Error: Build has timed out.
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-04T06:53:38Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 2f30599
Duration 0:28:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

spraza · 2025-11-05T00:15:29Z

 #include "flow/UnitTest.h"
 #include "flow/flow.h"

+int getSimulatedTxnTimeoutSeconds() {


fyi for reviewers: this was just moved and renamed, logic is still the same

dlambrig

LGTM assuming all failed tests are proven to be unrelated.

The PR name is a little cryptic, for historical record could you rename it to something like "introduce fuzzed write transaction delay knob for simulation"

foundationdb-ci · 2025-11-05T19:41:46Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 153c733
Duration 0:24:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-05T19:52:19Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 153c733
Duration 0:35:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-05T19:56:40Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 153c733
Duration 0:39:48
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-05T20:02:01Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 153c733
Duration 0:45:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-05T20:07:26Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 153c733
Duration 0:50:31
Result: ❌ FAILED
Error: Error while executing command: export UPGRADE_VERSIONS ="${FDB_PREV_RELEASE_VERSION}:${FDB_CURRENT_VERSION}". Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-11-05T20:19:06Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 153c733
Duration 1:02:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

sbodagala · 2025-11-05T22:01:00Z

+	// When setting such knobs, we want both client and server knob to have its value updated
+	// Short circuiting would mean that server knob FOO won't be updated if client knob FOO was updated
+	// Instead, we attempt setting client and server knobs in separate statements, and return true
+	// if at least one of the set attempts was succesful.


When setting such knobs, we want both client and server knob to have its value updated

Is that the case for all such knobs? This PR automatically forces that change on all such knobs. But this maybe the correct behavior (or the behavior that we want in most cases) so it is probably fine.

"ClientKnobCollection" is following a similar model: (set a flow knob || set a client knob). We are applying this change only to ServerKnobs and not to ClientKnobs.

When setting such knobs, we want both client and server knob to have its value updated

This happens only if there is no such flow knob (in which case only the flow knob and the server knob would get updated). That's probably fine.

But this maybe the correct behavior (or the behavior that we want in most cases) so it is probably fine.

Since this only affects knobs with same names, and there are very little of them, my guess is we've not defined this behavior in the past. I personally feel it's the right behavior because it's unexpected if you update FOO via ServerKnobCollection, and only client FOO is updated but server FOO is not.

Let me actually do the exercise of finding how many knobs do we have which have the same name across client and server. It'd be good to double check that this new behavior does not conflict with anything. I can report back here.

"ClientKnobCollection" is following a similar model: (set a flow knob || set a client knob)

Yes, I wasn't sure if this behavior was needed there but regardless, keeping the behavior consistent is a good idea. I can do that in a follow-up PR after this.

These are the knobs that share the same name between ClientKnobs and ServerKnobs:

FASTRESTORE_ATOMICOP_WEIGHT

GLOBAL_CONFIG_REFRESH_TIMEOUT

MAX_WRITE_TRANSACTION_LIFE_VERSIONS (the example you mentioned)

VERSIONS_PER_SECOND

(3) and (4) we already know about.

(1) is interesting, client knob is used but server knob is not, I think I can delete the server knob in a follow-up PR.

(2)'s value is 10 in production across client and server. But under randomize && buggify, the server value can be 1 sometimes. It looks intentional so I'll leave it as is.

sbodagala · 2025-11-05T22:37:59Z

@@ -38,7 +33,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi

 	// Versions -- knobs that control 5s timeout
 	init( VERSIONS_PER_SECOND,                                   1e6 );


This knob's value should be same as the corresponding client knob's value (which is what I think you were trying to do in your other commit)?

Please confirm that this is a valid change, thanks!

Yes, doing an experiment to gather data before responding. Will report back.

This should work, let me make this change in the follow-up PR alongside other changes discussed.

spraza · 2025-11-05T22:57:48Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 153c733

Duration 0:39:48

Result: ❌ FAILED

Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1

Build Log terminal output (available for 30 days)

Build Workspace zip file of the working directory (available for 30 days)

Seems to be joshua infra issue:

[Container] 2025/11/05 19:47:49.149493 Running command python3 -m joshua.joshua tail --errors --xml ${ENSEMBLE_ID}
Results for test ensemble: 20251105-194746-pr12464-clang-153c733a-1318-2258edcffdbfb4cd
Ensemble stopped

[Container] 2025/11/05 19:50:42.241249 Running command python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID}
20251105-194746-pr12464-clang-153c733a-1318-2258edcffdbfb4cd compressed=True data_size=38486474 duration=117388 ended=2383 fail=10 fail_fast=10 max_runs=10000 pass=2373 priority=100 rem
aining=0 runtime=0:02:56 sanity=False started=5636 stopped=20251105-195042 submitted=20251105-194746 timeout=5400 username=pr12464-clang-153c733a-13185

[Container] 2025/11/05 19:50:42.560957 Running command if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL
&& exit 1; fi
FAIL

jzhou77

LGTM

foundationdb-ci · 2025-11-06T01:18:59Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 153c733
Duration 0:29:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-06T01:34:27Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 153c733
Duration 0:44:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-06T01:48:02Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 153c733
Duration 0:58:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-06T01:56:39Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 153c733
Duration 1:06:56
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-11-06T01:58:09Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 153c733
Duration 1:08:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

* remove server FASTRESTORE_ATOMICOP_WEIGHT * address other feedback * fix spacing * revert VERSIONS_PER_SECOND change

…ple#12464) * 5s: fuzz write knob * self review * self review * self review * looks like AutomaticIdempotency.toml works, so revert my changes there * fix spacing

* remove server FASTRESTORE_ATOMICOP_WEIGHT * address other feedback * fix spacing * revert VERSIONS_PER_SECOND change

spraza marked this pull request as draft October 15, 2025 22:19

spraza force-pushed the 5s-milestone1-iter34 branch from e5e2017 to cc54dc3 Compare November 3, 2025 21:56

spraza added 5 commits November 3, 2025 22:17

5s: fuzz write knob

b2633d2

self review

3a5a540

self review

a72fd2d

self review

906789b

looks like AutomaticIdempotency.toml works, so revert my changes there

2f30599

spraza force-pushed the 5s-milestone1-iter34 branch from cc54dc3 to 2f30599 Compare November 4, 2025 06:25

spraza commented Nov 4, 2025

View reviewed changes

Comment thread fdbclient/ServerKnobs.cpp

spraza requested review from jzhou77 and sbodagala November 5, 2025 00:14

spraza commented Nov 5, 2025

View reviewed changes

dlambrig approved these changes Nov 5, 2025

View reviewed changes

spraza changed the title ~~5s - fuzz write knob in simulation~~ 5s - introduce fuzzed write transaction delay knob for simulation Nov 5, 2025

fix spacing

153c733

sbodagala reviewed Nov 5, 2025

View reviewed changes

jzhou77 approved these changes Nov 5, 2025

View reviewed changes

sbodagala approved these changes Nov 5, 2025

View reviewed changes

spraza closed this Nov 6, 2025

spraza reopened this Nov 6, 2025

spraza merged commit 596f4cb into apple:main Nov 6, 2025
6 checks passed

spraza mentioned this pull request Nov 6, 2025

Follow-up to PR #12464 #12541

Merged

5 tasks

spraza added a commit that referenced this pull request Nov 10, 2025

Follow-up to PR #12464 (#12541)

f21ae29

* remove server FASTRESTORE_ATOMICOP_WEIGHT * address other feedback * fix spacing * revert VERSIONS_PER_SECOND change

spraza added a commit to spraza/foundationdb that referenced this pull request Jan 15, 2026

Follow-up to PR apple#12464 (apple#12541)

dae9d37

* remove server FASTRESTORE_ATOMICOP_WEIGHT * address other feedback * fix spacing * revert VERSIONS_PER_SECOND change

spraza added a commit to spraza/foundationdb that referenced this pull request Jan 15, 2026

Follow-up to PR apple#12464 (apple#12541)

5aada41

* remove server FASTRESTORE_ATOMICOP_WEIGHT * address other feedback * fix spacing * revert VERSIONS_PER_SECOND change

	Future<class Void> delay(double seconds, TaskPriority taskID, ProcessInfo* machine, bool ordered = false) {
	ASSERT(seconds >= -0.0001);

	when(wait(delay(lastPullAsyncDataWarningTime + SERVER_KNOBS->TLOG_PULL_ASYNC_DATA_WARNING_TIMEOUT_SECS -
	now()))) {

		@@ -38,7 +33,7 @@ void ServerKnobs::initialize(Randomize randomize, ClientKnobs* clientKnobs, IsSi

		// Versions -- knobs that control 5s timeout
		init( VERSIONS_PER_SECOND, 1e6 );

Uh oh!

Conversation

spraza commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 15, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Oct 29, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 3, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 3, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Nov 3, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Nov 3, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 3, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Nov 4, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

Uh oh!

foundationdb-ci commented Nov 4, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

spraza Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

dlambrig left a comment

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Nov 5, 2025

spraza commented Oct 15, 2025 •

edited

Loading

sbodagala Nov 5, 2025 •

edited

Loading

sbodagala Nov 5, 2025 •

edited

Loading

spraza Nov 6, 2025 •

edited

Loading