Fix a couple of simulation failures by kakaiu · Pull Request #11543 · apple/foundationdb

kakaiu · 2024-07-31T00:53:51Z

This PR includes following fixes:

When usableRegions changes, data moves should be triggered to satisfy the new usableRegions. When encode shard location metadata is on, it is possible that a restored data move is cancelled. If no further data move is triggered, the shard's usableRegions can never match the new usableRegions, which can block the simulation team size consistency check. To fix this issue, this PR adds a usableRegion checker per shard when a shard is created. If a shard destination team does not meet the new usableRegions, issue a data move for the shard. To avoid tons of shards triggering data moves at the same time, this PR introduces new knob DD_SHARD_USABLE_REGION_CHECK_RATE to control the speed of triggering data moves.
Fix SS assertion failed for a wrong data move type generated by an old binary which does not encode the data move type in the data move id
Physical shard move in release-7.3 is compatible with main branch (release-7.4) only since 7.3.29. So, changed the restarting tests accordingly. Note that before 7.3.5, physical shard move is not controlled by dd_physical_shard_move_probability.

100K correctness tests:
20240801-192101-zhewang-1da3aaa415686dc8 compressed=True data_size=37152167 duration=6111521 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:54:45 sanity=False started=100000 stopped=20240801-201546 submitted=20240801-192101 timeout=5400 username=zhewang

Reproduce sharded rocksdb assertion failure at commit ID: 4936acaa7addb946481777730ed7445363c7714f
../src/fdbserver-7.3.43 -r simulation -f ../src/foundationdb/tests/restarting/from_7.3.29/ClientTransactionProfilingCorrectness-1.toml --buggify on --seed 3322800254 --logsize 1024MiB
../build_output/bin/fdbserver -r simulation -f ../src/foundationdb/tests/restarting/from_7.3.29/ClientTransactionProfilingCorrectness-2.toml --buggify on --seed 3322800255 --logsize 1024MiB --restarting

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2024-07-31T01:17:08Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 11d6b10
Duration 0:23:07
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T01:20:34Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: e09f1a1
Duration 0:21:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T01:34:41Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 11d6b10
Duration 0:40:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T01:41:13Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 11d6b10
Duration 0:47:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T01:45:42Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 11d6b10
Duration 0:51:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T01:51:57Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: e09f1a1
Duration 0:53:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:07:08Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 11d6b10
Duration 1:13:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:07:18Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: e09f1a1
Duration 1:08:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:08:23Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 11d6b10
Duration 1:14:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:10:05Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: e09f1a1
Duration 1:11:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:15:58Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 11d6b10
Duration 1:21:58
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-07-31T02:17:54Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: e09f1a1
Duration 1:19:07
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-07-31T02:50:42Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cdb8aaf
Duration 0:07:33
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:51:59Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: cdb8aaf
Duration 0:08:50
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:52:04Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: cdb8aaf
Duration 0:08:54
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-07-31T02:52:09Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: cdb8aaf
Duration 0:09:03
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:52:21Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: cdb8aaf
Duration 0:09:12
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:54:16Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: cdb8aaf
Duration 0:11:09
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T02:54:41Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: cdb8aaf
Duration 0:11:30
Result: ❌ FAILED
Error: reference not found for primary source and source version cdb8aaf22c41b8ab63c9b4137f7b9ccc2609a0b4
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:09:02Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: f386793
Duration 0:24:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:12:57Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 781f6c4
Duration 0:24:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:36:13Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: f386793
Duration 0:51:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:39:05Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 781f6c4
Duration 0:50:36
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:55:45Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: f386793
Duration 1:10:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:58:49Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 781f6c4
Duration 1:10:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-07-31T03:58:54Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: f386793
Duration 1:14:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T20:05:45Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 4fe8028
Duration 0:52:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T20:12:58Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 4fe8028
Duration 1:00:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T20:34:10Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 4fe8028
Duration 1:21:24
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77 · 2024-08-01T21:30:50Z

  add_fdb_test(
-    TEST_FILES restarting/from_7.3.0/ClientTransactionProfilingCorrectness-1.toml
-    restarting/from_7.3.0/ClientTransactionProfilingCorrectness-2.toml)
+    TEST_FILES restarting/from_7.3.0_until_7.3.5/ClientTransactionProfilingCorrectness-1.toml
+    restarting/from_7.3.0_until_7.3.5/ClientTransactionProfilingCorrectness-2.toml)


Remove. I checked https://github.com/FoundationDB/fdb-joshua/blob/main/Dockerfile, where only 7.3.43 and 7.3.27 are listed. So we can ignore 7.3.0 to 7.3.5 here, as they are not supported.

Ok. One thing to mention here is the 7.3 binary < 7.3.5 does not have the knob dd_physical_shard_move_probability

jzhou77 · 2024-08-01T21:33:29Z

+	ASSERT(SERVER_KNOBS->SHARD_ENCODE_LOCATION_METADATA);
+	ASSERT(SERVER_KNOBS->DD_SHARD_USABLE_REGION_CHECK_RATE > 0);
+	wait(yieldedFuture(self()->readyToStart.getFuture()));
+	double expectedCompletionSeconds = self()->shards->size() * 1.0 / SERVER_KNOBS->DD_SHARD_USABLE_REGION_CHECK_RATE;


The comment above says this knob could be set to 0, which, if set, will definitely cause divide by 0 error and the above assertion failure.

We check the value of DD_SHARD_USABLE_REGION_CHECK_RATE should not be 0 at line 214. This actor is spawned only if DD_SHARD_USABLE_REGION_CHECK_RATE is not 0.

foundationdb-ci · 2024-08-01T22:21:43Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 289edf3
Duration 0:21:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:34:59Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 289edf3
Duration 0:34:42
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:38:54Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 289edf3
Duration 0:38:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:39:17Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:23:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:47:44Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 289edf3
Duration 0:47:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:54:06Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 289edf3
Duration 0:53:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T22:57:41Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:21:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:07:36Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:51:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:20:04Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 7f1abcf
Duration 1:04:13
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:23:07Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 289edf3
Duration 1:22:45
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-08-01T23:23:43Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 289edf3
Duration 1:23:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:28:36Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:52:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:28:57Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:53:08
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:31:39Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 7f1abcf
Duration 1:15:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:35:14Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 7f1abcf
Duration 0:59:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-08-01T23:41:54Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 7f1abcf
Duration 1:26:01
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-08-01T23:59:30Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 7f1abcf
Duration 1:23:41
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

kakaiu requested review from jzhou77 and liquid-helium July 31, 2024 00:53

kakaiu force-pushed the fix-encode-shard-location-and-usable-region branch 2 times, most recently from f386793 to 781f6c4 Compare July 31, 2024 02:48

jzhou77 suggested changes Aug 1, 2024

View reviewed changes

address comments

289edf3

kakaiu requested a review from jzhou77 August 1, 2024 22:00

nits

7f1abcf

kakaiu closed this Aug 1, 2024

kakaiu reopened this Aug 1, 2024

jzhou77 approved these changes Aug 2, 2024

View reviewed changes

kakaiu merged commit a245b96 into apple:main Aug 2, 2024

kakaiu mentioned this pull request Sep 19, 2024

[Release-7.3] Backport: add usable region check per shard for encode shard location metadata #11671

Merged

5 tasks

spraza mentioned this pull request Jan 8, 2025

Update downgrade tests to have a new split point #11867

Merged

5 tasks

Uh oh!

Conversation

kakaiu commented Jul 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jul 31, 2024

kakaiu commented Jul 31, 2024 •

edited

Loading