Add S3ErrorCode to S3BlobStore by spraza · Pull Request #12979 · apple/foundationdb

spraza · 2026-04-13T19:54:52Z

In 7.3, we reverted some backup related changes, but as part of that, also reverted S3 observability. This observability is useful so adding some of it back, specifically S3ErrorCode and S3BlobStoreBadRequest.

100K: 20260413-204156-praza-4386c6e11f96d348 compressed=True data_size=35381800 fail_fast=10 max_runs=100000 priority=100 sanity=False submitted=20260413-204156 timeout=5400 username=praza

Radar: rdar://173399798

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2026-04-13T19:59:09Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: a2829e8
Duration 0:04:05
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T20:04:08Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: a2829e8
Duration 0:09:04
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T20:04:21Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: a2829e8
Duration 0:09:16
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T20:04:41Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: a2829e8
Duration 0:09:34
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

gxglass · 2026-04-13T20:12:22Z

Could mention that this is rdar://173399798 (I believe)

spraza · 2026-04-13T20:44:41Z

 ERROR( backup_does_not_exist, 2319, "Backup does not exist")
 ERROR( backup_not_filterable_with_key_ranges, 2320, "Backup before 6.3 cannot be filtered with key ranges")
 ERROR( backup_not_overlapped_with_keys_filter, 2321, "Backup key ranges doesn't overlap with key ranges filter")
+ERROR( backup_parse_s3_response_failure, 2322, "cannot parse s3 response properly" )


@saintstack in #11965, the code was 2323, because 2322 was used by another error not in this PR. Over here, I used 2322. Please review this part too.

ok. still +1

foundationdb-ci · 2026-04-13T20:47:29Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 43c8cb1
Duration 0:04:15
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T20:51:47Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 43c8cb1
Duration 0:08:37
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T20:51:49Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 43c8cb1
Duration 0:08:38
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-13T20:52:04Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 43c8cb1
Duration 0:08:53
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2026-04-13T20:57:41Z

 ERROR( backup_does_not_exist, 2319, "Backup does not exist")
 ERROR( backup_not_filterable_with_key_ranges, 2320, "Backup before 6.3 cannot be filtered with key ranges")
 ERROR( backup_not_overlapped_with_keys_filter, 2321, "Backup key ranges doesn't overlap with key ranges filter")
+ERROR( backup_parse_s3_response_failure, 2322, "cannot parse s3 response properly" )


ok. still +1

foundationdb-ci · 2026-04-13T21:07:39Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 5f16bf0
Duration 0:04:11
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T21:11:59Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 5f16bf0
Duration 0:08:32
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T21:12:06Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 5f16bf0
Duration 0:08:41
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-13T21:12:12Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 5f16bf0
Duration 0:08:47
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T21:16:01Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 43c8cb1
Duration 0:32:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-13T21:22:34Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 43c8cb1
Duration 0:39:23
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

gxglass

Here's what AI said:

gxglass · 2026-04-13T21:48:14Z

+		return std::string(codeNode->value());
+	} catch (Error e) {
+		TraceEvent("BackupParseS3ErrorCodeFailure").errorUnsuppressed(e);
+		throw backup_parse_s3_response_failure();


I think this is problematic per the AI. It changes the retry behavior of the caller in a way that is different from today.

if we can't parse the XML just return an empty "" as per various intermediate failures above.

gxglass · 2026-04-13T23:13:22Z

 		event.suppressFor(60);
 		if (!err.present()) {
 			event.detail("ResponseCode", r->code);
+			std::string s3Error = parseErrorCodeFromS3(r->data.content);


A really simple alternative here would be to say

event.detail("Details", r->data.content);

spraza · 2026-04-14T00:12:01Z

Converting to draft for now. We're discussing how to proceed with this PR.

saintstack · 2026-04-14T16:38:01Z

#12986 adds the suggested changes on main.

Add S3ErrorCode to S3BlobStore

a2829e8

spraza marked this pull request as draft April 13, 2026 19:57

add definitions to fix compiltion failures

43c8cb1

saintstack previously approved these changes Apr 13, 2026

View reviewed changes

spraza dismissed saintstack’s stale review via 43c8cb1 April 13, 2026 20:43

spraza marked this pull request as ready for review April 13, 2026 20:43

spraza commented Apr 13, 2026

View reviewed changes

spraza requested review from gxglass and neethuhaneesha April 13, 2026 20:45

spraza requested a review from akankshamahajan15 April 13, 2026 20:51

saintstack approved these changes Apr 13, 2026

View reviewed changes

neethuhaneesha approved these changes Apr 13, 2026

View reviewed changes

Merge branch 'release-7.3' into backup-observability-v1

5f16bf0

gxglass reviewed Apr 13, 2026

View reviewed changes

spraza marked this pull request as draft April 14, 2026 00:12

saintstack mentioned this pull request Apr 15, 2026

Harden parseErrorCodeFromS3 to never throw on non-XML responses 7.3 #12997

Merged

spraza closed this Apr 16, 2026

Uh oh!

Conversation

spraza commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code-Reviewer Section

For Release-Branches

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

gxglass commented Apr 13, 2026

Uh oh!

spraza Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

saintstack Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

saintstack Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Apr 13, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

gxglass left a comment

Choose a reason for hiding this comment

Uh oh!

gxglass Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

gxglass Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

spraza commented Apr 14, 2026

Uh oh!

saintstack commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

spraza commented Apr 13, 2026 •

edited

Loading