Skip to content

S3: return part checksums in ListParts - #10528

Open
mehransaeed7810 wants to merge 1 commit into
getmoto:masterfrom
mehransaeed7810:s3-listparts-part-checksums
Open

mehransaeed7810 wants to merge 1 commit into
getmoto:masterfrom
mehransaeed7810:s3-listparts-part-checksums

Conversation

@mehransaeed7810

Copy link
Copy Markdown

Problem

When a multipart upload is created with a ChecksumAlgorithm, AWS keeps a checksum for every uploaded part and returns it in ListParts (Part.ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME, ChecksumSHA1, ChecksumSHA256). Moto drops the part's checksum in FakeMultipart.set_part, so ListParts never returns one.

That breaks clients that resume an upload by part. They call ListParts and count a part as uploaded only when its listed checksum matches the one they expected. Under moto no part ever matches, so no multipart upload can complete.

AWS documentation:

  • Part: "This checksum is present if the multipart upload request was created with the CRC64NVME checksum algorithm" (and the same for the other algorithms).
  • ListParts: each Part carries its Checksum*, and the result carries ChecksumAlgorithm and ChecksumType.
  • UploadPart: the checksum arrives in x-amz-checksum-*. x-amz-sdk-checksum-algorithm is sent only by the SDKs, and "this checksum algorithm must be the same for all parts and it match the checksum value supplied in the CreateMultipartUpload request". The x-amz-checksum-* response header "will only be present if the checksum was provided in the request".
  • CompleteMultipartUpload: the result carries the object's Checksum* and ChecksumType.
  • UploadPartCopy / CopyPartResult: the copied part's Checksum*.

Change

  • FakeMultipart.set_part stores a checksum on each part when the upload has a ChecksumAlgorithm. If the part supplied a value for that algorithm, that value is stored. Otherwise the value is computed from the part with compute_checksum. An upload without a ChecksumAlgorithm behaves as before.
  • UploadPart passes the supplied checksum to the backend. That covers the SDK path (x-amz-sdk-checksum-algorithm with a header or a trailer, through the existing _get_checksum). It also covers a request that sends only the x-amz-checksum-* header, which is how a presigned upload_part URL signed with ChecksumSHA256 arrives. x-amz-sdk-checksum-algorithm is removed from the response, as PutObject already does.
  • ListParts returns Checksum<ALGORITHM> on each Part, plus ChecksumAlgorithm and ChecksumType on the result.
  • UploadPartCopy returns the part checksum in CopyPartResult.
  • CompleteMultipartUpload returns Checksum<ALGORITHM> and ChecksumType. A COMPOSITE checksum carries the -<parts> suffix, matching what moto's GetObject already returns for the same object.

Out of scope:

  • Moto still does not verify a supplied checksum against the bytes (the existing TODO in _get_checksum).
  • CompleteMultipartUpload still does not compare the part checksums in its request body with the stored ones.
  • HeadObject still returns the composite checksum without the -<parts> suffix, unlike GetObject.

Tests

New tests in tests/test_s3/test_s3_multipart.py:

  • test_list_parts_returns_part_checksums: one case per algorithm (CRC32, CRC32C, CRC64NVME, SHA1, SHA256). Each part's listed checksum equals the UploadPart response and compute_checksum of the part, and no other algorithm's checksum is listed.
  • test_list_parts_computes_the_checksum_when_the_part_supplies_none: the client uses request_checksum_calculation="when_required", so no checksum is sent.
  • test_list_parts_has_no_checksum_when_the_upload_has_no_algorithm: the case with no algorithm.
  • test_list_parts_returns_the_checksum_signed_into_a_presigned_upload_part: the checksum is sent only as the signed x-amz-checksum-sha256 header.
  • test_multipart_round_trip_with_listed_part_checksums: the boto3 round trip. It uploads three parts, pages through ListParts with MaxParts=1, completes with the listed checksums, and checks the composite ChecksumSHA256/ChecksumType in the response against GetObject(ChecksumMode="ENABLED").

Local results:

  • make lint checks pass: ruff check, ruff format --check and mypy, with ruff 0.14.10 from requirements-dev.txt.
  • pytest tests/test_s3 in decorator mode: 578 passed, 2 xfailed, 2 failed. The two failures also fail on master on the same machine. They resolve *.localhost hostnames (test_server.py::test_s3_server_post_cors_multiple_origins and test_multiple_accounts_server.py::TestAccountIdResolution::test_with_custom_request_header).
  • tests/test_s3/test_s3_multipart.py in server mode (TEST_SERVER_MODE=true against moto_server): 48 passed, 18 skipped. CRC32C and CRC64NVME are skipped in server mode, as for the existing checksum tests.

None of the new tests are marked aws_verified: they have not been run against AWS.

A multipart upload created with a ChecksumAlgorithm now keeps each part's
checksum for that algorithm: the value the part supplied (as an
x-amz-checksum-* header, a trailer, or a header signed into a presigned URL),
or one computed from the part when none was supplied. ListParts returns it per
Part, with the upload's ChecksumAlgorithm and ChecksumType. UploadPartCopy
returns it in CopyPartResult, and CompleteMultipartUpload returns the object
checksum and ChecksumType, as AWS documents. Uploads without a
ChecksumAlgorithm are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant