Repository navigation
S3: return part checksums in ListParts - #10528
Open
mehransaeed7810 wants to merge 1 commit into
Open
mehransaeed7810 wants to merge 1 commit into
mehransaeed7810 wants to merge 1 commit into
Conversation
A multipart upload created with a ChecksumAlgorithm now keeps each part's checksum for that algorithm: the value the part supplied (as an x-amz-checksum-* header, a trailer, or a header signed into a presigned URL), or one computed from the part when none was supplied. ListParts returns it per Part, with the upload's ChecksumAlgorithm and ChecksumType. UploadPartCopy returns it in CopyPartResult, and CompleteMultipartUpload returns the object checksum and ChecksumType, as AWS documents. Uploads without a ChecksumAlgorithm are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a multipart upload is created with a
ChecksumAlgorithm, AWS keeps a checksum for every uploaded part and returns it inListParts(Part.ChecksumCRC32,ChecksumCRC32C,ChecksumCRC64NVME,ChecksumSHA1,ChecksumSHA256). Moto drops the part's checksum inFakeMultipart.set_part, soListPartsnever returns one.That breaks clients that resume an upload by part. They call
ListPartsand count a part as uploaded only when its listed checksum matches the one they expected. Under moto no part ever matches, so no multipart upload can complete.AWS documentation:
CRC64NVMEchecksum algorithm" (and the same for the other algorithms).Partcarries itsChecksum*, and the result carriesChecksumAlgorithmandChecksumType.x-amz-checksum-*.x-amz-sdk-checksum-algorithmis sent only by the SDKs, and "this checksum algorithm must be the same for all parts and it match the checksum value supplied in theCreateMultipartUploadrequest". Thex-amz-checksum-*response header "will only be present if the checksum was provided in the request".Checksum*andChecksumType.Checksum*.Change
FakeMultipart.set_partstores a checksum on each part when the upload has aChecksumAlgorithm. If the part supplied a value for that algorithm, that value is stored. Otherwise the value is computed from the part withcompute_checksum. An upload without aChecksumAlgorithmbehaves as before.UploadPartpasses the supplied checksum to the backend. That covers the SDK path (x-amz-sdk-checksum-algorithmwith a header or a trailer, through the existing_get_checksum). It also covers a request that sends only thex-amz-checksum-*header, which is how a presignedupload_partURL signed withChecksumSHA256arrives.x-amz-sdk-checksum-algorithmis removed from the response, asPutObjectalready does.ListPartsreturnsChecksum<ALGORITHM>on eachPart, plusChecksumAlgorithmandChecksumTypeon the result.UploadPartCopyreturns the part checksum inCopyPartResult.CompleteMultipartUploadreturnsChecksum<ALGORITHM>andChecksumType. ACOMPOSITEchecksum carries the-<parts>suffix, matching what moto'sGetObjectalready returns for the same object.Out of scope:
_get_checksum).CompleteMultipartUploadstill does not compare the part checksums in its request body with the stored ones.HeadObjectstill returns the composite checksum without the-<parts>suffix, unlikeGetObject.Tests
New tests in
tests/test_s3/test_s3_multipart.py:test_list_parts_returns_part_checksums: one case per algorithm (CRC32, CRC32C, CRC64NVME, SHA1, SHA256). Each part's listed checksum equals theUploadPartresponse andcompute_checksumof the part, and no other algorithm's checksum is listed.test_list_parts_computes_the_checksum_when_the_part_supplies_none: the client usesrequest_checksum_calculation="when_required", so no checksum is sent.test_list_parts_has_no_checksum_when_the_upload_has_no_algorithm: the case with no algorithm.test_list_parts_returns_the_checksum_signed_into_a_presigned_upload_part: the checksum is sent only as the signedx-amz-checksum-sha256header.test_multipart_round_trip_with_listed_part_checksums: the boto3 round trip. It uploads three parts, pages throughListPartswithMaxParts=1, completes with the listed checksums, and checks the compositeChecksumSHA256/ChecksumTypein the response againstGetObject(ChecksumMode="ENABLED").Local results:
make lintchecks pass:ruff check,ruff format --checkandmypy, with ruff 0.14.10 fromrequirements-dev.txt.pytest tests/test_s3in decorator mode: 578 passed, 2 xfailed, 2 failed. The two failures also fail onmasteron the same machine. They resolve*.localhosthostnames (test_server.py::test_s3_server_post_cors_multiple_originsandtest_multiple_accounts_server.py::TestAccountIdResolution::test_with_custom_request_header).tests/test_s3/test_s3_multipart.pyin server mode (TEST_SERVER_MODE=trueagainstmoto_server): 48 passed, 18 skipped. CRC32C and CRC64NVME are skipped in server mode, as for the existing checksum tests.None of the new tests are marked
aws_verified: they have not been run against AWS.