Repository navigation
Add PutMultipartOptions::mode for conditional multipart completion - #863
Open
chromafish wants to merge 1 commit into
Open
chromafish wants to merge 1 commit into
chromafish wants to merge 1 commit into
Conversation
put_opts supports PutMode::Create, but a multipart upload has no way to complete as create-if-absent, so large immutable objects can't get the same guarantee. S3 supports If-None-Match: * on CompleteMultipartUpload, and the S3 client already sends it for S3CopyIfNotExists::Multipart. PutMultipartOptions gains a `mode`, applied when the upload completes. AmazonS3 completes a Create upload with If-None-Match: * (requiring conditional put to be enabled, as put_opts does) and reports an existing object as AlreadyExists. InMemory honors it too. LocalFileSystem, GCS and Azure return NotImplemented from put_multipart_opts for any mode other than Overwrite. Update isn't supported for multipart uploads. MultipartStore::create_multipart_opts returns NotSupported for any mode other than Overwrite: the upload is stateless between creation and complete_multipart, so the mode couldn't be applied. As with Create in put_opts, a Create completion is no longer marked idempotent, so it isn't retried on timeout: if the first attempt succeeded, the retry would fail its If-None-Match and report the caller's own write as AlreadyExists. This also applies to copy_if_not_exists with S3CopyIfNotExists::Multipart. Adds a put_multipart_opts_create integration test, run for S3, InMemory, LocalFileSystem, GCS and Azure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #289
This adds support for S3 and
InMemoryonly. GCS, Azure andLocalFileSystemreturnNotImplemented. Azure's Put Block List acceptsIf-None-Match: *, so it could be implemented, but I'm not very familiar with Azure Blob Storage. I can take a shot at it if it's wanted.Rationale for this change
Multipart uploads are currently always unconditional overwrites. S3 supports
If-None-Match: *onCompleteMultipartUpload, and the S3 client already sends it internally forS3CopyIfNotExists::Multipart. It just isn't exposed through the public multipart API.What changes are included in this PR?
PutMultipartOptionsgains amodefield, allowing multipart uploads to usePutMode::Create.AmazonS3::put_multipart_optsresolves the mode the same wayput_optsdoes. On completion, it uses the same error handling asput_opts, which I extracted into amap_create_errorfunction.MultipartStore::create_multipart_optsreturnsNotSupportedfor any mode other thanOverwrite(see limitations below).complete_multipartno longer marksCreatecompletions as idempotent, so they aren't retried on timeout, matchingput_optswithCreate. This is to avoid the scenario where the first attempt succeeded but its response was lost, so the retry would fail itsIf-None-Matchand report the caller's own write asAlreadyExists. This also applies to the existingcopy_if_not_existswithS3CopyIfNotExists::Multipart.Limitations
modeonly takes effect throughObjectStore::put_multipart_opts.MultipartStoreuploads are stateless betweencreate_multipart_optsandcomplete_multipart, and on S3 the condition is a header onCompleteMultipartUpload, so there's nowhere to carry the mode to completion. Supporting it would needMultipartStore::complete_multipartto accept the mode, e.g. through a newcomplete_multipart_optsmethod. I've left that out since it adds a new API surface to the trait, but I can follow up with this if you'd like.PutMode::Updateisn't supported for multipart uploads.Are there any user-facing changes?
Yes.
PutMultipartOptionshas a new public field, so code that builds it with astruct literal and no
..Default::default()will stop compiling.