Repository navigation
Retrying failed part PUTs in multipart writes #831
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
SlateDB currently has a RetryingObjectStore that handles ObjectStore retries. The object_store retry mechanisms are unsuitable for us. This is something we've discussed in various GH issues in the past. We do something like this:
#[inline]
fn should_retry(err: &object_store::Error) -> bool {
let retry = !matches!(
err,
object_store::Error::AlreadyExists { .. }
| object_store::Error::Precondition { .. }
| object_store::Error::NotModified { .. }
| object_store::Error::NotFound { .. }
| object_store::Error::NotImplemented { .. }
| object_store::Error::NotSupported { .. }
);
if !retry {
debug!("not retrying object store operation [error={:?}]", err);
}
retry
}This approach is somewhat similar to what Lance does (#737).
The problem we have right now is there's no good way for us to handle retries for multipart PUTs. We currently inherit object_store's behavior for those writes. This behavior is insufficient (slatedb/slatedb#1956).
Describe the solution you'd like
We've been wrestling a bit with the best way to handle this. The current object_store API makes it impossible for us to retry a multipart upload without buffering the entire byte object and retrying from scratch (as far as I can tell--please correct me if I'm wrong). I am seeking advice/suggestions at this point.
There are a few challenges here:
- The multipart puts are async. We don't see a part failure until later.
- We don't know which parts failed AFAICT.
- There's no high-level API for us to retry a failed part. I see
MultipartStore::put_part(.., part_idx, data), but that isn't exposed at theObjectStorelevel.
What do you think? Should object_store to support custom retry policies for multipart uploads (or for uploads in general)? Should users be allowed to retry a specific part write?
It's quite possible I'm misunderstanding something here, so LMK if that's the case.
Describe alternatives you've considered
Living with what we have, which can halt SlateDB databases in certain scenarios.
Additional context
See GH links above.
Activity
tustvold commentedon Aug 23, 2026
For context apache/arrow-rs#5500 contains the original design intent for the trait and some of the reasons for the current design.
The big challenge is providing a mechanism that works for LocalFileystem without forcing double-buffering. This is why part IDs are not exposed, as they don't actually mean anything for LocalFileystem.
That being said I do remember an early design where UploadPart was actually a stream that you could poll to keep retrying, but eventually that got scrapped for being too complex, as it made the trait very unwieldy.
I suspect providing customisable retry policies is probably the simplest solution here. Whilst MultipartStore gives full control, LocalFileystem can't implement it.
An alternative might be to somehow extend MultipartUpload with a method that returns a retryable version of UploadPart, but that would almost certainly be a breaking change
criccomini commentedon Aug 23, 2026
+1 I would be fine with this. I think it would help us in other areas, too. Do you have a design in mind, or something that was tried before and scrapped?
tustvold commentedon Aug 24, 2026
I don't have a design in mind, but it occurs to me that we already have the Http service abstraction. The ideal solution in my mind would be something that could just slot into this, as then the retry machinery has all the context it could want, and is fully in control of what requests are being made.
Perhaps the existing retry logic becomes an HttpService wrapper or something, I'm not sure what is possible.
IIRC the reason I didn't do this initially is you need to pass context on whether the request is idempotent, etc... but perhaps that could be passed as a request extension?
See also #624 and #840
criccomini commentedon Aug 24, 2026
Yea, we already use extensions pretty extensively. I could see an "is_idempotent" working. We currently attach user defined metadata with a unique ID and on collision, verify the unique ID in the request matches a pre-existing file. I'll give it some thought.