This repository was archived by the owner on Jul 27, 2022. It is now read-only.
Repository navigation
This repository was archived by the owner on Jul 27, 2022. It is now read-only.
Ensure all ObjectStores have publicly available client #10
Description
ObjectStores might want to be extended with additional traits that use the underlying client. For example, in delta-rs we would want to extend the client to implement a version of rename that failed if the destination already exists (see delta-io/delta-rs#610).
AmazonS3 does expose a method .client():
Lines 533 to 535 in 5f488c5
MicrosoftAzure only has a private field container_client:
Lines 73 to 74 in 5f488c5
GoogleCloudStorage only has a private field client:
Lines 103 to 104 in 5f488c5
They all will return a different type, but maybe we should implement for each a client() method like the S3 has?
Activity
tustvold commentedon May 20, 2022
I'm a bit wary of exposing the clients used internally as I suspect we may want to change these down the line. Both Rusoto and cloud_storage are not official crates, with somewhat patchy support.
Perhaps we could add copy support directly to the trait, it seems like a reasonable addition to me?
wjones127 commentedon May 20, 2022
As in, copy the client rather than returning a reference / shared pointer?
Yeah, I agree. The alternative would be crates should bring their object store extensions into this crate as optional features. So delta-rs would bring in the
rename_noreplace()method as an optional trait. In the S3 case, that might mean bringing over dynamodb lock, which may be more than you wish to accept (or maybe not!).Or even a final alternative would be that in delta-rs we wrap the
ObjectStoreimplementors instead of using them directly.tustvold commentedon May 20, 2022
I meant adding
CopyObjectsupport toObjectStoreitself. A rename is just a copy followed by a delete.I think an optional feature that adds CreateIfNotExists and CopyIfNotExists operations would be really cool. It's a real pain that S3 doesn't natively support it, and if we can somehow hide that from users that would be awesome.
roeap commentedon May 20, 2022
In case of Adls Gen2 the *IfNotExist methods are natively supported. In delta-rs we also have an implementation for local file systems, which only on windows needs some extra care - I think... for blob and S3 afaik some external service is needed.
I'd be happy to contribute adls gen2 to this crate 🙂.
wjones127 commentedon May 21, 2022
Sounds good. I've started implementing that in #14. I will close #11.