feat: add object store based on aws-sdk-rust

[timvw]

No description provided.

[timvw]

Files changed seems huge, but that's because I added 1005 empty files for testing the list operations...

[tustvold]

but that's because I added 1005 empty files for testing the list operations...

Could we please generate these for the tests, as opposed to checking them in?

add object store based on aws-sdk-rust

I have a couple of concerns about this, I would have commented sooner, but I didn't realise that you were working on this.

I would rather not support and maintain multiple parallel AWS implementations. This adds cognitive burden both to maintainers and users, and I can't see a particularly compelling user-facing benefit to supporting both.

The AWS SDK has a fairly big disclaimer about using it for production workloads.

The SDK is currently released as a developer preview and is intended strictly for feedback purposes only. Do not use this SDK for production workloads.

I have a long-term hope to remove the use of SDKs altogether #18 as they aren't really adding significant value, and complicate code-reuse when the network APIs are basically the same across all providers.

This leaves me in the uncomfortable position of not really being sure I can merge this...

[timvw]

The only reason for having an implementation based on aws-sdk is the fact that I need support for aws sso profiles and the AWS_PROFILE environment variable. Sample from my ~/.aws/config

[profile XXX]
sso_start_url = https://XXX.awsapps.com/start#/
sso_region = eu-central-1
sso_account_id = YYY
sso_role_name = ZZZ
region = eu-central-1

This does not seem to work with rusoto and seems that active development/new features are not happening anymore (eg: rusoto/rusoto#1842).

From one side, aws sdk claims not to be ready for production, but rusoto is also not actively developed anymore. Being able to choose between two not-ideal options seems most flexible to me.

[tustvold]

sso_start_url

I'm somewhat confused about how this works. From my memory of the SSO support within AWS there is a hard 24 hour cap on the produced tokens before a new challenge must be performed, with the nature of SSO forcing this to be "interactive". As such I wonder if you can just use something like aws-vault or awsudo. In the past I've always either used one of these tools, or one vended by the company's infrastructure team... Many tools do not consistently support all the weird and wonderful things supported in profiles, and this is the simplest and most reliable approach I have found.

[alamb]

@timvw and @tustvold --

Given we plan to donate the object_store crate's code into the arrow-rs repo soon (TM) see apache/arrow-rs#2030, what is the plan for this PR? Will we merge it here prior to the donation or do you plan to retarget the changes to arrow-rs once it is there?

[alamb]

The archiving of this crate is imminent -- see #52

@timvw shall we try and port this PR to arrow-rs so that it doesn't get lost? I am happy to help