Expose underlying object store capabilities (e.g. ordered listing, negative ranges)

Which part is this question about
API

Describe your question
ArrowObjectStore provides a very strong abstraction over object storage and solves many common problems well.

However, there are cases where leveraging features of the underlying storage API could lead to meaningful performance improvements. Currently, these capabilities are not exposed, which forces users into less efficient patterns.

Two concrete examples:

Ordered list results
Today, users of the ArrowObjectStore list API must assume results are unordered, requiring them to fetch all items and sort them in memory.
However, some backends (e.g., Amazon S3 for non-directory buckets) already return results in lexicographical order. If users could detect this at runtime, they could avoid unnecessary buffering and sorting, and only fall back when ordering is not guaranteed.
Negative byte ranges
Fetching ranges from the end of an object currently requires an additional HEAD request to determine object size before issuing a GET.
If the underlying store supports negative ranges directly, this extra request could be avoided, reducing latency and request overhead.

Proposal
Would the ArrowObjectStore maintainers be open to introducing a "capabilities" API that exposes which features are supported by the underlying storage implementation?

For example, something along the lines of:

supports_ordered_listing()
supports_negative_ranges()

This would allow users to write adaptive logic that takes advantage of backend-specific optimizations while preserving portability.

Curious to hear your thoughts on whether this aligns with the project's design goals.

Additional context

@alamb

@alamb what do you think about this proposal?

Since users of ObjectStore need to instantiate an ObjectStore instance already, it seems like they already know (or could track) what capabilities their particular ObjectStore instance supports

What is the usecase for a programatic API?

(The proposal sounds reasonable to me, FWIW, though I would want to minimize any change to the ObjectStore trait itself (e.g. maybe it could return a Capablities object where we could add methods like supports_ordered_listing etc

(thank you for this nicely written description)

I think that this would be useful for systems like SlateDB which accept ObjectStore as an argument and therefore cannot control what object storage service is used.

@tustvold

I wonder if we should design this similar to the attributes system in object_store, i.e. having a string-like dict or set. This would make it easier to extend in the future and may also allow custom capabilities, like this:

enum Capability {
   OrderedListing,
   NegativeRange
   Custom(Cow<'static, str>),
}

impl Capability {
    fn as_str(&self) -> &'str {...}
    fn from_str(s: &str) -> Self {...}
}

struct Capabilities {
    // impl. detail!
    caps: HashSet<Capability>,
}

impl Capabilities {
    fn has(&self, cap: &Capability) -> bool {...}
}

trait ObjectStore {
    fn capabilties(&self) -> &Capabilities;
}

We probably need some work on the internals, ideally Capabilities can be constructed in a const context so that the stores can return a reference w/o creating a singelton on the heap.

@tustvold WDYT?

My biggest question is whether ObjectStore actually always knows the answer, particularly in the case of S3, there are various S3-compatible stores that don't all behave consistently like S3.

This kind of ties into Andrew's point, that perhaps this is more a question for the config of the application object_store is being integrated into, than for object_store itself?

That being said, not opposed to this, just also not sure we always know

I made a PR for this proposal #732
Let me know what you think folks

I think enumerating object storage capabilities is interesting and helpful for callers. We just need to make sure the advertised capabilities are up to date with the latest behavior 😄, although object storage behaviors rarely change.

One thing I want to point out is that capabilities might also be tied to the configured store instance, not just the object storage behavior. There are configs that can change behavior, and thus the capabilities.

For example, Azure's list_with_offset behavior changes in emulator mode. For normal Azure, it uses the server-side list_with_offset path. But when is_emulator is enabled, it falls back to list(prefix) plus client-side filtering because Azurite does not support the startFrom query parameter.

I agree this is an interesting potential feature. My concern is that it will be hard to maintain and keep up to date as object store vendors change their support feature sets, etc

I think the ObjectStore API is already quite extensible at runtime (you can add your own ObjectStore wrappers that intercept calls and do what you want)

Maybe we could add a as_any API that would allow people who are interested in specializing the implementation to do so by checking as_any().downcast_ref() 🤔

My concern is that it will be hard to maintain and keep up to date as object store vendors change their support feature sets, etc

From researching the ordered listing behavior, we already found some edge cases for aws and azure. There's a real concern on maintainability. For example, we can mark GCS with ordered listing capability today. If and when GCS comes out with a product line without this guarantee, we'd have to keep the capabilities up to date.

Maybe we could add a as_any API that would allow people who are interested in specializing the implementation to do so by checking as_any().downcast_ref() 🤔

I had to ask AI to make sense of this 😄
Basically instead of object_store saying:

store.capabilities().has(Capability::OrderedListing)

the caller could say:

if store.as_any().downcast_ref::<AmazonS3>().is_some() {
    // I know this is AmazonS3, so I can specialize
}

as_any approach is going to work if we are dealing with vanilla object storage, like S3, GCP, or Azure. These days, many providers implement S3 APIs, but might have slightly different capabilities. Only one who creates the ObjectStore entity is aware of this.

That is why it's important to let users override default capabilities.

At the moment, only one storage GCP has enabled capabilities (ordered listing) other have none. Which means that we only need to support this for GCP.

However, to address vendor features concerns, we can just default to an empty set of capabilities and delegate to the user this configuration.

If the user is already configuring these capabilities, and they're not used internally within ObjectStore, what is the advantage of embedding them? If an application wants to make assumptions about the backing object store, that's perhaps part of that application's configuration not object stores?

I'm pretty lukewarm on adding as_any given that people tend to wrap ObjectStore in multiple layers.

I think something like as_any would only make sense if we were to implement other, more specialized methods on the specific instance.

Right now AmazonS3 doesn't supply any functionality other than the ObjectStore trait, so downcasting to AmazonS3 doesn't really give you anything. But if we were to implement AmazonS3::ordered_list then downcasting would have utility

I'm pretty lukewarm on adding as_any given that people tend to wrap ObjectStore in multiple layers.

Yeah, this makes sense to me

If the user is already configuring these capabilities, and they're not used internally within ObjectStore, what is the advantage of embedding them?

As I understood the usecase it is for conveneince so the calling application can do something special (maybe)

I think something like as_any would only make sense if we were to implement other, more specialized methods on the specific instance.

👍

There is one use case that wasn't mentioned here.

ObjectStore crate is used as a third part dependency in other application. User of this application configures ObjectStore and provides it as an argument. Application cannot make any assumptions about actual object storage, and thus resorts to most conservative implementation that will results in many extra request and therefore worst performance and higher cost.

So what do you folks think about this proposal?

I like the general direction but im fairly new to the codebase and maintaining this crate so i'll defer to others 😄

Any updates on this issue?

I am sorry -- I don't have bandwidth to help add a new API to the object store crate at this time 😢

Perhaps one of the other maintainers does

My core concern about this API still stands:

it will be hard to ... keep up to date as object store vendors change their support feature sets, etc

As far as I know there was no precedent of Object store vendors removing existing features. Therefore we should not worry about features being removed.

With this being said I think that we should keep capabilities disabled by default and delegate users to enable them.

As far as I know there was no precedent of Object store vendors removing existing features.

There is actually some precedent for this. S3 Express doesn't support all the features of standard S3, Azure has 5?? different things that claim to be blob storage with different capabilities. Whether this counts as them removing features is largely semantics, but there is a case for vendors at least muddying the waters in this area.

IMO the only coherent way to do this is for the config to be provided to the application instructing it on what capabilities are available, and given that ObjectStore itself is not a customer of this, it seems odd for it to be included here

I think that S3 Express should be considered as a different version of S3 at least it's a different bucket type.

Your point regarding application will work for some cases, however there are some APIs changes that need to be made to ObjectStore crate, for example for supporting negative ranges.

Perhaps I am missing something but we've supported negative ranges for a while - https://docs.rs/object_store/latest/object_store/enum.GetRange.html

This will require range to be defined using i64 instead of u64, right?

There are a few different form of range request - https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Range

We support all of them apart from multi-range (as very little/nothing supports it), including suffix requests (what I presume you mean by negative range requests).

Ok, thanks for explaining this, yes I was talking about suffix requests.

Azure blob storage doesn't not support suffix range requests. This would be a case when application could benefit from knowing object storage capabilities.

I see that in case user will Error::NotSupported. I guess it's good enough.

Navigation Menu

Repository navigation

Expose underlying object store capabilities (e.g. ordered listing, negative ranges) #675

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Expose underlying object store capabilities (e.g. ordered listing, negative ranges) #675

Description

Activity

vitoordaz commented on Mar 27, 2026

alamb commented on Mar 29, 2026

alamb commented on Mar 29, 2026

vitoordaz commented on Mar 30, 2026

crepererum commented on Mar 31, 2026

tustvold commented on Mar 31, 2026

vitoordaz commented on May 28, 2026

kevinjqliu commented on Jun 9, 2026

alamb commented on Jun 10, 2026

kevinjqliu commented on Jun 10, 2026

vitoordaz commented on Jun 10, 2026

tustvold commented on Jun 11, 2026

kylebarron commented on Jun 11, 2026

alamb commented on Jun 12, 2026

vitoordaz commented on Jun 12, 2026

vitoordaz commented on Jun 18, 2026

kevinjqliu commented on Jun 19, 2026

vitoordaz commented on Jul 22, 2026

alamb commented on Jul 23, 2026

alamb commented on Jul 23, 2026

vitoordaz commented on Jul 24, 2026

tustvold commented on Jul 24, 2026

vitoordaz commented on Jul 24, 2026

tustvold commented on Jul 24, 2026

vitoordaz commented on Jul 24, 2026

tustvold commented on Jul 24, 2026

vitoordaz commented on Jul 24, 2026

vitoordaz commented on Jul 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions