No way to downcast Arc<dyn Array> to Arc<T: Array>

[scovich]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

arrow-rs makes heavy use of ArrayRef (= Arc<dyn Array>), and it's impossible to impl Array for !Any types because Array::as_any is a required method. But for some reason we have Array: Send + Sync instead of Array: Any + Send + Sync, so there's no way to Arc::downcast an ArrayRef to e.g. Arc<StructArray>.

Describe the solution you'd like

trait Array: std::any::Any + ... { ... }

Which allows

let array: ArrayRef = ...;
let struct_array: Arc<StructArray> = Arc::downcast(array).map_err(...)?;

One big problem tho -- the blanket impl<T: Array> Array for &T means Array cannot require Any (because Any: 'static). I'm not sure how to work around that.

Describe alternatives you've considered

As far as I can tell, with today's code you have to downcast to &StructArray, clone it, and allocate a new Arc:

let array: ArrayRef = ...;
let struct_array = array.as_struct_opt().ok_or_else(...)?;
let struct_array = Arc::new(struct_array.clone());

Sure, cloning an array is probably fairly inexpensive, but there's no way it's as cheap as an arc clone.

[alamb]

Yes this is really annoying

I would really like to be able to go from ArrayRef to Arc<StructArray> or other equivalent without having to clone

[scovich]

So I did some pathfinding, and it looks like we have two conflicting objectives.

First, there's a desire for "everything possible" to impl Array so that functions which accept &dyn Array Just Work in ~all cases. In particular, blanket impl<T: Array> Array for &T and impl Array for Arc<dyn Array> mean that e.g. &Arc<Arc<dyn Array>> is an Array. Very convenient, but &T can never impl Any and so Array: Any is forbidden.

Second, we would like to downcast Arc<dyn Array> to e.g. Arc<StructArray>. But that requires knowing that we have an Arc in the first place... which we usually don't because it coerces so easily to &dyn Array. Or we have a double-arc Arc<Arc<StructArray>> which can only be safely downcast if unwrapped first.

[scovich]

One idea that's been percolating:

We already have Array::as_any(), which cuts through all the layers of wrapping and returns a reference to the underlying concrete array. What if we introduced a similar Array::as_arc() method?

It wouldn't be perfect, but in case the caller passed &Arc<dyn Array> or even &Arc<Arc<dyn Array>> -- which does happen! -- we could clone the arc instead of creating a new one. And even if we do need to create a new one, at least it's centralized (and can use the unsafe/unchecked Array -> ArrayData -> Array path) instead of scattering that logic all through the code base.

Details

The tricky part is figuring out how to handle the recursion nicely -- it would be most convenient for the method to return Option<Arc<dyn Array>>, so that nesting and full unwrapping Just Works. But then the caller ends up with an Option they have to unwrap.

Maybe we compromise by having both as_arc and as_arc_opt, where the former is the convenience wrapper (provided method) and the latter does the real work (but also has a provided default)?

pub trait Array: Debug + Send + Sync {
      ...
    fn as_arc_opt(&self) -> Option<ArrayRef> {
        None // most impl Array know nothing about presence/absence of Arc
    }
    fn as_arc(&self) -> ArrayRef {
        self.as_arc_opt().unwrap_or_else(|| make_array(self.to_data()))
    }
      ...
}

impl<T: Array> Array for &T {
      ...
    fn as_arc_opt(&self) -> Option<ArrayRef> {
        T::as_arc_opt(self)
    }
      ...
}

impl Array for Arc<dyn Array> {
      ...
    fn as_arc_opt(&self) -> Option<ArrayRef> {
        // NOTE: Recurse in order to properly unwrap e.g. `&&Arc<&Arc<dyn Array>>`
        self.as_ref().as_arc_opt().or_else(|| Some(self.clone())
    }
      ...
}

Relevant: cast_with_options could benefit from the ability to recover an Arc from a &dyn Array in case the types match -- even tho it doesn't need to downcast the result.