Skip to content

Array functions fall back to Spark for binary and struct element types (ArraysBase type gate) #5582

Description

@andygrove

Part of #5572.

ArraysBase.isTypeSupported (spark/src/main/scala/org/apache/comet/serde/arrays.scala:888) rejects BinaryType outright and StructType pending #1307, and childTypesSupportLevel turns that into Unsupported:

.map(dt => Unsupported(Some(s"data type not supported: $dt")))

arrays.scala:911

Seven serdes mix in ArraysBase: CometArrayRemove (:36), CometFlatten (:700), CometArrayPosition (:791), CometShuffle (serde/collectionOperations.scala:64), CometArrayInsert (:409), CometCreateArray (:503) and CometArrayReverse (:572). None routes the declined types through the dispatcher, so array<struct<...>> and array<binary> — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.

docs/source/user-guide/latest/expressions.md already records this for three of them: array_position, flatten and shuffle all carry "Binary/struct/map elements fall back".

Both element types are admitted by CometBatchKernelCodegen.isSupportedDataType, which handles ArrayType, StructType and MapType recursively and accepts BinaryType as a leaf.

One important carve-out: the "null elements" half of those doc notes is not dispatchable. NullType is absent from isSupportedDataType, so array<null> will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.

CometShuffle is nondeterministic, which is fine — canHandle explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single init(partitionIndex) call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.

CometArrayInsert, CometCreateArray and CometArrayReverse also decline inside convert, which the dispatcher cannot see; see the prerequisite issue on convert-side declines.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions