Part of #5572.
ArraysBase.isTypeSupported (spark/src/main/scala/org/apache/comet/serde/arrays.scala:888) rejects BinaryType outright and StructType pending #1307, and childTypesSupportLevel turns that into Unsupported:
.map(dt => Unsupported(Some(s"data type not supported: $dt")))
arrays.scala:911
Seven serdes mix in ArraysBase: CometArrayRemove (:36), CometFlatten (:700), CometArrayPosition (:791), CometShuffle (serde/collectionOperations.scala:64), CometArrayInsert (:409), CometCreateArray (:503) and CometArrayReverse (:572). None routes the declined types through the dispatcher, so array<struct<...>> and array<binary> — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.
docs/source/user-guide/latest/expressions.md already records this for three of them: array_position, flatten and shuffle all carry "Binary/struct/map elements fall back".
Both element types are admitted by CometBatchKernelCodegen.isSupportedDataType, which handles ArrayType, StructType and MapType recursively and accepts BinaryType as a leaf.
One important carve-out: the "null elements" half of those doc notes is not dispatchable. NullType is absent from isSupportedDataType, so array<null> will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.
CometShuffle is nondeterministic, which is fine — canHandle explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single init(partitionIndex) call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.
CometArrayInsert, CometCreateArray and CometArrayReverse also decline inside convert, which the dispatcher cannot see; see the prerequisite issue on convert-side declines.
Part of #5572.
ArraysBase.isTypeSupported(spark/src/main/scala/org/apache/comet/serde/arrays.scala:888) rejectsBinaryTypeoutright andStructTypepending #1307, andchildTypesSupportLevelturns that intoUnsupported:arrays.scala:911Seven serdes mix in
ArraysBase:CometArrayRemove(:36),CometFlatten(:700),CometArrayPosition(:791),CometShuffle(serde/collectionOperations.scala:64),CometArrayInsert(:409),CometCreateArray(:503) andCometArrayReverse(:572). None routes the declined types through the dispatcher, soarray<struct<...>>andarray<binary>— both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.docs/source/user-guide/latest/expressions.mdalready records this for three of them:array_position,flattenandshuffleall carry "Binary/struct/map elements fall back".Both element types are admitted by
CometBatchKernelCodegen.isSupportedDataType, which handlesArrayType,StructTypeandMapTyperecursively and acceptsBinaryTypeas a leaf.One important carve-out: the "null elements" half of those doc notes is not dispatchable.
NullTypeis absent fromisSupportedDataType, soarray<null>will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.CometShuffleis nondeterministic, which is fine —canHandleexplicitly admits nondeterministic and stateful expressions, and each cache entry gets a singleinit(partitionIndex)call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.CometArrayInsert,CometCreateArrayandCometArrayReversealso decline insideconvert, which the dispatcher cannot see; see the prerequisite issue onconvert-side declines.