You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Run array_contains on float elements natively with Spark's equality instead of through the codegen dispatcher #6520
array_contains on an array whose elements hold a FLOAT or DOUBLE, at any depth, reports Incompatible, because DataFusion's array_contains compares elements by their bits, while Spark's genEqual treats -0.0 and 0.0 as equal and all NaNs as equal. CometArrayContains mixes in CodegenDispatchFallback, so by default these calls go through the JVM codegen dispatcher. That matches Spark, but the expression leaves the native path. With spark.comet.expression.ArrayContains.allowIncompatible=true they get the bitwise kernel, and the wrong answer for those values.
Describe the potential solution
A native array_contains for float elements built on float_semantics::spark_equality, the way spark_array_remove in #6518 is: for a constant value, one test per element in one pass over all the values, and for a value per row or nested elements, spark_equality. It has to keep Spark's three-valued result, which is null when nothing matches and the array holds a null element. CometArrayContains could then report Compatible for float elements and stop dispatching.
array_remove, array_position and array_contains all find the elements of a row that equal a value under Spark's rules, today in three different ways, so one shared kernel that produces that match could serve all three.
Activity
zhangfengcdt commentedon Oct 2, 2026
take
feat: run array_contains on float elements natively with Spark's equa… (