Spark <= 4.2 codegen omits the days field when hashing CalendarIntervalType (hash / xxhash64), and Comet's native hasher matches that (#5135). SPARK-58236 (apache/spark commit a2616be0f) fixes Spark's codegen to include days starting in Spark 4.3, aligning it with the interpreted path.
When Comet adds a Spark 4.3 profile, the hash becomes version-dependent:
- resolve an
include_days boolean in ShimCometConf per the shim convention
- carry it through the protobuf hash expression
- branch on it in
hash_array_interval_month_day_nano! in native/spark-expr/src/hash_funcs/utils.rs
Until then, all supported versions (v3.4.3, v3.5.8, v4.0.1, v4.1.2, v4.2.0) omit days and the fix has not been backported to any release branch, so no native change is needed yet.
Follow-up from review discussion on #5135.
Spark <= 4.2 codegen omits the
daysfield when hashingCalendarIntervalType(hash/xxhash64), and Comet's native hasher matches that (#5135). SPARK-58236 (apache/spark commita2616be0f) fixes Spark's codegen to includedaysstarting in Spark 4.3, aligning it with the interpreted path.When Comet adds a Spark 4.3 profile, the hash becomes version-dependent:
include_daysboolean inShimCometConfper the shim conventionhash_array_interval_month_day_nano!innative/spark-expr/src/hash_funcs/utils.rsUntil then, all supported versions (
v3.4.3,v3.5.8,v4.0.1,v4.1.2,v4.2.0) omitdaysand the fix has not been backported to any release branch, so no native change is needed yet.Follow-up from review discussion on #5135.