Part of #5572.
CometUnixTimestamp only accepts date and timestamp input; anything else, which in practice means string, falls the whole projection back to Spark:
Unsupported(Some(s"unix_timestamp does not support input type: $inputType"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala:319
unix_timestamp(str, fmt) on a string column is the common spelling of this function, so the supported case is arguably the rarer one. UnixTimestamp is a normal codegen expression, and StringType is in CometBatchKernelCodegen.isSupportedDataType, so the dispatcher handles it.
to_unix_timestamp is already Hybrid — CometToUnixTimestamp is a CometCodegenDispatch[ToUnixTimestamp] (serde/datetime.scala:929) — so the two functions, which differ only in argument order, behave completely differently today.
The same serde also returns a bare Incompatible(collationReason) for collated input (:314), which the same mixin would route through the dispatcher rather than falling back.
Two things to watch:
- The input-type check is duplicated in
convert (:329). Only the getSupportLevel copy is reachable by the dispatcher, so the convert copy should be removed as part of this change or it will mask the fix. See the prerequisite issue on convert-side declines.
- Parser-policy behavior.
spark.sql.legacy.timeParserPolicy is honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note in docs/source/contributor-guide/spark_configs_support.md.
Part of #5572.
CometUnixTimestamponly accepts date and timestamp input; anything else, which in practice means string, falls the whole projection back to Spark:spark/src/main/scala/org/apache/comet/serde/datetime.scala:319unix_timestamp(str, fmt)on a string column is the common spelling of this function, so the supported case is arguably the rarer one.UnixTimestampis a normal codegen expression, andStringTypeis inCometBatchKernelCodegen.isSupportedDataType, so the dispatcher handles it.to_unix_timestampis already Hybrid —CometToUnixTimestampis aCometCodegenDispatch[ToUnixTimestamp](serde/datetime.scala:929) — so the two functions, which differ only in argument order, behave completely differently today.The same serde also returns a bare
Incompatible(collationReason)for collated input (:314), which the same mixin would route through the dispatcher rather than falling back.Two things to watch:
convert(:329). Only thegetSupportLevelcopy is reachable by the dispatcher, so theconvertcopy should be removed as part of this change or it will mask the fix. See the prerequisite issue onconvert-side declines.spark.sql.legacy.timeParserPolicyis honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note indocs/source/contributor-guide/spark_configs_support.md.