Part of #5572.
timestamp_seconds accepts only int, long, float and double, and falls the whole projection back to Spark for anything else:
case IntegerType | LongType | FloatType | DoubleType => Compatible()
case dt => Unsupported(Some(s"timestamp_seconds does not support input type $dt"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala:495-496
The serde's own getUnsupportedReasons spells out what that leaves behind: "DecimalType, ByteType, and ShortType fall back to Spark."
The decimal case is the one that matters. timestamp_seconds(decimal_col) is how you get sub-microsecond-safe epoch conversion in Spark, and SecondsToTimestamp.doGenCode has a dedicated decimal branch that does the rounding Spark's semantics require. DecimalType, ByteType and ShortType are all in CometBatchKernelCodegen.isSupportedDataType.
Fix: mix CodegenDispatchFallback into CometSecondsToTimestamp. It currently extends CometScalarFunction[SecondsToTimestamp], which is fine — the mixin composes with it, as CometConcat and CometMapFromEntries already demonstrate.
Part of #5572.
timestamp_secondsaccepts only int, long, float and double, and falls the whole projection back to Spark for anything else:spark/src/main/scala/org/apache/comet/serde/datetime.scala:495-496The serde's own
getUnsupportedReasonsspells out what that leaves behind: "DecimalType,ByteType, andShortTypefall back to Spark."The decimal case is the one that matters.
timestamp_seconds(decimal_col)is how you get sub-microsecond-safe epoch conversion in Spark, andSecondsToTimestamp.doGenCodehas a dedicated decimal branch that does the rounding Spark's semantics require.DecimalType,ByteTypeandShortTypeare all inCometBatchKernelCodegen.isSupportedDataType.Fix: mix
CodegenDispatchFallbackintoCometSecondsToTimestamp. It currently extendsCometScalarFunction[SecondsToTimestamp], which is fine — the mixin composes with it, asCometConcatandCometMapFromEntriesalready demonstrate.