Skip to content

unix_timestamp on string input falls back to Spark, while to_unix_timestamp already uses the codegen dispatcher #5577

Description

@andygrove

Part of #5572.

CometUnixTimestamp only accepts date and timestamp input; anything else, which in practice means string, falls the whole projection back to Spark:

Unsupported(Some(s"unix_timestamp does not support input type: $inputType"))

spark/src/main/scala/org/apache/comet/serde/datetime.scala:319

unix_timestamp(str, fmt) on a string column is the common spelling of this function, so the supported case is arguably the rarer one. UnixTimestamp is a normal codegen expression, and StringType is in CometBatchKernelCodegen.isSupportedDataType, so the dispatcher handles it.

to_unix_timestamp is already Hybrid — CometToUnixTimestamp is a CometCodegenDispatch[ToUnixTimestamp] (serde/datetime.scala:929) — so the two functions, which differ only in argument order, behave completely differently today.

The same serde also returns a bare Incompatible(collationReason) for collated input (:314), which the same mixin would route through the dispatcher rather than falling back.

Two things to watch:

  • The input-type check is duplicated in convert (:329). Only the getSupportLevel copy is reachable by the dispatcher, so the convert copy should be removed as part of this change or it will mask the fix. See the prerequisite issue on convert-side declines.
  • Parser-policy behavior. spark.sql.legacy.timeParserPolicy is honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note in docs/source/contributor-guide/spark_configs_support.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions