Review use of hard-coded UTC assumptions

[andygrove]

What is the problem the feature request solves?

This code pattern worries me. We should fall back to Spark if we do not know the timezone.

spark/src/main/spark-3.5/org/apache/comet/shims/CometExprShim.scala:                .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/spark-4.0/org/apache/comet/shims/CometExprShim.scala:                .setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:        castBuilder.setTimezone(timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/structs.scala:              .setTimezone(expr.timeZoneId.getOrElse("UTC"))
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")
spark/src/main/scala/org/apache/comet/serde/datetime.scala:      val timeZone = expr.timeZoneId.getOrElse("UTC")

Describe the potential solution

No response

Additional context

No response

[andygrove]

Also serde.rs to_arrow_datatype has hard-coded UTC.

[andygrove]

All of the timeZoneId.getOrElse("UTC") sites listed here are gone. #6351 replaced them with CometTimeZone.nativeId, which normalizes the timezone each serde passes to native code and reports the expression as unsupported when native code can't represent its timezone, so it falls back or runs in the codegen dispatcher. An expression with no timezone still gets "UTC", but Spark only leaves the timezone unset on casts that don't use it, which the audit in #6335 confirmed across the test suites.

The hard-coded UTC in to_arrow_datatype is intended. Every TimestampType value in a native plan is labelled Timestamp(Microsecond, "UTC"), and the session timezone travels with each expression instead of with the values. docs/source/contributor-guide/timezones.md describes that model. Closing.