Part of #5572.
CometStringTranslate reports Incompatible unconditionally:
override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible(
Some(incompatReason))
spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120
The reason is real — DataFusion's translate iterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with no CodegenDispatchFallback mixin, the consequence is that translate falls the whole projection back to Spark unless the user sets spark.comet.expression.StringTranslate.allowIncompatible=true and accepts wrong answers.
Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See CometInitCap (:146), CometStringReplace (:177), CometCaseConversionBase for upper / lower (:50), and the regex family at :368, :470, :508, :545, :598.
Fix: mix CodegenDispatchFallback into CometStringTranslate. StringTranslate is a plain codegen expression over StringType, so canHandle admits it.
Once that lands, docs/source/user-guide/latest/expressions.md should report translate as Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.
Part of #5572.
CometStringTranslatereportsIncompatibleunconditionally:spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120The reason is real — DataFusion's
translateiterates Unicode graphemes where Spark uses code points, and substitutes U+0000 rather than treating it as a deletion sentinel. But with noCodegenDispatchFallbackmixin, the consequence is thattranslatefalls the whole projection back to Spark unless the user setsspark.comet.expression.StringTranslate.allowIncompatible=trueand accepts wrong answers.Every comparable string function in the same file already has the better shape: compatible by default through the dispatcher, native as an explicit opt-in. See
CometInitCap(:146),CometStringReplace(:177),CometCaseConversionBaseforupper/lower(:50), and the regex family at:368,:470,:508,:545,:598.Fix: mix
CodegenDispatchFallbackintoCometStringTranslate.StringTranslateis a plain codegen expression overStringType, socanHandleadmits it.Once that lands,
docs/source/user-guide/latest/expressions.mdshould reporttranslateas Hybrid, and the existing note about the grapheme/code-point difference becomes an opt-in caveat rather than a description of default behavior.