Part of #5572.
CometStaticInvoke dispatches on an allowlist of (functionName, staticObject) pairs and falls the whole operator back to Spark for anything not in it:
case None =>
withFallbackReason(
expr,
s"Static invoke expression: ${expr.functionName} is not supported")
None
spark/src/main/scala/org/apache/comet/serde/statics.scala:61-65
StaticInvoke is a codegen-friendly node — doGenCode emits a static method call — and CometStaticInvokeCodegenDispatch (statics.scala:120) already exists and is wired for aesEncrypt, aesDecrypt and StringDecode.decode. Making it the case None default turns a hard fallback into in-pipeline execution for every lowering nobody has gotten to yet.
The compatibility guide already records the symptom. docs/source/user-guide/latest/expressions.md lists encode as "🔜 Lowers to StaticInvoke(encode) (not allowlisted); falls back", and to_binary as "Hex form accelerated; other formats fall back". Both would be covered by the default.
The same applies to Invoke, which has no entry in the serde map at all. Spark4xCometExprShim.sparkVersionSpecificExprToProtoInternal recognizes StructsToJsonEvaluator and ParseUrlEvaluator and returns None for everything else (spark/src/main/spark-4.x/org/apache/comet/shims/Spark4xCometExprShim.scala:94), after which the generic path reports "not supported". Spark 4.x lowers a growing number of RuntimeReplaceable expressions to evaluator-backed Invoke nodes, so the same catch-all is worth having there.
This is safe by construction for the encoder and deserializer trees that make up most StaticInvoke usage in typed Dataset operations: their arguments and return types are ObjectType, which is outside CometBatchKernelCodegen.isSupportedDataType, so canHandle rejects them and they fall back exactly as they do now.
Depends on the closure-serialize guard, since this is the change that starts handing the dispatcher arbitrary trees.
Part of #5572.
CometStaticInvokedispatches on an allowlist of(functionName, staticObject)pairs and falls the whole operator back to Spark for anything not in it:spark/src/main/scala/org/apache/comet/serde/statics.scala:61-65StaticInvokeis a codegen-friendly node —doGenCodeemits a static method call — andCometStaticInvokeCodegenDispatch(statics.scala:120) already exists and is wired foraesEncrypt,aesDecryptandStringDecode.decode. Making it thecase Nonedefault turns a hard fallback into in-pipeline execution for every lowering nobody has gotten to yet.The compatibility guide already records the symptom.
docs/source/user-guide/latest/expressions.mdlistsencodeas "🔜 Lowers toStaticInvoke(encode)(not allowlisted); falls back", andto_binaryas "Hex form accelerated; other formats fall back". Both would be covered by the default.The same applies to
Invoke, which has no entry in the serde map at all.Spark4xCometExprShim.sparkVersionSpecificExprToProtoInternalrecognizesStructsToJsonEvaluatorandParseUrlEvaluatorand returnsNonefor everything else (spark/src/main/spark-4.x/org/apache/comet/shims/Spark4xCometExprShim.scala:94), after which the generic path reports "not supported". Spark 4.x lowers a growing number ofRuntimeReplaceableexpressions to evaluator-backedInvokenodes, so the same catch-all is worth having there.This is safe by construction for the encoder and deserializer trees that make up most
StaticInvokeusage in typed Dataset operations: their arguments and return types areObjectType, which is outsideCometBatchKernelCodegen.isSupportedDataType, socanHandlerejects them and they fall back exactly as they do now.Depends on the closure-serialize guard, since this is the change that starts handing the dispatcher arbitrary trees.