Skip to content

Route unrecognized StaticInvoke and Invoke through the codegen dispatcher instead of falling back #5575

Description

@andygrove

Part of #5572.

CometStaticInvoke dispatches on an allowlist of (functionName, staticObject) pairs and falls the whole operator back to Spark for anything not in it:

case None =>
  withFallbackReason(
    expr,
    s"Static invoke expression: ${expr.functionName} is not supported")
  None

spark/src/main/scala/org/apache/comet/serde/statics.scala:61-65

StaticInvoke is a codegen-friendly node — doGenCode emits a static method call — and CometStaticInvokeCodegenDispatch (statics.scala:120) already exists and is wired for aesEncrypt, aesDecrypt and StringDecode.decode. Making it the case None default turns a hard fallback into in-pipeline execution for every lowering nobody has gotten to yet.

The compatibility guide already records the symptom. docs/source/user-guide/latest/expressions.md lists encode as "🔜 Lowers to StaticInvoke(encode) (not allowlisted); falls back", and to_binary as "Hex form accelerated; other formats fall back". Both would be covered by the default.

The same applies to Invoke, which has no entry in the serde map at all. Spark4xCometExprShim.sparkVersionSpecificExprToProtoInternal recognizes StructsToJsonEvaluator and ParseUrlEvaluator and returns None for everything else (spark/src/main/spark-4.x/org/apache/comet/shims/Spark4xCometExprShim.scala:94), after which the generic path reports "not supported". Spark 4.x lowers a growing number of RuntimeReplaceable expressions to evaluator-backed Invoke nodes, so the same catch-all is worth having there.

This is safe by construction for the encoder and deserializer trees that make up most StaticInvoke usage in typed Dataset operations: their arguments and return types are ObjectType, which is outside CometBatchKernelCodegen.isSupportedDataType, so canHandle rejects them and they fall back exactly as they do now.

Depends on the closure-serialize guard, since this is the change that starts handing the dispatcher arbitrary trees.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions