Background
SPARK-55278: Language-agnostic UDF Protocol for Spark is an accepted SPIP, currently under implementation and scoped at Spark 4.3.0/4.4.0. It defines a unified API and execution protocol for user code that runs outside the Spark engine process, standardizing on Arrow for data, gRPC for the service definition, a declarative worker specification, and language-agnostic Catalyst logical and physical nodes so the planner can reason about an external UDF without knowing what language implements it.
This was raised by @wForget on #4459, which adds Comet's own native scalar UDF path. That PR should not wait on this, but the two surfaces will have to be reconciled, and it is cheaper to know where they diverge before Comet's API has users.
Where the two designs actually differ
Comet's path and the SPIP solve overlapping problems with opposite process models, so this is not a matter of adopting one wholesale.
|
Comet native UDFs (#4459) |
SPARK-55278 |
| Locality |
in-process, dlopen of a cdylib in the executor |
out-of-process worker, IPC |
| Data |
Arrow C Data Interface, zero copy |
Arrow over gRPC |
| Coupling |
C ABI of function pointers, versioned per Comet release |
gRPC service definition, versioned by Spark |
| Planning |
Comet serde emits NativeScalarUdf in its own proto |
language-agnostic Catalyst nodes |
| Failure domain |
a UDF bug can take down the executor |
worker crash is isolated |
The in-process design is the reason Comet's path exists: it puts user code inside the DataFusion plan with no serialization boundary, which is exactly what an out-of-process protocol cannot offer. So the SPIP does not replace it.
What alignment could mean
Worth deciding, roughly in order of how much it constrains #4459's API:
- Recognize the SPIP's Catalyst nodes. Once Spark ships language-agnostic logical/physical UDF nodes, Comet's
CometExecRule and QueryPlanSerde will see them in plans whether or not Comet participates. At minimum Comet needs a deliberate fallback story for them; at best a Comet-native execution of a worker-backed UDF.
- Reuse the SPIP's registration vocabulary.
CometNativeUDF.register currently invents its own signature declaration (name, library path, input types, return type, determinism). If the SPIP's worker specification names the same concepts, Comet should borrow the names and shapes rather than diverge for no reason. This is the cheapest thing to get right and the most annoying to change later.
- Serve a SPIP worker from a Comet-native plan. A worker-backed UDF inside a
CometNativeExec would otherwise force a transition back to the JVM. Whether Comet can drive the protocol from Rust is an open question and probably the largest piece here.
- Decide whether the C ABI stays. If the SPIP's protocol becomes the way non-JVM UDFs reach Spark, Comet's C ABI is either a fast path for the in-process case that the protocol cannot cover, or a duplicate surface to deprecate. The versioning argument recorded in
comet-udf-sdk (no DataFusion type crosses the boundary, so a compiled UDF survives Comet's DataFusion upgrades) applies to the C ABI and not obviously to a gRPC protocol Spark versions on its own cadence.
Not blocking
Comet supports Spark 3.4 through 4.1 today, so anything landing in Spark 4.3+ is some way off, and the SPIP's own subtasks are still in progress. This issue is to track the design and to keep #4459's follow-ups from painting Comet into a corner, not to gate that PR.
References
Background
SPARK-55278: Language-agnostic UDF Protocol for Spark is an accepted SPIP, currently under implementation and scoped at Spark 4.3.0/4.4.0. It defines a unified API and execution protocol for user code that runs outside the Spark engine process, standardizing on Arrow for data, gRPC for the service definition, a declarative worker specification, and language-agnostic Catalyst logical and physical nodes so the planner can reason about an external UDF without knowing what language implements it.
This was raised by @wForget on #4459, which adds Comet's own native scalar UDF path. That PR should not wait on this, but the two surfaces will have to be reconciled, and it is cheaper to know where they diverge before Comet's API has users.
Where the two designs actually differ
Comet's path and the SPIP solve overlapping problems with opposite process models, so this is not a matter of adopting one wholesale.
dlopenof a cdylib in the executorNativeScalarUdfin its own protoThe in-process design is the reason Comet's path exists: it puts user code inside the DataFusion plan with no serialization boundary, which is exactly what an out-of-process protocol cannot offer. So the SPIP does not replace it.
What alignment could mean
Worth deciding, roughly in order of how much it constrains #4459's API:
CometExecRuleandQueryPlanSerdewill see them in plans whether or not Comet participates. At minimum Comet needs a deliberate fallback story for them; at best a Comet-native execution of a worker-backed UDF.CometNativeUDF.registercurrently invents its own signature declaration (name, library path, input types, return type, determinism). If the SPIP's worker specification names the same concepts, Comet should borrow the names and shapes rather than diverge for no reason. This is the cheapest thing to get right and the most annoying to change later.CometNativeExecwould otherwise force a transition back to the JVM. Whether Comet can drive the protocol from Rust is an open question and probably the largest piece here.comet-udf-sdk(no DataFusion type crosses the boundary, so a compiled UDF survives Comet's DataFusion upgrades) applies to the C ABI and not obviously to a gRPC protocol Spark versions on its own cadence.Not blocking
Comet supports Spark 3.4 through 4.1 today, so anything landing in Spark 4.3+ is some way off, and the SPIP's own subtasks are still in progress. This issue is to track the design and to keep #4459's follow-ups from painting Comet into a corner, not to gate that PR.
References