Skip to content

Honor the deterministic flag for Rust UDFs #5249

Description

@andygrove

Follow-up from review of #4459.

CometRustUDF.register accepts a deterministic parameter and carries it in the RustUdfCall proto, but the native side ignores it: ImportedCScalarUdf::try_new hardcodes Volatility::Immutable. A nondeterministic UDF planned as immutable may be constant-folded, evaluated once and reused, or eliminated as a common subexpression.

#4459 closes the hole by rejecting deterministic = false at registration, so the flag cannot silently lie. This issue is about actually supporting it.

The obstacle is lifetime mismatch. The Signature is built once per library load and cached process-wide per (library_path), while determinism is declared per CometRustUDF.register call. Two registrations of the same kernel with different determinism would need different volatility from one cached ScalarUDFImpl.

Options worth weighing:

  • Key the cached adapter on (library_path, name, volatility) rather than on the library alone.
  • Build the ScalarUDFImpl per call site in the planner from the cached kernel, taking volatility from RustUdfCall.deterministic, and keep only the loaded library in the cache.
  • Let the kernel declare its own volatility over the ABI and validate the registration against it, which overlaps with the get_property idea in Add a forward-flexible property slot to the Comet UDF C ABI #5254.

Once this lands, drop the guard in CometRustUDF.register, the note on the deterministic field in expr.proto, the "Immutable functions only" limitation in docs/source/user-guide/latest/rust_udfs.md, and the corresponding note on the CometCScalarUdf trait docs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions