Skip to content

Separate JNI-free core logic and error classification from native JNI entry points #6434

Description

@viirya

What is the problem the feature request solves?

The native JNI entry points in native/core/src/execution/jni_api.rs and native/core/src/lib.rs
mix three concerns in one function body: converting JNI arguments (JLongArray, JString,
JByteBuffer, ...), the actual native logic, and raising the JVM exception on failure. As a result:

  • The native logic of most entry points cannot be unit-tested or benchmarked without a JVM, even
    when it never calls back into the JVM (for example shuffle block decoding or the feature and
    object store checks).
  • The mapping from CometError / SparkError to the JVM exception (class, message, Spark error
    JSON, rethrown Java throwables, panic backtraces) is interleaved with the JNI calls that throw it
    in throw_exception, so the classification itself can only be tested through a JVM.

Describe the potential solution

  • Split the exception raising into a JNI-free classification step, which describes the exception
    a failed native call surfaces as (with a status code and a serializable form), and a JNI step
    that throws it. Every entry point goes through the classification via try_unwrap_or_throw,
    with exception classes and messages unchanged.
  • Turn each Java_org_apache_comet_* export into a thin wrapper that converts JNI arguments into
    plain Rust values and calls a core function without JNI types in its signature.
  • Do this incrementally: the error classification plus a few entry points first, then the
    remaining entry points that do not call back into the JVM, then the argument handling of the
    plan entry points (createPlan, executePlan, releasePlan, setShufflePartitionPusher),
    which keep depending on JVM callbacks.

Additional context

No change to the JVM side or to the exceptions users see.

Activity

  1. viirya commented on Oct 11, 2026

    @viirya
    MemberAuthor

    This is done:

    The remaining exports (columnarToRowInit/columnarToRowClose, NativeBase.release, the tracing entry points and getRustThreadId) already only call JNI-free functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ffiArrow FFI / JNI boundaryenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions