What is the problem the feature request solves?
The native JNI entry points in native/core/src/execution/jni_api.rs and native/core/src/lib.rs
mix three concerns in one function body: converting JNI arguments (JLongArray, JString,
JByteBuffer, ...), the actual native logic, and raising the JVM exception on failure. As a result:
- The native logic of most entry points cannot be unit-tested or benchmarked without a JVM, even
when it never calls back into the JVM (for example shuffle block decoding or the feature and
object store checks).
- The mapping from
CometError / SparkError to the JVM exception (class, message, Spark error
JSON, rethrown Java throwables, panic backtraces) is interleaved with the JNI calls that throw it
in throw_exception, so the classification itself can only be tested through a JVM.
Describe the potential solution
- Split the exception raising into a JNI-free classification step, which describes the exception
a failed native call surfaces as (with a status code and a serializable form), and a JNI step
that throws it. Every entry point goes through the classification via try_unwrap_or_throw,
with exception classes and messages unchanged.
- Turn each
Java_org_apache_comet_* export into a thin wrapper that converts JNI arguments into
plain Rust values and calls a core function without JNI types in its signature.
- Do this incrementally: the error classification plus a few entry points first, then the
remaining entry points that do not call back into the JVM, then the argument handling of the
plan entry points (createPlan, executePlan, releasePlan, setShufflePartitionPusher),
which keep depending on JVM callbacks.
Additional context
No change to the JVM side or to the exceptions users see.
What is the problem the feature request solves?
The native JNI entry points in
native/core/src/execution/jni_api.rsandnative/core/src/lib.rsmix three concerns in one function body: converting JNI arguments (
JLongArray,JString,JByteBuffer, ...), the actual native logic, and raising the JVM exception on failure. As a result:when it never calls back into the JVM (for example shuffle block decoding or the feature and
object store checks).
CometError/SparkErrorto the JVM exception (class, message, Spark errorJSON, rethrown Java throwables, panic backtraces) is interleaved with the JNI calls that throw it
in
throw_exception, so the classification itself can only be tested through a JVM.Describe the potential solution
a failed native call surfaces as (with a status code and a serializable form), and a JNI step
that throws it. Every entry point goes through the classification via
try_unwrap_or_throw,with exception classes and messages unchanged.
Java_org_apache_comet_*export into a thin wrapper that converts JNI arguments intoplain Rust values and calls a core function without JNI types in its signature.
remaining entry points that do not call back into the JVM, then the argument handling of the
plan entry points (
createPlan,executePlan,releasePlan,setShufflePartitionPusher),which keep depending on JVM callbacks.
Additional context
No change to the JVM side or to the exceptions users see.