native: `JVMClasses::with_env: JAVA_VM not initialized` aborts a multi-suite JVM

[ErikBPF]

Issue B — native: JVMClasses::with_env: JAVA_VM not initialized aborts a multi-suite JVM

What it does: reports a native/JNI lifecycle abort that appears when many
Comet suites share one JVM, and asks whether that mode is supported.
Expected gain: unblocks one-JVM local runs (the current local gate must split
into four JVMs, ~52 min instead of ~37 min); if it is a real bug it can also
affect any long-lived process that tears down and re-initialises the bridge.
Type: bug report / question. No CI impact today (CI splits suites into four
groups).

Symptom

Running the Comet suites in one JVM aborts partway through:

thread '<unnamed>' panicked at jni-bridge/src/lib.rs:353:9:
JVMClasses::with_env: JAVA_VM not initialized
fatal runtime error: Rust cannot catch foreign exceptions, aborting

Observed after CometNativeShuffleSuite in a combined run of 132 suites (37:03
min, every displayed test passing up to that point, no summary printed).

Resource constraints ruled out (validated 2026-09-21)

A reviewer asked whether this could be a resource limit rather than a lifecycle
bug. Validated: it is not resource-bound. Two combined 132-suite one-JVM runs
were executed on the same revision (a51ff5cd8), image and host, each with
--memory=64g --memory-swap=64g --cpus=28 --pids-limit=-1:

• Run 1: BUILD SUCCESS, Suites: completed 132, aborted 0, Tests: succeeded 3506, failed 0, canceled 6, ignored 18.
• Run 2: aborted with the identical panic, immediately after the first
  CometNativeShuffleSuite test (native shuffle callback registration preserves the existing createPlan JNI signature).

The earlier run at --memory=24g aborted at the same point. So 2 of 3
combined-JVM runs abort at the same suite, and one at 64 GB passes — the trigger
is nondeterministic (ordering/state/timing), not memory or thread limits:

• No OutOfMemoryError, no Killed/SIGKILL, no exit 137 in either log.
• journalctl -k since 2026-09-21 20:00 has no OOM-kill or pthread_create
  entries.
• The abort is a Rust debug_assert!(JAVA_VM.get().is_some()) — a
  lifecycle/initialisation condition, not an allocation failure — which fires
  because the ci profile enables debug-assertions.
• One-JVM execution is not inherently broken: a 262-suite single-JVM run on the
  older ae19c6bbe revision completed 3296 tests / 0 failed.

Remaining unknown is the trigger, not the resource class. The ae19c6bbe vs
a51ff5cd8 (134-commit) difference is still a confound for whether it is a
regression or pre-existing ordering. Also observed, caught and harmless: run 1
logged repeated Comet native panic: ... datafusion.no_such_namespace.option from
datafusion-execution-55.1.0/src/config.rs:186 and recovered, with all tests
passing.

Why it looks like ordering/state, not a suite bug

• org.apache.comet.exec.CometNativeShuffleSuite alone: 57 succeeded / 0 failed
  in 54 s.
• The same 132 suites split into the four CI linux-test groups, each in its own
  JVM: 3506 succeeded / 0 failed (scans 539, shuffle 497, exec 952,
  expressions 1518).

Where

native/jni-bridge/src/lib.rs: JAVA_VM: OnceCell<JavaVM> (:49), JVM_CLASSES
(:256), JVMClasses::init(env) (:260), with_env (:348) begins with
debug_assert!(JAVA_VM.get().is_some(), ...); the assert fires because the ci
profile enables debug-assertions. Callers include
shuffle_partition_pusher.rs:206,225,254.

Ask

Is one-JVM execution of the whole suite a supported mode? If not, document it. If
yes, the JAVA_VM lifecycle needs to survive a suite that tears down and
re-initialises the JVM/native bridge (likely a re-JNI_OnLoad/detach
interaction). The trigger is nondeterministic — roughly 2 of 3 combined-JVM runs
abort at the first CometNativeShuffleSuite test — so the reproducer is to run the
combined 132-suite invocation twice on the same revision and host; no existing
upstream issue matches "JAVA_VM" / "JNI_OnLoad".

[zhangfengcdt]

I have a fix PR and I can take it.