Interrupt a native scalar Arrow UDF when its task is killed #6811

What is the problem the feature request solves?

The experimental native scalar Arrow UDF path from #6130 runs the Python callable on the Spark task thread, inside tokio::task::block_in_place. Nothing interrupts that call when Spark kills the task. Spark's own path handles this: its MonitorThread destroys the Python worker spark.python.task.killTimeout (2s by default) after the task is interrupted.

With the native path, a cancelled job or the losing copy of a speculative task keeps its executor core, and keeps competing for the embedded interpreter's GIL, until the UDF returns on its own. A long-running or stuck UDF never releases them.

Describe the potential solution

Interrupt the running call when the task is killed:

• Record the Python thread ID before entering the callable.
• Watch the task's kill state, either through a monitor like Spark's or by having the JVM side signal native code when TaskContext.isInterrupted() becomes true.
• Raise an exception in that thread with PyThreadState_SetAsyncExc, then surface the resulting Python error as the task's failure.

PyThreadState_SetAsyncExc only takes effect when the thread next executes Python bytecode, so a call blocked inside a C extension (for example, a long PyArrow kernel that has released the GIL) is not interrupted until it returns to Python. The issue should decide how to handle that case, for example by keeping Spark's kill timeout semantics as a best effort and documenting the gap.

Additional context

The current limitation is documented in the PyArrow UDF user guide added by #6130. The concern was raised in review: #6130 (comment)