[SPARK-58751][SS][PYTHON] Stop leaking Python workers on TransformWithState init failure - #57941
Closed
jon-gao-db wants to merge 2 commits into
Closed
jon-gao-db wants to merge 2 commits into
jon-gao-db wants to merge 2 commits into
Conversation
jon-gao-db
marked this pull request as ready for review
August 12, 2026 19:08
jon-gao-db
changed the title
jiateoh
approved these changes
Aug 12, 2026
jiateoh
left a comment
jiateoh
left a comment
Contributor
There was a problem hiding this comment.
Fixes LGTM, thanks for the test coverage as well. Added two small comments and we'll also need an additional spark reviewer for the final approval + merge.
Comment on lines
+59
to
+62
| val workerAlive = new AtomicBoolean(false) | ||
| val initCount = new AtomicInteger(0) | ||
| val processCount = new AtomicInteger(0) | ||
| val stopCount = new AtomicInteger(0) |
Contributor
There was a problem hiding this comment.
Is there any actual concurrency here? Seems like regular ints/booleans would suffice? (Doesn't hurt to have atomic though)
| assert(runners.count(_.workerAlive.get()) === 0) | ||
| } | ||
|
|
||
| test("a stop failure does not mask the original init failure") { |
Contributor
There was a problem hiding this comment.
do we need to also test stop + process failures? (since this is only stop + init)
uros-b
approved these changes
Aug 13, 2026
Member
|
Thank you @jon-gao-db and @jiateoh! Adding @zhengruifeng here too |
jon-gao-db
force-pushed
the
fix/transform-with-state-pre-init-cleanup
branch
from
August 13, 2026 22:32
d24ac86 to
d7e55e6
Compare
jon-gao-db
force-pushed
the
fix/transform-with-state-pre-init-cleanup
branch
from
August 13, 2026 22:35
d7e55e6 to
568f167
Compare
Contributor
|
The CI failure is only from docker integration which is unrelated. |
HeartSaVioR
pushed a commit
that referenced
this pull request
Aug 14, 2026
…hState init failure ### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. JIRA: https://issues.apache.org/jira/browse/SPARK-58751 ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). Closes #57941 from jon-gao-db/fix/transform-with-state-pre-init-cleanup. Lead-authored-by: jon-gao-db <242085654+jon-gao-db@users.noreply.github.com> Co-authored-by: Jonathan Gao <jonathan-gao@hotmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com> (cherry picked from commit 3b39c59) Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
HeartSaVioR
pushed a commit
that referenced
this pull request
Aug 14, 2026
…hState init failure ### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. JIRA: https://issues.apache.org/jira/browse/SPARK-58751 ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). Closes #57941 from jon-gao-db/fix/transform-with-state-pre-init-cleanup. Lead-authored-by: jon-gao-db <242085654+jon-gao-db@users.noreply.github.com> Co-authored-by: Jonathan Gao <jonathan-gao@hotmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com> (cherry picked from commit 3b39c59) Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
HeartSaVioR
pushed a commit
that referenced
this pull request
Aug 14, 2026
…hState init failure ### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. JIRA: https://issues.apache.org/jira/browse/SPARK-58751 ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). Closes #57941 from jon-gao-db/fix/transform-with-state-pre-init-cleanup. Lead-authored-by: jon-gao-db <242085654+jon-gao-db@users.noreply.github.com> Co-authored-by: Jonathan Gao <jonathan-gao@hotmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com> (cherry picked from commit 3b39c59) Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
HeartSaVioR
pushed a commit
that referenced
this pull request
Aug 14, 2026
…hState init failure ### What changes were proposed in this pull request? This patch makes the driver-side `TransformWithStateInPySpark` pre-initialization runner clean up on every lifecycle path. It moves `init()` and `process()` into `Utils.tryWithSafeFinally`, guaranteeing that `stop()` runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying on `tryWithSafeFinally`'s suppressed-exception behavior. The runner's state-server daemon thread is nullable until the end of initialization, so `stop()` now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup. JIRA: https://issues.apache.org/jira/browse/SPARK-58751 ### Why are the changes needed? `StreamingPythonRunner.init()` creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reached `stop()`, leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting. ### Does this PR introduce _any_ user-facing change? No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it. ### How was this patch tested? Added `TransformWithStateInPySparkPreInitCleanupSuite` with six tests and no real Python worker dependency. Static validation passed with `git diff --check` and the changed-file line-length check. The focused Spark test could not run because this checkout does not have `sbt` installed and its launcher download was unavailable. A Maven `test-compile` fallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (Codex CLI). Closes #57941 from jon-gao-db/fix/transform-with-state-pre-init-cleanup. Lead-authored-by: jon-gao-db <242085654+jon-gao-db@users.noreply.github.com> Co-authored-by: Jonathan Gao <jonathan-gao@hotmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com> (cherry picked from commit 3b39c59) Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch makes the driver-side
TransformWithStateInPySparkpre-initialization runner clean up on every lifecycle path. It movesinit()andprocess()intoUtils.tryWithSafeFinally, guaranteeing thatstop()runs when initialization or processing fails. It also preserves the original initialization failure when cleanup itself throws by relying ontryWithSafeFinally's suppressed-exception behavior.The runner's state-server daemon thread is nullable until the end of initialization, so
stop()now checks for null before interrupting it. A six-case regression suite covers initialization failure cleanup, repeated failures, suppressed cleanup errors, process failure wrapping, the success path, and stopping before state-server startup.JIRA: https://issues.apache.org/jira/browse/SPARK-58751
Why are the changes needed?
StreamingPythonRunner.init()creates the Python worker before it finishes initialization. If initialization fails afterward, the previous code never reachedstop(), leaking the worker and its associated resources for the driver's lifetime. Repeated streaming restarts can accumulate these leaked resources and eventually prevent new isolated workers from starting.Does this PR introduce any user-facing change?
No. Successful execution behavior is unchanged. This only restores cleanup on existing failure paths and preserves the original initialization error instead of allowing cleanup failures or a null-thread error to obscure it.
How was this patch tested?
Added
TransformWithStateInPySparkPreInitCleanupSuitewith six tests and no real Python worker dependency. Static validation passed withgit diff --checkand the changed-file line-length check.The focused Spark test could not run because this checkout does not have
sbtinstalled and its launcher download was unavailable. A Maventest-compilefallback also could not resolve dependencies because the configured Maven mirrors were unreachable from the environment.Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (Codex CLI).