Fix EdgeWorker multiprocessing pickle error on Windows - #55284
Merged
jscheffl merged 3 commits intoSep 5, 2025
Conversation
boring-cyborg
Bot
added
area:providers
provider:edge
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Move _run_job_via_supervisor from nested function to static method to resolve "Can't pickle local object" error when launching edge worker jobs on Windows. The nested function _run_job_via_supervisor inside _launch_job_af3 could not be pickled by the multiprocessing module on Windows, causing the edge worker to fail with AttributeError when attempting to spawn child processes for job execution. Changes: - Extract _run_job_via_supervisor from _launch_job_af3 method scope - Convert to static method @staticmethod _run_job_via_supervisor - Update Process target reference to EdgeWorker._run_job_via_supervisor - Preserve all original functionality and error handling This fix enables edge worker multiprocessing compatibility on Windows while maintaining identical behavior on other platforms. The static method approach ensures the function is picklable and accessible from the module namespace.
dheerajturaga
force-pushed
the
bugfix/win-multiproc-pickle-error
branch
from
September 5, 2025 04:47
d49029e to
15db33f
Compare
dheerajturaga
force-pushed
the
bugfix/win-multiproc-pickle-error
branch
from
September 5, 2025 05:10
15db33f to
07c5a3e
Compare
dheerajturaga
force-pushed
the
bugfix/win-multiproc-pickle-error
branch
from
September 5, 2025 06:12
07c5a3e to
1cd0524
Compare
uranusjr
reviewed
Sep 5, 2025
jscheffl
approved these changes
Sep 5, 2025
jscheffl
left a comment
jscheffl
left a comment
Contributor
There was a problem hiding this comment.
Works good and look good.
Need to test Windows (later) and then will merge. My last Test with WIndows was with 2.10, thanks that you picked this up and found a gap.
Member
Author
|
This is one of several PRs needed to fix #55297 |
mangal-vairalkar
pushed a commit
to mangal-vairalkar/airflow
that referenced
this pull request
Sep 7, 2025
* Fix EdgeWorker multiprocessing pickle error on Windows Move _run_job_via_supervisor from nested function to static method to resolve "Can't pickle local object" error when launching edge worker jobs on Windows. The nested function _run_job_via_supervisor inside _launch_job_af3 could not be pickled by the multiprocessing module on Windows, causing the edge worker to fail with AttributeError when attempting to spawn child processes for job execution. Changes: - Extract _run_job_via_supervisor from _launch_job_af3 method scope - Convert to static method @staticmethod _run_job_via_supervisor - Update Process target reference to EdgeWorker._run_job_via_supervisor - Preserve all original functionality and error handling This fix enables edge worker multiprocessing compatibility on Windows while maintaining identical behavior on other platforms. The static method approach ensures the function is picklable and accessible from the module namespace. * Fix clear text url print for secutiry fix * Cleanup launch job methods to be static methods
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Sep 8, 2025
* Fix EdgeWorker multiprocessing pickle error on Windows Move _run_job_via_supervisor from nested function to static method to resolve "Can't pickle local object" error when launching edge worker jobs on Windows. The nested function _run_job_via_supervisor inside _launch_job_af3 could not be pickled by the multiprocessing module on Windows, causing the edge worker to fail with AttributeError when attempting to spawn child processes for job execution. Changes: - Extract _run_job_via_supervisor from _launch_job_af3 method scope - Convert to static method @staticmethod _run_job_via_supervisor - Update Process target reference to EdgeWorker._run_job_via_supervisor - Preserve all original functionality and error handling This fix enables edge worker multiprocessing compatibility on Windows while maintaining identical behavior on other platforms. The static method approach ensures the function is picklable and accessible from the module namespace. * Fix clear text url print for secutiry fix * Cleanup launch job methods to be static methods
78 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move _run_job_via_supervisor from nested function to static method to resolve
"Can't pickle local object" error when launching edge worker jobs on Windows.
The nested function _run_job_via_supervisor inside _launch_job_af3 could not
be pickled by the multiprocessing module on Windows, causing the edge worker
to fail with AttributeError when attempting to spawn child processes for job
execution.
Changes:
This fix enables edge worker multiprocessing compatibility on Windows while
maintaining identical behavior on other platforms. The static method approach
ensures the function is picklable and accessible from the module namespace.