[v3-3-test] Fix orphaned subprocesses and supervisor crash on heartbeat 409 (#65738) - #71146
Merged
Conversation
* Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination. (cherry picked from commit 6145746)
amoghrajesh
approved these changes
Aug 5, 2026
vatsrahul1001
added a commit
that referenced
this pull request
Aug 5, 2026
…) (#71146) * Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination. (cherry picked from commit 6145746) Co-authored-by: Christoph <116812500+cmettler@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #65738 to
v3-3-test. Places the task-runner in its own process group (os.setpgid(0, 0)at fork + parent-sidesetpgid(pid, pid)mirror) so a heartbeat-409 kill reaches the whole subprocess tree viaos.killpg, fixing orphaned subprocesses and the_cleanup_open_socketsselector crash.One conflict in
supervisor.py— thestart()docstring. v3-3-test still restrictsuse_execto_subprocess_main(theValueErrorguard is present), whereas main's version rehydrates any entry point. Resolved by keeping v3-3-test'suse_execrestriction note and adding the new:param new_process_group:documentation, and dropping main's rehydration wording (which does not apply on this branch). The setpgid code, the_signal_subprocessown-group guard, and the tests applied cleanly.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines — cherry-pick backport with one docstring conflict resolution (documented above).