Fix lingering task supervisors when EOF is missed - #51180
Merged
Conversation
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
2 tasks
amoghrajesh
approved these changes
May 29, 2025
amoghrajesh
left a comment
amoghrajesh
left a comment
Contributor
There was a problem hiding this comment.
Seems like something that could work, lets ship it
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
kaxil
commented
May 29, 2025
sanederchik
pushed a commit
to sanederchik/airflow
that referenced
this pull request
Jun 7, 2025
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
ashb
pushed a commit
that referenced
this pull request
Jun 20, 2025
closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1)
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 20, 2025
…1180) closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Contributor
Backport successfully created: v3-0-test
|
jedcunningham
pushed a commit
that referenced
this pull request
Jun 20, 2025
…1180) (#51970) closes #50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly. (cherry picked from commit a2651f1) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
jose-lehmkuhl
pushed a commit
to jose-lehmkuhl/airflow
that referenced
this pull request
Jul 11, 2025
closes apache#50500 Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF. The supervisor now records the process exit time and forcibly closes any sockets still open after `workers.socket_cleanup_timeout`. This stops the supervisor loop from hanging indefinitely and allows the process to exit cleanly.
cmettler
added a commit
to cmettler/airflow
that referenced
this pull request
Jun 3, 2026
When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR apache#51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: apache#65505
cmettler
added a commit
to cmettler/airflow
that referenced
this pull request
Jun 22, 2026
When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR apache#51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: apache#65505
cmettler
added a commit
to cmettler/airflow
that referenced
this pull request
Jul 4, 2026
When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR apache#51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: apache#65505
2 tasks
eladkal
pushed a commit
to cmettler/airflow
that referenced
this pull request
Aug 4, 2026
When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR apache#51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: apache#65505
vatsrahul1001
pushed a commit
that referenced
this pull request
Aug 5, 2026
* Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination.
vatsrahul1001
added a commit
that referenced
this pull request
Aug 5, 2026
…) (#71146) * Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination. (cherry picked from commit 6145746) Co-authored-by: Christoph <116812500+cmettler@users.noreply.github.com>
vatsrahul1001
added a commit
that referenced
this pull request
Aug 5, 2026
…) (#71146) * Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination. (cherry picked from commit 6145746) Co-authored-by: Christoph <116812500+cmettler@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #50500
Adds a new safeguard for cases where the task subprocess closes before all pipe sockets send EOF.
While the supervisor "shouldn't" get stuck, the reality is that under extreme load the selector can miss final EOF events - which might have happened previously under high CPU load + the bug in #51023. This adds a safety net to prevent resource leaks and hanging processes (Pods accumulate for CE & KE). The timeout is generous (60s) to avoid premature cleanup of legitimate slow socket operations. I tried reproducing it reliably after my first attempt in #51020 but haven't been able to reproduce it.
Changes
workers.socket_cleanup_timeoutconfig (default 60s) to force-close stuck sockets after task process exits^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.