Fix orphaned subprocesses and supervisor crash on heartbeat 409 by cmettler · Pull Request #65738 · apache/airflow

cmettler · 2026-04-23T17:35:52Z

When a running TaskInstance is forcibly transitioned out of running (scheduler reset, REST PATCH, etc.), the next heartbeat from the still-running task-runner returns HTTP 409 and the supervisor kills the task. On Linux this produced two bugs:

Orphaned subprocesses — any Popen child the task-runner had spawned (@task.virtualenv, DockerOperator, BashOperator, Cosmos dbt, etc.) was reparented to PID 1 and kept running until it finished on its own, wasting CPU/RAM/API quota.
Supervisor crash — ~60s later _cleanup_open_sockets() closed the selector while _service_subprocess() was still polling it, raising ValueError: I/O operation on closed epoll object (regression from Fix lingering task supervisors when EOF is missed #51180).

Fix

Place the task-runner in its own process group with os.setpgid(0, 0) immediately after fork (task execution opts in; the DAG processor and triggerer keep the supervisor's group), then have kill() signal the whole group via os.killpg(os.getpgid(pid), sig). This reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly and close their inherited pipes, so the supervisor drains _open_sockets normally and never enters the cleanup-the-selector-mid-loop path.

Two safeguards make the group-signalling robust:

Parent-side setpgid(pid, pid) mirror. The parent repeats the child's setpgid right after fork so the group is guaranteed to exist regardless of ordering. Without it, a signal arriving before the child's own setpgid ran (e.g. _on_child_started failing synchronously) could resolve the child's PGID to the supervisor's own group and killpg it.
Own-group guard in _signal_subprocess(). If both setpgid calls failed and the child still shares the supervisor's process group, the code signals the pid directly instead of killpg-ing its own group, so the supervisor never signals itself.

killpg/getpgid fall back to self._process.send_signal(sig) on ProcessLookupError or PermissionError, preserving behaviour when the group has vanished or permissions are lacking.

Tests

test_task_runner_starts_in_new_process_group — real-fork regression: asserts the child's PGID == its own PID after ActivitySubprocess.start() for task execution.
test_child_keeps_supervisor_process_group_by_default — DAG processor / triggerer path stays in the supervisor's group.
test_kill_signals_process_group — primary path uses killpg.
test_kill_does_not_signal_supervisors_own_process_group — the own-group guard: signals the pid, not the supervisor's group.
test_kill_signals_pid_only_without_new_process_group — pid-only signalling when no separate group was created.
test_kill_falls_back_to_send_signal_when_group_signal_fails (4 params: {ProcessLookupError, PermissionError} × {getpgid, killpg}).
test_kill_process_already_exited / test_kill_process_custom_signal — existing kill tests, updated to mock os.getpgid / os.killpg.

closes: #65505

Description refreshed to match the current setpgid-mirror revision (per review) — the implementation moved from the earlier setsid()/session-leader design.

Was generative AI tooling used to co-author this PR?

Yes (Claude Opus 4.7 (1M context))

Generated-by: Claude Opus 4.7 (1M context) following the guidelines

boring-cyborg · 2026-04-23T17:35:58Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

cmettler · 2026-06-22T17:51:33Z

CI failures were unrelated — caused by azure-storage-blob 12.30.0 breaking the WASB SAS-token tests (issue #68482), fixed upstream in #68490. After rebasing onto current main, that fix is now in our sources and CI should pass. PR is ready for review when you have time.

Drafted-by: Claude Code (Opus 4.7); reviewed by @cmettler before posting

seanmuth · 2026-06-30T20:10:08Z

Following up on @ashb’s internal note about the session-leader TODO:

Confirmed this PR does make the task-runner a session leader — os.setsid() is added right after the fork in start(), and kill() now signals the whole group via os.killpg(os.getpgid(self._process.pid), sig) with a send_signal() fallback (plus a test asserting the child’s PGID == its own PID post-fork).

That makes the pre-existing # TODO: Make this process a session leader in _fork_main obsolete, but the PR doesn’t currently touch it:

airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py

Line 420 in 0bfc7ea

# TODO: Make this process a session leader

Could you drop that TODO comment as part of this change? Thanks!

…nst self-signalling Review feedback on apache#65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination.

When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR apache#51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: apache#65505

…nst self-signalling Review feedback on apache#65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination.

vatsrahul1001 · 2026-08-05T05:07:44Z

Refreshed the PR description to match the current revision (the setpgid(pid, pid) parent-side mirror + own-group guard, replacing the earlier setsid()/session-leader wording) and updated the test list — test_child_is_session_leader is now test_task_runner_starts_in_new_process_group. Did this on the author's behalf to unblock for the 3.3.1 patch; code is unchanged. Should be good to merge + backport now.

boring-cyborg · 2026-08-05T05:08:32Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

github-actions · 2026-08-05T05:09:58Z

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 6145746 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…) (#71146) * Fix orphaned subprocesses and supervisor crash on heartbeat 409 When a running TaskInstance is forcibly transitioned out of `running` (e.g. the scheduler resets a stale heartbeat, or an operator PATCHes the state to `failed`), the task-runner's next heartbeat returns HTTP 409 and the supervisor kills the task. Before this change two things went wrong on Linux: 1. Subprocesses the task-runner had spawned (`@task.virtualenv` / `PythonVirtualenvOperator` children, `DockerOperator` exec, Bash shells) were reparented to PID 1 and kept running as orphans until they finished on their own - wasting CPU, RAM and third-party API quota. 2. About 60s later, `_cleanup_open_sockets()` closed the selector while `_service_subprocess()` was still using it, so the supervisor crashed with `ValueError: I/O operation on closed epoll object` (regression from PR #51180). The task-runner is now placed in its own session via `os.setsid()` immediately after fork, so its process group ID equals its PID. The supervisor's `kill()` signals the whole group via `os.killpg(os.getpgid(pid), sig)`, which reaches every subprocess the task-runner spawned. Grandchildren without a SIGTERM handler exit promptly, close their inherited pipes, and the supervisor drains `_open_sockets` normally - so `_cleanup_open_sockets()` is never triggered and the selector is never closed mid-loop. `os.killpg`/`os.getpgid` fall back to `self._process.send_signal(sig)` on `ProcessLookupError` or `PermissionError`, preserving prior behaviour when the group has vanished (e.g. the task was already reaped) or permissions are lacking. closes: #65505 * Scope process-group handling to the task runner and guard kill() against self-signalling Review feedback on #65738: os.killpg(os.getpgid(child)) trusted that the child had already run setsid() -- if setpgid failed or kill() ran before the child was first scheduled (task_instances.start() raising synchronously), getpgid resolved to the supervisor's own group and killpg would have signalled the supervisor and all its siblings, with no exception for the fallback to catch. Use a plain process group (setpgid, matching airflow.utils.process_utils.set_new_process_group) instead of a new session, set it from both sides of the fork so the group exists as soon as start() returns, refuse to killpg our own group, and make the whole behaviour opt-in per subclass (like use_exec) so the DAG processor, triggerer and callback subprocesses keep direct signalling. The graceful SIGTERM-forwarding path now also signals the group, closing the same orphan leak on e.g. K8s pod termination. (cherry picked from commit 6145746) Co-authored-by: Christoph <116812500+cmettler@users.noreply.github.com>

cmettler requested review from amoghrajesh, ashb and kaxil as code owners April 23, 2026 17:35

boring-cyborg Bot added the area:task-sdk label Apr 23, 2026

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 27, 2026

cmettler force-pushed the fix_65505 branch from af93c4a to e3061c9 Compare June 3, 2026 09:51

potiuk marked this pull request as draft June 22, 2026 06:34

potiuk assigned cmettler Jun 22, 2026

cmettler force-pushed the fix_65505 branch from e3061c9 to ea63821 Compare June 22, 2026 17:48

cmettler marked this pull request as ready for review June 24, 2026 05:52

kaxil reviewed Jun 30, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

kaxil reviewed Jun 30, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/execution_time/supervisor.py Outdated

cmettler force-pushed the fix_65505 branch from ea63821 to b944be6 Compare July 4, 2026 18:48

eladkal added this to the Airflow 3.3.1 milestone Jul 4, 2026

eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-3-test Backport to v3-3-test labels Jul 4, 2026

vatsrahul1001 modified the milestones: Airflow 3.3.1, Airflow 3.3.2 Jul 28, 2026

This was referenced Aug 3, 2026

Make Task SDK supervisor subprocess a session leaderFix supervisor session leader #70950

Closed

Make Task SDK supervisor subprocess a session leader #70949

Closed

cmettler added 2 commits August 4, 2026 07:53

eladkal force-pushed the fix_65505 branch from b944be6 to 3da9b1d Compare August 4, 2026 04:53

vatsrahul1001 modified the milestones: Airflow 3.3.2, Airflow 3.3.1 Aug 4, 2026

kaxil approved these changes Aug 4, 2026

View reviewed changes

vatsrahul1001 merged commit 6145746 into apache:main Aug 5, 2026
107 checks passed

vatsrahul1001 mentioned this pull request Aug 5, 2026

[v3-3-test] Fix orphaned subprocesses and supervisor crash on heartbeat 409 (#65738) #71146

Merged

1 task

Navigation Menu

Uh oh!

Fix orphaned subprocesses and supervisor crash on heartbeat 409 - #65738

Fix orphaned subprocesses and supervisor crash on heartbeat 409#65738
vatsrahul1001 merged 2 commits into
apache:mainfrom
cmettler:fix_65505

cmettler commented Apr 23, 2026 •

edited by vatsrahul1001

Loading

Uh oh!

boring-cyborg Bot commented Apr 23, 2026

Uh oh!

cmettler commented Jun 22, 2026

Uh oh!

Uh oh!

Uh oh!

seanmuth commented Jun 30, 2026 •

edited

Loading

Uh oh!

vatsrahul1001 commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

boring-cyborg Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

cmettler commented Apr 23, 2026 • edited by vatsrahul1001 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix

Tests

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg Bot commented Apr 23, 2026

Uh oh!

cmettler commented Jun 22, 2026

Uh oh!

Uh oh!

Uh oh!

seanmuth commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vatsrahul1001 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

boring-cyborg Bot commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Backport failed to create: v3-3-test. View the failure log Run details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cmettler commented Apr 23, 2026 •

edited by vatsrahul1001

Loading

seanmuth commented Jun 30, 2026 •

edited

Loading

vatsrahul1001 commented Aug 5, 2026 •

edited

Loading