Repository navigation
test(eval): harden relay descendant cleanup against process-group timing #3770
Description
Problem
The test job for #3722 failed in the unrelated eval lifecycle suite at packages/eval/harbor/test_relay_lifecycle.py:648:
RelayLifecycleTest.test_settle_kills_descendants_before_verification_boundary
self.assertFalse(late_write.exists())
AssertionError: late_write remained present
The test starts a setsid --wait process whose parent exits while a forked descendant writes late-write after 300 ms. The relay should quiesce the entire process group before the verifier observes the workspace, but this run observed the delayed write.
Failure evidence: https://github.com/apache/maka/actions/runs/32754131081/job/97517633860
Scope
This is pre-existing and unrelated to #3722:
- test(windows): close packaged sandbox lifecycle evidence #3722 does not modify
packages/eval/harbor/test_relay_lifecycle.pyorrelay_agent.py. - The file blobs are identical between the PR head and
upstream/main. - The same eval suite passed on a later main run: https://github.com/apache/maka/actions/runs/32765873057/job/97559885543
- test(windows): close packaged sandbox lifecycle evidence #3722's package and Windows sandbox W0 checks passed; only the generic
testjob failed.
Suspected area
packages/eval/harbor/relay_agent.py uses the scope PID file and kill -0 -- -<pgid> to decide whether a process group is active before sending TERM/KILL. The failure suggests a timing or process-group observability gap: the scope was treated as inactive before the descendant's delayed write, so no effective termination was observed. The exact pgid/signal trace is not currently logged.
Suggested follow-up
- Add raw scope PID, process-group liveness, signal return code, and post-signal verification evidence.
- Make descendant quiescence robust when the group leader exits before the descendant.
- Keep the regression test deterministic on the Linux CI runner and verify repeated runs.
This issue should be tracked separately from the Windows sandbox lifecycle work in #3722.
Activity
testikun commentedon Aug 27, 2026
I would like to work on this issue.
Proposed scope
I will keep this change limited to the Eval relay settlement boundary described here:
No Windows-specific changes, verifier policy changes, or unrelated Eval refactors are in scope.
中文摘要
我会把改动限制在 Eval relay 的 settlement 边界：保留正常退出 subject 不应被误杀的现有语义，修复父进程先退出时 relay 误判进程组已经 quiescent 的时序窗口；scope PID/PGID 证据缺失或不明确时继续 fail closed；增加 leader 先退出、正常退出、取消与 settlement 竞态、证据不可用等回归测试。macOS 只能验证 Python 逻辑和 mock，真实进程组行为最终以 Linux CI 为准，不修改 Windows 或无关 Eval 代码。