SIGABRT: panic in wait-timeout's SIGCHLD handler kills `wt` when the self-pipe send() fails

[tomascamargo]

Summary

Any wt command that waits on a child process via shell_exec::Cmd::delayed_stream aborts with SIGABRT if the SIGCHLD notification write fails. wt inherits this from wait-timeout 0.2.1, whose SIGCHLD handler panics on any errno other than WouldBlock — and because the handler is extern "C", the panic cannot unwind and goes straight to abort().

I hit this reliably by running wt switch --create inside the OpenAI Codex CLI's Linux sandbox, which denies the sendto(2) syscall. But the sandbox is only the trigger; the crash-instead-of-degrade behaviour is worktrunk's to absorb.

What happens

wt -C <repo> -y switch --create <branch> --base <base> --no-hooks --no-cd --format json
→ thread caused non-unwinding panic. aborting.
→ SIGABRT, coredump

Symbolized backtrace (Arch worktrunk 0.68.0-1, symbols via debuginfod):

wt::main
 → wt::commands::worktree::switch::handle_switch_command
 → SwitchPipeline::run
 → execute_switch
 → worktrunk::git::repository::Repository::run_command_delayed_stream
 → worktrunk::shell_exec::Cmd::delayed_stream
 → wait_timeout::ChildExt::wait_timeout          →  poll()   ← blocked on child `git`
<signal handler called>                                      ← SIGCHLD
 → wait_timeout::imp::sigchld_handler
 → core::panicking::panic_cannot_unwind
 → std::sys::pal::unix::abort_internal
 → abort()

Panic message recovered from the core dump:

bad error on write fd: Operation not permitted (os error 1)

Mechanism

wait-timeout builds a self-notification AF_UNIX socketpair and pokes it from the SIGCHLD handler with send(fd, buf, 1, MSG_NOSIGNAL). Its notify() tolerates exactly one error:

match state.write.write(&[1]) {
    Ok(..) => {}
    Err(e) if e.kind() == WouldBlock => {}
    Err(e) => panic!("bad error on write fd: {}", e),   // ← anything else aborts the process
}

sigchld_handler is extern "C", so that panic hits panic_cannot_unwind and terminates the process immediately. There is no way for wt to catch it and no useful diagnostic for the user — just SIGABRT.

Reproduction

Deterministic, in a throwaway repo:

mkdir repro && cd repro && git init -b main && git commit --allow-empty -m init
codex sandbox -- wt switch --create repro-test --base main --no-cd --no-hooks
# thread caused non-unwinding panic. aborting.

(codex sandbox runs the command under Codex's default workspace-write sandbox, which denies sendto. Confirmed independently: send() on an AF_UNIX socketpair returns EPERM there while write() on the same fd succeeds.)

I have two independent coredumps six days apart — different repositories, different branches, different terminal sessions — with byte-identical stacks, so this is not a race.

Expected

Failing to poke an internal wakeup socket should not kill the process. Ideally wt either reports a normal error or, better, doesn't have this failure mode at all.

Notes on a fix

This is a known problem in the crate: alexcrichton/wait-timeout#45 flags exactly this notify() panic as an async-signal-safety violation (panic! in a signal handler allocates and takes locks). It's open and currently classified as minor/theoretical; this report is a real-world instance of it. Options, roughly in order of preference:

1. Replace wait-timeout in src/shell_exec.rs. Since delayed_stream already polls, waiting on the child directly (e.g. pidfd on Linux — cf. Add pidfd-based implementation alexcrichton/wait-timeout#29 — or a waitpid-with-timeout loop) removes the global SIGCHLD handler entirely.
2. Vendor/patch notify() to ignore all write errors, not just WouldBlock. A missed wakeup is at worst a delayed timeout; it is never worth an abort().

src/commands/picker/pager.rs and src/commands/picker/prs.rs use wait_timeout too and presumably have the same exposure.

Environment

• worktrunk 0.68.0-1 (Arch extra); wait-timeout 0.2.1 is still in Cargo.lock on v0.74.0, and src/shell_exec.rs still uses it, so I believe current main is affected
• Arch Linux, kernel 7.1.8-arch1-3, glibc-based
• Trigger: openai/codex 0.148.0 with sandbox_mode = "workspace-write"; the sendto deny is tracked at Restricted Linux sandbox breaks connected Unix socket sends and asyncio wakeups openai/codex#33793

Filed by Claude Opus 5 via Claude Code.

[worktrunk-bot]

Thanks — the mechanism checks out exactly as you described, and the crate source confirms it: notify() treats every errno but WouldBlock as a panic!, and sigchld_handler is extern "C", so it can only reach abort(). Your preferred option (1) is the right one, so #3857 removes wait-timeout rather than working around it.

Cmd::delayed_stream, run_with_timeout_impl, and the picker's pager now wait through a small in-tree ChildWaiter: a dedicated thread owns the child and blocks in Child::wait(), and the status comes back over a channel, so the timed wait is a recv_timeout. No signal disposition is touched, so there is no handler left to panic in — and no polling loop either, so a child that exits early is still observed the moment it does. (picker/prs.rs was a false alarm: that's Condvar::wait_timeout_while from std, unrelated.)

Reproducing the EPERM itself needs a sandbox that denies the send, which we can't stand up in CI, so the regression test pins the root cause instead: after exercising both deadline paths, the SIGCHLD bit in /proc/self/status's SigCgt mask must be clear. It fails on main (SigCgt=0000000100010440 — bit 16 set) and passes on the branch. That holds regardless of which errno a given sandbox returns.

If you're able to re-run your codex sandbox repro against the branch once it lands, that would be a useful confirmation from the environment that actually triggers it. A maintainer will review the PR.

Note

Superseded on 2026-08-22: #3857 now waits through the shared_child crate rather than the in-tree waiter described above, so three details here no longer describe what ships — the SigCgt regression test (a SIGCHLD handler legitimately exists again, through signal_hook, which cannot abort on a failed wake), the TERM → KILL escalation note (that came from the waiter's prompt reaping and is gone with it), and the stale-pid window (the new timed wait is non-reaping, so the pid stays reserved). See the PR description and this comment.

Notes

• wait-timeout was the only thing pulling it in, so it's gone from Cargo.lock too.
• One incidental fix: forward_signal_with_escalation decides whether to escalate TERM → KILL by polling killpg(pgid, 0). Under wait-timeout the unreaped zombie leader kept that true through the whole 200 ms grace, so the group SIGKILL fired essentially every time — defeating the "git's lockfile handlers run on TERM" reason for sending TERM first. With prompt reaping, a child that exits on TERM now skips the SIGKILL.
• Signalling a timed-out child now goes by pid rather than through the Child handle, which opens a narrow stale-pid window (the child can exit and be reaped between the expired deadline and the syscall). It's documented at both call sites; closing it would mean re-introducing the blocking wait the type exists to avoid.

[max-sixty]

thanks, nice find @tomascamargo

@worktrunk-bot is there any other dependency we could add that would handle this? I'm loathe to have too much code I don't really understand in this crate, but seems like we may have to...

[worktrunk-bot]

Yes — shared_child, and it's a good fit. #3857 now uses it instead of the in-tree waiter: same API shape the hand-rolled type had grown (wait / wait_timeout / kill / id / take_std*), from the author of duct and blake3, ~10M downloads/month. It adds exactly two crates to Cargo.lock — shared_child and sigchld — because signal-hook, os_pipe, libc and windows-sys are already in the graph. Net −88 lines.

The honest caveat: it does install a SIGCHLD handler, which the in-tree waiter avoided entirely by owning a thread per child. Unix has no "wait for this child, but give up after N ms" syscall, so every option that isn't a polling loop ends up on SIGCHLD — process_control, the other credible crate, does the same thing. What makes it safe is whose handler: sigchld wakes through signal_hook, which discards write errors on purpose — "we ignore errors, on purpose. We don't have any means to handling them" — where wait-timeout panic!s. A denied wake there costs one early return, not the process. And signal_hook probes the wake fd and uses write() when it isn't a socket, which is what sigchld hands it — so send(), the syscall Codex's sandbox denies, isn't even on the path that broke.

Two things got better rather than just smaller: the timed wait is waitid(WNOWAIT), so the child stays unreaped across the deadline and the stale-pid window I'd documented at both kill sites is gone; and kill() goes through the Child handle instead of a raw signal by pid.

The one thing lost is the regression test. It asserted the SIGCHLD bit was clear in SigCgt, which shared_child legitimately sets — and that was never the property that mattered (wt already installs signal_hook handlers for SIGINT/SIGTERM). It's replaced with a graph assertion that wait-timeout doesn't come back, however it gets there.

If you'd rather keep the bespoke version, a1dc5d4 is the last commit with it and the conversion is a single commit on top.

Sources

• shared_child wait_deadline_noreap — waitid(WEXITED|WNOWAIT), deadline enforced by a sigchld::Waiter; timeout is a default feature gating dep:sigchld.
• sigchld::Waiter::new — os_pipe::pipe() + signal_hook::low_level::pipe::register(SIGCHLD, writer), then poll() on the read end against the deadline. Its os_pipe feature is on by default; os_pipe 1.2.3 is already in our lock via another dep.
• signal_hook::low_level::pipe::register_raw — probes with send(fd, NULL, 0, MSG_DONTWAIT); a pipe fails that probe, so the wake method is write().
• signal-hook-registry sets SA_RESTART and chains to any previous handler, so registering SIGCHLD composes with our SIGINT/SIGTERM handlers rather than displacing them — both signal-hook 0.3 (what sigchld needs, already present via crossterm/termwiz) and our 0.4 share the one signal-hook-registry 1.4.8.
• process_control 5.2.0 — same mechanism (signal_hook::iterator::Signals on SIGCHLD + waitid(WNOWAIT)), so equally safe; heavier API and ~40× fewer downloads, so it lost on fit rather than correctness.