test(tui): name the command a dispatch smoke stalls on, and skip /pin on Windows - #5922
Conversation
… on Windows The two dispatch-everything smoke tests burned nextest's full 600 s timeout on windows-latest with no attribution (#5919). The stderr breadcrumbs from run 34013201624 / job 101432543520 name the offender: the registered-command test stopped at `/pin` and the alias test at `/mini` — the same handler, `commands::groups::core::pin`, neither of which printed a "returned in" line. `/pin` is not a state toggle. It calls `tui::window_control::toggle_pin()`, which resolves the *host terminal window* and drives it with Win32 `SetWindowPos`/`ShowWindow`. That HWND belongs to another process, so those calls are delivered to that window's thread and block until it pumps them. On a headless CI window station there is nothing to pump, and the call never returns. Skip `/pin` in the smoke tests on Windows only, with the reason in the doc comment; macOS and Linux compile `toggle_pin()` to a no-op, so dispatch coverage for the command is kept there. Fixing only the known offender leaves the next one just as opaque, so each dispatch now runs under a per-command watchdog: the app is built and the command executed on its own thread and the test waits 30 s for the result. A blocking handler fails the test in milliseconds naming the invocation instead of costing a ten-minute CI slot. Handler panics still surface as that panic (the payload is resumed), which is the property the old inline loop had and the reason the smoke tests exist. The unconditional per-command eprintln pair is gone — 15k log lines whose only job the watchdog now does — replaced by a single line for any dispatch slower than a second. Verified locally on macOS: cargo fmt --all -- --check clean cargo test -p codewhale-tui --lib -- commands::tests::every_ test result: ok. 3 passed; 0 failed; 0 ignored; 11704 filtered out cargo test -p codewhale-tui --lib -- commands:: test result: ok. 977 passed; 0 failed; 0 ignored; 10730 filtered out The watchdog itself was proved by temporarily setting DISPATCH_WATCHDOG to Duration::ZERO (failed in 0.05 s with "/anchor did not return within 0ns") and by temporarily panicking inside the worker (the panic propagated to the test with the handler's name on the thread). Both temporaries were reverted before the runs quoted above. The Windows hang itself is only provable on windows-latest CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0188XYyJaw9Mh9uSrqQBoqhm Signed-off-by: CodeWhale Bot <bot@codewhale.net>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_351bd910-d7af-4f1e-8d87-61b1b4911ad4) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @Hmbown's task in 4m 2s —— View job Review: PR #5922
Scope confirmed: diff is entirely inside Finding (posted inline at line 1634)The watchdog's own
Suggested direction: build the This doesn't affect the specific bug this PR fixes ( Good choices
No other correctness, provider/model-safety, or security issues found in this test-only diff. |
There was a problem hiding this comment.
🟡 Changes recommended
The watchdog timeout path can leave a blocked worker thread holding the global test env mutex and the success path currently risks swallowing worker-thread panics by ignoring join() errors.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the TUI command dispatch smoke tests to fail fast (and attribute the offending command) when a handler blocks on Windows CI, addressing the /pin-related hang described in #5919. All changes are confined to #[cfg(test)] code in the TUI commands module.
Changes:
- Add a per-command watchdog that runs each dispatch in a dedicated thread and times out after 30s, surfacing panics from handlers.
- Skip
/pinon Windows in the dispatch smoke loops, with an in-code rationale tied to the Win32 window-control behavior. - Reduce log noise by removing unconditional per-command breadcrumbs and only logging slow dispatches.
File summaries
| File | Description |
|---|---|
| crates/tui/src/commands/mod.rs | Refactors dispatch smoke tests to use a per-command watchdog thread, skips /pin on Windows, and reduces per-command logging. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Err(std::sync::mpsc::RecvTimeoutError::Timeout) => panic!( | ||
| "{label} did not return within {DISPATCH_WATCHDOG:?}: its handler blocks. \ | ||
| Fix the handler or add it to skip_in_dispatch_smoke with a reason." | ||
| ), |
| let started = std::time::Instant::now(); | ||
| match rx.recv_timeout(DISPATCH_WATCHDOG) { | ||
| Ok(message) => { | ||
| let _ = handle.join(); |
| // 16 MiB the CI runner sets via RUST_MIN_STACK for the main thread. | ||
| .stack_size(16 * 1024 * 1024) | ||
| .spawn(move || { | ||
| let (mut app, tmpdir, _guard) = create_isolated_test_app(); |
There was a problem hiding this comment.
create_isolated_test_app() → ConfigPathGuard::new() acquires lock_test_env(), a process-wide Mutex<()> (crates/tui/src/test_env_lock.rs:159) shared by dozens of other tests (runtime_api/tests.rs, model_inventory.rs, config_persistence.rs, …).
That guard now lives inside the spawned worker thread's closure. On the RecvTimeoutError::Timeout path, dispatch_under_watchdog panic!s on the main test thread and returns — but the worker thread that's actually stuck in the blocking handler is never joined or killed, so its ConfigPathGuard is never dropped and the global env mutex is never released.
Under cargo nextest run this is contained (nextest runs each test in its own process, so a leaked thread in a doomed process can't affect anything else). But under plain cargo test — which this repo also uses, including the verification commands quoted in this PR's own description (cargo test -p codewhale-tui --lib -- commands::tests::every_) — all tests in this binary share one process and its threads. If a future handler hangs (the exact scenario this watchdog exists to catch), the offending test now fails fast and by name as intended, but it leaves the shared mutex permanently locked for the rest of that cargo test invocation, so every other test that later calls lock_test_env() (and there are ~30+ call sites) hangs too — trading one attributable timeout for an unattributed pile of them.
Suggest acquiring the env-lock guard (and building App/tempdir) on the main thread before spawning, and only moving the already-built App into the worker for the execute() call — that way the guard drops promptly regardless of whether the worker ever returns. Worth double-checking App: Send if going this route.
There was a problem hiding this comment.
Codewhale review
PR adds a per-command 30s watchdog for the TUI dispatch smoke tests and skips /pin on Windows to avoid the CI hang from Win32 window calls. The change is test-only in commands/mod.rs.
Findings
- [WARNING] Watchdog timeout leaks the worker thread and its isolation guard (
crates/tui/src/commands/mod.rs)
When a handler blocks longer than DISPATCH_WATCHDOG, the test panics but the worker thread remains parked inside execute(). Rust cannot kill threads, so the TempDir and the _guard returned by create_isolated_test_app() are never dropped. If that guard serializes process-wide state (e.g. current directory or environment), later tests may hang or run in the wrong directory, turning a single timeout into a cascade. Consider a design that can clean up on timeout, such as running each command in a separate process or otherwise avoiding leaked guards. - [INFO] No automated regression test for the watchdog itself (
crates/tui/src/commands/mod.rs)
The watchdog timeout and panic-resume paths are not covered by a committed test; the PR only verified them temporarily with local edits. A future refactor could break the watchdog without any test failure. Consider adding a small test-only command that blocks or panics and asserting that the smoke test fails with the named invocation. - [INFO] Worker panic after sending a result is swallowed (
crates/tui/src/commands/mod.rs:1644)
In the Ok(message) arm,let _ = handle.join();discards the join result. If the worker thread panics after sending the result (for example while dropping _guard or tmpdir), the test does not fail even though the stated goal is to surface all handler panics. Consider resuming a join error here as well.
Assessment
The change is reasonable and directly addresses the Windows CI hang, but the leaked worker thread/guard on watchdog timeout is a latent hazard for subsequent tests, and the watchdog lacks committed regression coverage.
Advisory review by Codewhale (codewhale review --pr 5922 --post, head e09fc7db1790958bb57b39d7c1c38637eadc0bc8). Line-specific findings are also posted as inline review comments; mechanical fixes arrive as committable suggestions you can apply from the Files tab. CODEOWNERS approval still governs merge.
| let started = std::time::Instant::now(); | ||
| match rx.recv_timeout(DISPATCH_WATCHDOG) { | ||
| Ok(message) => { | ||
| let _ = handle.join(); |
There was a problem hiding this comment.
[INFO] Worker panic after sending a result is swallowed
In the Ok(message) arm, let _ = handle.join(); discards the join result. If the worker thread panics after sending the result (for example while dropping _guard or tmpdir), the test does not fail even though the stated goal is to surface all handler panics. Consider resuming a join error here as well.
Closes #5919
The offender:
/pinThe stderr breadcrumbs in run 34013201624 / job 101432543520 name it. In
every_registered_command_dispatches_to_a_handlerthe last line is:with no matching
/pin returned in …. Inevery_command_alias_dispatches_to_a_handlerthe last line is
dispatch smoke: /mini— andminiis the first alias ofpin(crates/tui/src/commands/groups/core/pin.rs). Two independent tests,same handler, neither returned.
/pinis not a state toggle. It callstui::window_control::toggle_pin(),which resolves the host terminal window (
GetConsoleWindow, else theforeground window, else an
EnumWindowswalk of the parent process chain)and then drives it with Win32
SetWindowPos/ShowWindow. ThatHWNDbelongs to a different process: those calls are delivered to that window's
thread and block until it pumps them. On a headless CI window station there
is nothing pumping, so the call never returns and the test runs out the
600 s nextest timeout.
What this PR does
executeon its own thread; the test waits 30 s for the result. Ablocking handler now fails the test in milliseconds, naming the
invocation, instead of costing a ten-minute CI slot with no attribution.
Handler panics still surface as that panic (the payload is resumed), which
is a property the smoke tests exist for.
/pinon Windows only, with the reason in the doc comment nextto the existing
/restoreskip. On macOS and Linuxtoggle_pin()is acompiled-out no-op, so dispatch coverage for
/pinis kept there.eprintlnpair (≈15k log lines perrun) whose only job the watchdog now does; a single line remains for any
dispatch slower than a second.
No production code changed — the diff is entirely inside
#[cfg(test)] mod tests.Verified locally (macOS, aarch64)
cargo check -p codewhale-tuidoes not compilecfg(test)code, so thecargo testruns above are the compile evidence for this diff.The watchdog mechanism itself was proved, not assumed:
DISPATCH_WATCHDOGtemporarily set toDuration::ZERO→ the test failedin 0.05 s with
/anchor did not return within 0ns: its handler blocks.test, with the handler's name on the thread (
thread 'dispatch-smoke-anchor').Both temporary edits were reverted before the runs quoted above.
What only Windows CI can prove
I am on macOS, so I could not reproduce the hang. The
/pinattribution isread off the CI breadcrumbs plus the Win32 code, not from a Windows run;
toggle_pin()compiles tofalseon this host, so the skip is a no-oplocally and the two tests would pass here either way. The acceptance
criterion "Windows matrix green on three consecutive runs" can only be
checked on windows-latest.
Also deliberately not attempted: hardening
window_control::toggle_pin()itself (e.g.
SWP_ASYNCWINDOWPOS/ShowWindowAsyncso the TUI thread neverblocks on the terminal host's message pump). That is a real improvement for
Windows users, but it changes the semantics of the existing
apply-then-verify-then-retry logic and I cannot run it on this host. It
belongs in its own issue with a Windows author.
🤖 Generated with Claude Code
https://claude.ai/code/session_0188XYyJaw9Mh9uSrqQBoqhm
Note
Low Risk
Test-only changes in the TUI command module; no runtime or dispatch logic in production builds.
Overview
Fixes Windows CI hangs (#5919) where the “every command dispatches” smoke tests could stall for the full nextest timeout when
/pin(and alias/mini) blocked on Win32 window calls in headless CI./pinis excluded from the smoke loop on Windows only, with the same style of rationale as the existing/restoreskip; macOS/Linux still exercise/pinbecausetoggle_pin()is a no-op there.The smoke tests now run each dispatch on a dedicated thread with a 30s watchdog, so a blocking handler fails fast and names the invocation instead of burning CI time. Handler panics still propagate via
resume_unwind. Verbose per-commandeprintlnbreadcrumbs were removed; only dispatches slower than one second log a line.All changes are under
#[cfg(test)]incommands/mod.rs— no production behavior changes.Reviewed by Cursor Bugbot for commit e09fc7d. Bugbot is set up for automated code reviews on this repo. Configure here.