Skip to content

Client timeout recovery ends sibling sessions: host-wide runner sweep on every timeout, daemon reset on shared daemons #3177

Description

@thymikee

Problem

A client-side request timeout recovers by acting on the whole host, not on the request that timed out. When one daemon serves several sessions, one slow request ends the others.

  • The runner sweep runs on every local timeout. handleRequestTimeout calls cleanupTimedOutIosRunnerBuilds before it checks the command's timeout policy. The sweep runs pkill -f with patterns that match every agent-device runner xcodebuild on the host: other sessions, other daemons, other workspaces. A timed-out snapshot, whose policy is preserve-daemon, still kills every runner.
  • Reset-policy commands kill the shared daemon. open, back, scroll, settings and the other commands on DEFAULT_TIMEOUT_POLICY SIGKILL the daemon (resetDaemonAfterTimeout), which ends every session it owns.

Consumer impact: tester-army/e2e runs one session per worker slot on one local daemon. A cold iOS runner outlasted the 90 s open envelope, and the reset ended the sibling worker's session (tester-army/e2e#627). The engine now starts each runner with prepare ios-runner and warms devices one at a time only to stay clear of this path.

Required behavior (needs design)

  • The sweep targets only the runner of the device or session the timed-out request used, never every runner on the host.
  • When the daemon has other live sessions, a timed-out request recovers its own session (cancel the request, retire that session's runner) and leaves the daemon and the other sessions running.
  • A full daemon reset stays available for a daemon that does not answer at all.

Open question: the client knows only process-name patterns. Scoping the recovery probably means the daemon owns it, through a cancel or retire call, or through its own request deadline. Decide that before implementation.

Done when

  • Two sessions on one daemon: a timed-out open in session A leaves session B able to run snapshot and press.
  • A timed-out snapshot kills no process outside its own session's runner.

Related: #3105 (timeout reset removes metadata without proving ownership), #3116 (daemon retirement ownership). This issue is about the scope of recovery, not how a reset retires the daemon.

From the tester-army/e2e integration review.

Activity

  1. thymikee commented on Oct 4, 2026

    @thymikee
    MemberAuthor

    #3116 and #3127 are landed in main (2212eacf5b). They fix retirement identity, lock exclusion and metadata/report cleanup authority; they do not fix the recovery scope described here. This remains the explicit timeout-cleanup follow-up.

    The per-call AbortSignal transport from #3200 is also landed. It closes the caller's connection and preserves typed cancellation, including abort during restart-health probing, without timeout retirement. Reuse that request-scoped boundary when designing recovery; daemon/process identity alone does not authorize disrupting other sessions.

    The #3116 audit independently confirmed the host-wide pkill/shared-daemon-reset behavior in code. Its audit did not reproduce interruption of a concurrent recording export; retain the existing tester-army evidence with its original attribution. Add a pending-recording-export case alongside the existing two-session completion criteria: another request's timeout must leave the export and unrelated runner alive. Recording finish/retry must continue using its owned handle and durable completion, and full daemon recovery must remain explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageNew or unreviewed issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions