[JoeZhang-0x000]

Closes #744

Summary

When an SSH, Slurm, or OpenResearch host drops off the network, the supervisor retried forever: the run stayed running, and a cancel stayed cancelling because it could never be delivered.

• Cancel now ends the run. If a requested cancel stays undelivered while the host is unreachable for 30s (UNDELIVERED_CANCEL_GRACE), the run is marked cancelled. Its result notes that the remote job may still be running and where its run dir is. Existing result notes are kept, and the new note is appended.
• The real cancel is still delivered later. After the local cancel, the supervisor keeps retrying the remote cancel every 30s for 30 minutes, in case the host comes back. For OpenResearch boxes (non-lease), the box is deleted as before, which stops the job.
• Calls to the host have a time limit. Each supervisor round-trip (inspect, cancel, exit-code read, container status) gives up after 30s. Without this, a dead multiplexed connection could stall the loop for ssh's ~90s keepalive window.
• The dashboard shows "Unreachable". A live run whose backend.monitoringError is set now displays as Unreachable instead of Running (badge and lineage view). The Stop button is unchanged. The new label is translated in all 9 catalogs.

A host that is reachable but refuses the cancel is still retried, never assumed stopped. An unreachable run is never ended unless someone asked to cancel it, so a long job survives a network blip.

Changed files: src/commands/supervise.rs, ui/src/api.ts, ui/src/components/{StatusBadge,ExperimentLineage}.tsx, ui/src/components/experimentLineageModel.ts, ui/messages/*.json, regenerated ui/dist.

Test plan

• cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, cargo test --locked (1179 unit tests plus integration tests, all passing)
• New tests: an_undelivered_cancel_gives_up_only_while_the_host_stays_unreachable, an_unreachable_cancel_ends_the_run_and_keeps_its_notes
• UI: i18n lint, style lint, pnpm build (includes typecheck), 394 UI unit tests; ui/dist regenerated
• End to end in a dev slot: two SSH runs pointed at a blackholed 10.255.255.1. Cancel through the API ended one as cancelled in 59s. The other showed Unreachable in the dashboard, and pressing Stop ended it in 75s.
• Real stuck runs: 7 production runs (6 on an offline host, 1 whose supervisor had been failing for 8 days) were stuck in cancelling. Restarting their supervisors with this build cancelled all 7 within ~70s.

Notes for reviewers

• HF, Kubernetes, Modal, and Ray talk to an API rather than a host and still retry indefinitely if that API stays unreachable. They are out of scope here.
• Separate issue seen while testing, not fixed here: a supervisor that runs for days keeps its SSH config snapshot path (/tmp/orx-ssh-<uid>-<hash>/<digest>.config) cached in memory. If macOS cleans that file out of /tmp, every ssh call then fails with Can't open user config file …, even though the host is reachable.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 3/5

[Medium impact] The changes affect remote-run cancellation and status display for SSH and Slurm workflows.

Fix host reachability checks and restart recovery for pending remote cancels before merging.

Findings

1.  Reachable jobs appear cancelled ▶
2.  Cancel retries disappear after restart ▶

Fix with agent prompt

### Issue 1
src/commands/supervise.rs:858-860
`observed.is_ok()` does not tell whether the SSH host is reachable. If Docker is down while its containers keep running, both inspection and cancellation return errors even though SSH works. After 30 seconds, this branch marks the run `cancelled` and stops watching it, removing the Stop action while the job still runs.

Distinguish connection failures from remote command failures before starting this clock. A reachable host that cannot complete cancellation should keep being retried.

### Issue 2
src/commands/supervise.rs:97-100
`deliver_late_cancel` loses all remaining retries if the supervisor exits after the local cancellation. The run is already `cancelled`, so `list_active_runs` excludes it, a restarted supervisor immediately exits, and the cancel API returns `alreadyTerminal`. A host that returns within the promised 30-minute window receives no cancel after a supervisor crash or laptop restart, leaving the remote job running.

Save the pending remote cancel and its deadline, and let recovery resume it independently of the local run status.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds local cancellation for unreachable SSH-backed and Slurm runs, bounded remote calls, and a 30-minute window for retrying the remote cancel.

• A requested cancel ends locally when its remote host stays unreachable.
• Live runs with monitoring errors show as Unreachable.

Acknowledged scope:

• JoeZhang-0x000 explicitly describes remote work surviving the retry window as intentional; the result note warns that manual cancellation may be needed.
• JoeZhang-0x000 explicitly defers cached SSH configuration expiry as a separate issue.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Cancel requested] --> B[Try remote cancel and inspect]
    B --> C{Inspection succeeds?}
    C -->|Yes| D[Reset offline clock and keep watching]
    C -->|No, for 30 seconds| E[Save cancelled status and warning]
    E --> F{Legacy managed box?}
    F -->|Yes| G[Delete box]
    F -->|No| H[Retry remote cancel for 30 minutes]
    H --> I{Supervisor exits?}
    I -->|Yes| J[Retries lost; terminal run skipped by recovery]
    I -->|No| K[Stop after acknowledgement or deadline]

Loading

_{Reviews (1) · Last reviewed commit: "fix: let runs on unreachable remote host..." · Reviewed by Greptile}

[greptile-apps]

Reachable jobs appear cancelled

observed.is_ok() does not tell whether the SSH host is reachable. If Docker is down while its containers keep running, both inspection and cancellation return errors even though SSH works. After 30 seconds, this branch marks the run cancelled and stops watching it, removing the Stop action while the job still runs.

Distinguish connection failures from remote command failures before starting this clock. A reachable host that cannot complete cancellation should keep being retried.

Knowledge Base Used: Remote jobs, hosts, and supervision

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/commands/supervise.rs
Line: 858-860

Comment:
**Reachable jobs appear cancelled**

`observed.is_ok()` does not tell whether the SSH host is reachable. If Docker is down while its containers keep running, both inspection and cancellation return errors even though SSH works. After 30 seconds, this branch marks the run `cancelled` and stops watching it, removing the Stop action while the job still runs.

Distinguish connection failures from remote command failures before starting this clock. A reachable host that cannot complete cancellation should keep being retried.

**Knowledge Base Used:** [Remote jobs, hosts, and supervision](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/remote-job-supervision.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Cancel retries disappear after restart

deliver_late_cancel loses all remaining retries if the supervisor exits after the local cancellation. The run is already cancelled, so list_active_runs excludes it, a restarted supervisor immediately exits, and the cancel API returns alreadyTerminal. A host that returns within the promised 30-minute window receives no cancel after a supervisor crash or laptop restart, leaving the remote job running.

Save the pending remote cancel and its deadline, and let recovery resume it independently of the local run status.

Knowledge Base Used:

• Remote jobs, hosts, and supervision
• Experiment rental lifecycle

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/commands/supervise.rs
Line: 97-100

Comment:
**Cancel retries disappear after restart**

`deliver_late_cancel` loses all remaining retries if the supervisor exits after the local cancellation. The run is already `cancelled`, so `list_active_runs` excludes it, a restarted supervisor immediately exits, and the cancel API returns `alreadyTerminal`. A host that returns within the promised 30-minute window receives no cancel after a supervisor crash or laptop restart, leaving the remote job running.

Save the pending remote cancel and its deadline, and let recovery resume it independently of the local run status.

**Knowledge Base Used:**
- [Remote jobs, hosts, and supervision](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/remote-job-supervision.md)
- [Experiment rental lifecycle](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/experiment-rental-lifecycle.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[paperclover]

thanks for this, and for the clear write-up in #744. the orx supervise loop this patch changes goes away in #862, so closing rather than asking you to rebase onto code that's being deleted.

in #862 a run on an unreachable host shows as unreachable instead of running, a cancel waits and lands once the host is back, and Abandon ends a run whose host is gone for good. runs never end on a timer, since an outage isn't proof the job died. #862 closes #744.