feat(ssh): add optional on-demand monitoring

SKILL.md

@@ -82,6 +82,7 @@ group below has a module (`orx skill <name>`) with the full flags and rules.
 ### Run evidence (run-scoped) — module `orx-logs`
 | Command | What it does |
 |---|---|
+| `orx refresh <runId> [--collect-logs] [--json]` | Explicitly refresh a direct SSH run; on-demand runs otherwise retain cached state. |
 | `orx logs <runId>` | Show the local log path, size, and short preview; inspect the file for full evidence. |
 
 ### Create and run experiments (write) — modules `orx-create`, `orx-compute`, `orx-git`

agent-skills/orx-compute/SKILL.md

@@ -61,6 +61,11 @@ change the configured backend or authorize bypassing the launch contract.
 - `--force` permits a deliberate concurrent run on the same experiment.
   Without it, `orx` rejects a launch while that node already has a run in
   flight.
+- Direct SSH runs default to automatic monitoring (status every 5 seconds,
+  logs every 2 seconds). `--monitor-interval <seconds>` overrides both waits.
+  `--monitor-mode on-demand` disables periodic observation; schedule explicit
+  `orx refresh <runId> --json` calls using the agent's scheduler or wait tools.
+  Add `--collect-logs` only when logs are needed. Read the SSH guide below.
 - Every run gets `ORX_RUN_ID`, `ORX_EXPERIMENT`, `ORX_EXPERIMENT_TITLE`,
   `ORX_PARENT_EXPERIMENT` (when it has a parent), `ORX_COMMIT` and
   `TRACKIO_PROJECT`. Saved environment variables reach every backend except

agent-skills/orx-compute/references/ssh.md

@@ -25,18 +25,69 @@ reuse it. For command-line proxy overrides, use a separate configured alias or
 `-S none`; native SSH does not expose those overrides to the sharing predicate.
 Experiment monitors share one persistent SSH channel per resolved host and authentication
 context. Windows Connect keeps the authenticated SSH process alive after its dialog closes.
+OpenSSH supplies the protocol keepalives; there is no separate periodic helper
+heartbeat. Helper liveness is checked when monitoring requests run.
 Background monitoring never starts a fresh login on user hosts. Transport failures
 back off together (5 seconds up to 5 minutes); restarting OpenResearch or adding
 runs does not reset the retry schedule. Remote command errors do not reset the connection.
-Monitoring outages are reported after 60 seconds without declaring the remote
-job stopped. Reconnecting resumes status and log polling.
+Monitoring outages are reported after failures persist for at least 60 seconds,
+at a monitoring check, without declaring the remote job stopped.
+Reconnecting resumes status and log polling.
 
 ```sh
 orx exp run <expId> --backend ssh --host lab
 orx exp run <expId> --backend ssh --host lab --container research
 orx exp run <expId> --backend ssh --host lab --no-container
+orx exp run <expId> --backend ssh --host lab --no-container --monitor-interval 60
 ```
 
+`--monitor-mode auto` is the default. Status checks wait 5 seconds and log
+passes wait 2 seconds after returning. `--monitor-interval <seconds>` sets both
+waits to a positive whole number of seconds, for example 60 or 600. These settings
+are saved with each run; application restarts do not change existing policies.
+Both monitoring options apply only to direct SSH runs. For a host with a saved
+container, add `--no-container`; container runs retain their existing automatic monitoring.
+Use a matching updated CLI and application before enabling the new options;
+older running applications and supervisors are not automatically upgraded.
+With an explicit interval, each automatic log pass collects up to 1 MiB before
+waiting. Legacy defaults still catch up immediately; final draining remains time-bounded. Protocol keepalives
+still send lightweight traffic without collecting logs.
+
+For a direct host run that should only be checked when needed:
+
+```sh
+orx exp run <expId> --backend ssh --host lab --no-container --monitor-mode on-demand
+orx refresh <runId> --json
+orx refresh <runId> --collect-logs --json
+```
+
+Keep the run ID returned by submission. Use the agent's own scheduler or wait
+facility to call `refresh` at the user's requested cadence. This mode starts no
+periodic status/log supervisor, including after restart. It currently supports
+only direct SSH runs; containers, Slurm and managed rentals keep automatic
+monitoring. A changed SSH monitoring default in Settings applies to future direct
+runs only; command-line options override it.
+
+`refresh` performs one status observation. `--collect-logs` also collects up to
+1 MiB of new bytes within a 20-second log budget. Large logs need repeated explicit
+refreshes; `run.backend.monitoring.logsComplete` distinguishes a finished job
+from complete log collection. Remote cleanup is queued only after final log
+collection. Failed refreshes exit nonzero, retain the last observed job outcome
+and record `lastError` without declaring the job stopped.
+
+`orx runs`, `orx logs` and `orx exp wait --interval` read local records. They do
+not contact the remote host or refresh an on-demand run. Wait/wake completion
+signals become available when a refresh observes the terminal result. End the
+periodic agent schedule once the run is terminal and the needed evidence has
+been collected.
+
+Cancellation is independent of the monitoring interval. `orx exp cancel` saves
+intent and starts one bounded control attempt. If the host is unreachable,
+cancellation stays pending; a later refresh, repeated cancel, or application
+startup retries it. On-demand mode does not create an indefinite background
+retry loop. A successful cancel signal still needs observation of termination.
+The remote job is never assumed stopped merely because the host is unreachable.
+
 Machine-local SSH settings can save a container name or ID for each host, and
 an absolute remote storage root (`--remote-root /data/alice/orx`) used instead
 of `~/.orx` for that host's runs and cached source archives. Each run keeps the

src/commands/exp.rs

@@ -232,6 +232,24 @@ const SUPERVISOR_LOG_MAX_BYTES: u64 = 1024 * 1024;
 /// Spawn `orx supervise <runId>` fully detached (own process group, stderr to a log),
 /// so it outlives this command and any SSH session that launched it.
 pub(crate) fn spawn_detached_supervise(run_id: &str) -> Result<()> {
+    let store = Store::open()?;
+    let run = crate::compute::load_run_with_submission_handle(&store, run_id)?;
+    let descriptor = crate::jobs::BackendDescriptor::parse(&run.backend_json)?;
+    if crate::jobs::ssh::monitor::on_demand(&descriptor) {
+        return if run.cancel_requested {
+            spawn_ssh_cancel_once(run_id)
+        } else {
+            Ok(())
+        };
+    }
+    spawn_supervisor(run_id, false)
+}
+
+pub(crate) fn spawn_ssh_cancel_once(run_id: &str) -> Result<()> {
+    spawn_supervisor(run_id, true)
+}
+
+fn spawn_supervisor(run_id: &str, cancel_once: bool) -> Result<()> {
     let exe = crate::paths::spawnable_exe().map_err(|e| {
         anyhow!(
             "Could not locate the orx binary to spawn the supervisor: {}",
@@ -244,13 +262,13 @@ pub(crate) fn spawn_detached_supervise(run_id: &str) -> Result<()> {
         // Rename rather than truncate: a still-running supervisor keeps writing to its handle.
         let _ = std::fs::rename(&path, path.with_extension("log.1"));
     }
-    let mut child = match supervise_command(&exe, run_id, &path).spawn() {
+    let mut child = match supervise_command(&exe, run_id, &path, cancel_once).spawn() {
         // `/proc/self/exe` can name a path this process cannot exec (OR-334); the `orx` on PATH still runs.
         Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
             match crate::local::shell_env::find_on_path("orx").filter(|orx| *orx != exe) {
                 Some(orx) => {
                     let tried = format!("`{}`{}: {error}", exe.display(), current_exe_note(&exe));
-                    let child = supervise_command(&orx, run_id, &path)
+                    let child = supervise_command(&orx, run_id, &path, cancel_once)
                         .spawn()
                         .map_err(|e| {
                             anyhow!(
@@ -285,6 +303,7 @@ fn supervise_command(
     exe: &std::path::Path,
     run_id: &str,
     log_path: &std::path::Path,
+    cancel_once: bool,
 ) -> std::process::Command {
     let stderr = std::fs::OpenOptions::new()
         .create(true)
@@ -298,6 +317,9 @@ fn supervise_command(
         .stdin(std::process::Stdio::null())
         .stdout(std::process::Stdio::null())
         .stderr(stderr);
+    if cancel_once {
+        cmd.arg("--cancel-once");
+    }
     // The supervisor re-resolves its directories from its own environment, so
     // without this a run launched from the macOS app is tracked in a different
     // store than the app is reading.
@@ -365,6 +387,19 @@ pub(crate) fn exp_cancel_reason(chat_session_id: Option<&str>) -> String {
 
 /// Persist cancel intent and ensure an orphaned run gets a fresh supervisor.
 pub(crate) fn request_local_run_cancel(store: &Store, run_id: &str, reason: &str) -> Result<()> {
+    let run = crate::compute::load_run_with_submission_handle(store, run_id)?;
+    let descriptor = crate::jobs::BackendDescriptor::parse(&run.backend_json)?;
+    if descriptor.kind == "ssh_job" && descriptor.job_id.is_none() {
+        store.request_cancel(run_id, reason)?;
+        return Ok(());
+    }
+    if crate::jobs::ssh::monitor::direct(&descriptor) {
+        store.request_cancel(run_id, reason)?;
+        if !crate::local::is_terminal(&run.status) {
+            spawn_ssh_cancel_once(run_id)?;
+        }
+        return Ok(());
+    }
     let lock_path = crate::store::log_path(run_id).with_extension("cancel.lock");
     request_local_run_cancel_with(
         store,

src/commands/mod.rs

@@ -39,6 +39,7 @@ pub mod paper;
 pub mod plan_gate;
 pub mod project;
 pub mod projects;
+pub mod refresh;
 pub mod remote_host;
 pub mod runs;
 pub mod serve;

src/commands/refresh.rs

@@ -0,0 +1,31 @@
+use crate::error::Result;
+
+pub async fn run(args: crate::RefreshArgs) -> Result<()> {
+    let run = match crate::local::chat::trusted_up_port()? {
+        Some(port) => {
+            crate::commands::up::require_ssh_monitoring_support(port).await?;
+            crate::commands::up::chat_request_via_up(
+                port,
+                reqwest::Method::POST,
+                &format!("/api/runs/{}/refresh", args.run_id),
+                &serde_json::json!({"collectLogs":args.collect_logs}),
+            )
+            .await?
+        }
+        None => crate::commands::up::run_response(
+            &crate::jobs::ssh::monitor::refresh(&args.run_id, args.collect_logs).await?,
+        ),
+    };
+    if args.json {
+        println!("{}", serde_json::to_string_pretty(&run)?);
+    } else {
+        let r = run.get("run").unwrap_or(&run);
+        println!(
+            "Run {}: {}",
+            args.run_id,
+            r["status"].as_str().unwrap_or("unknown")
+        );
+        println!("{}", serde_json::to_string_pretty(&run)?);
+    }
+    Ok(())
+}