Skip to content

Choose the Slurm partition per run with --partition - #666

Closed
ChenHsieh wants to merge 2 commits into
alphaXiv:mainfrom
ChenHsieh:chen-slurm-partition
Closed

ChenHsieh wants to merge 2 commits into
alphaXiv:mainfrom
ChenHsieh:chen-slurm-partition

Conversation

@ChenHsieh

Copy link
Copy Markdown
Contributor

Closes #665

Summary

  • orx exp run --backend slurm --partition <name> sets #SBATCH --partition for one run and overrides the saved default. POST /api/runs accepts partition too.
  • A comma-separated list (gpu,batch) is passed through, as Nextflow and Slurm itself allow.
  • The value is checked before it reaches the #SBATCH line (letters, digits, _, -, ., commas between names), for the same reason parse_mem is strict. The saved default is passed through unchanged, as before.
  • The partition a run used is recorded on its backend descriptor and printed in the launch summary.
  • --partition on any other backend is an error.
  • No dashboard change: the per-run value is CLI and API only, like --cpus and --mem at launch.

Test plan

  • cargo test --locked (1098 passed): partition_accepts_names_and_lists_and_rejects_the_rest, per_run_partition_overrides_the_saved_default, partition_is_slurm_only_and_validated_before_submission, the run request round trip, and the descriptor round trip
  • cargo fmt --all --check
  • sbatch --test-only on a Slurm 24.11.6 cluster accepts a script with #SBATCH --partition=batch,highmem_p
  • Not exercised: a full orx exp run against a live cluster

Independent of #664; either can merge first.

The partition was only a saved machine-wide default, so a cluster that keeps
GPU and CPU nodes in separate partitions could not launch both kinds of run
without reconfiguring in between. orx exp run --backend slurm --partition
overrides the default for one run, accepts a comma-separated list, is
validated before it reaches a #SBATCH line, and is recorded on the run.

Co-Authored-By: Claude <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; both launch paths now show the requested partition.

Summary

Adds a per-run Slurm partition through the CLI and API.

  • Slurm runs can choose a partition per run.

The previous finding is fully addressed. No new actionable issues were found in the follow-up changes.

Reviews (2) · Last reviewed commit: "Show the partition when orx up forwards ..." · Reviewed by Greptile

Comment thread src/local/slurm.rs Resolved
The forwarded launch summary carried cores and memory but not the
partition, so a run started while orx up was running did not say where it
went. Both paths now print the same partition line.

Co-Authored-By: Claude <noreply@anthropic.com>
@paperclover

Copy link
Copy Markdown
Contributor

comment written by Claude Opus 5.5

thanks for this. #862 moves Slurm launches onto the new run engine, so instead of asking you to rebase, the per-run --partition flag was ported there in fd1c60b, with you credited as co-author. #862 closes #665 when it merges.

@paperclover paperclover closed this Nov 3, 2026
paperclover added a commit that referenced this pull request Nov 3, 2026
Until now each run had its own detached `orx supervise` process, which
polled over SSH, wrote SQLite directly and lived or died with the laptop
session. Runs went silently stale, a key-auth host could demand an
interactive login, and background reconnects kept logging in after an
authentication failure (OR-638). This moves launch, observation, cancel,
log keeping and result delivery into one engine inside `orx up`; the
dashboard, the CLI and agents become network clients of it.

State. A run has a phase (submitting, starting, running, done, failed,
cancelled, lost) that only moves forward, and an independent contact
(Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh
that is.
Nothing becomes terminal on a timer: only a receipt, a verified cancel, a
changed boot id, a scheduler verdict, a proven-absent launch or Abandon
end a run. NeedsAuth and Unreconciled raise attention notices immediately;
an Unreachable outage raises one after ten minutes.

Submission. The run and its full launch spec are committed before any
side effect, then fenced per attempt. A request that never left the
laptop stays submitting with its contact; a lost reply goes to recovery
(Found, Absent or Unknown per backend). A relaunch that is interrupted
again backs off with its error shown instead of looping.

Transport. One `HostConn` actor per (alias, root) owns one plain `ssh`
child and multiplexes requests over it in priority lanes. Authentication
state is one breaker per alias: after an auth failure no root of that
alias logs in again until the user presses Connect. First contact is one
BatchMode attempt; interactive hosts use a detached master started by
Connect through an askpass bridge. Engine ssh never has a tty.

Logs. The host keeps the whole log; the laptop keeps all of it up to
1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced
marker and retrying until done. Pages outside the kept ranges come from
the host on demand, or a structured 503 while it is away.

Results. Terminal, attention and notify notices land in an outbox and are
delivered to subscribed chats as hidden turns, exactly once per frozen
batch, with backoff and retirement. Agents launch, end their turn, and
are told; `orx exp wait` is one SSE request.

API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs,
paged logs and SSE events; the CLI finds or starts a headless `orx up`
and never polls. A headless server exits after 10 idle minutes; the
desktop app stays alive without a window while runs are active.

Backends. SSH and Slurm use `orx-host`; Slurm splits authority between
the receipt on the shared filesystem, squeue and sacct. Local and Tinker
use orx-host on Unix, including the same receipts and recovery as SSH.
Windows uses job objects to contain and stop local payloads. HF, Modal,
k8s, Ray and Vercel use a generic API handle, one provider table and a
batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop,
writing through the engine and its transport.

Coexistence. Runs held by a live 0.2.x supervisor are only read; free
ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay
for them. Plain-SSH connection sharing stays available through the
server-owned master, with its Include and ControlPath preserved for
other SSH tools.

Also: seeded chaos and property tests against real sshd, a live dashboard
fixture, release CI that builds and embeds the host binaries, and agent
skills documenting the notify file.

Cancellation now separates intent, terminal phase and stop proof. Pending
stops show Cancelling…; complete empty proof shows Cancelled; survivors
show Not fully stopped; incomplete empty proof shows Cancelled · couldn't
confirm. Terminal runs with pending kills stay watched as tombstones.
Abandon records a lost outcome without forgetting that kill, and later
proof settles the contact and sends the affected chats one follow-up.
Slurm allocation teardown settles those tombstones after the wrapper
releases its allocation; scheduler and shared-filesystem evidence keep
their separate authority.

Transport errors distinguish authentication failure from host-key,
network and bootstrap failures before opening the alias breaker.
Plain-SSH sharing is restored without giving background reconnects an
interactive login. macOS host and local process identity use the hardware
UUID, boot identity and native start stamps before trusting a PID.

Notify delivery records durable acknowledgements so reconnects and
restarts preserve pending messages. The API adapter records typed handles
and provider authority, including Vercel Sandbox process execution and
logs, with recovery retaining uncertainty when a process id was lost.

CLI additions include cancelling one run with `orx exp cancel --run`,
`orx exp abandon`, following engine logs with `orx logs -f`, contact-aware
run listings, headless startup, and per-run Slurm `--partition` from Chen
Hsieh's PR #666. Compute tests and interactive Connect use the server's
connections and prompt bridge.

Closes #744
Fixes OR-638
Fixes OR-579
Fixes OR-598
Fixes OR-628
Fixes OR-618
Fixes OR-570
Fixes OR-501
Fixes OR-376
Fixes OR-469
Fixes OR-541
Fixes OR-427
Fixes OR-584
Fixes OR-452
Fixes OR-637
Fixes OR-545
Fixes OR-369
Fixes OR-507
Fixes OR-639
Fixes OR-453
Fixes OR-490
Fixes OR-497
Fixes OR-631
Fixes OR-592
Fixes OR-576
Fixes OR-458
Fixes OR-459
Fixes OR-622
Closes #665

Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com>
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5
paperclover added a commit that referenced this pull request Nov 3, 2026
Until now each run had its own detached `orx supervise` process, which
polled over SSH, wrote SQLite directly and lived or died with the laptop
session. Runs went silently stale, a key-auth host could demand an
interactive login, and background reconnects kept logging in after an
authentication failure (OR-638). This moves launch, observation, cancel,
log keeping and result delivery into one engine inside `orx up`; the
dashboard, the CLI and agents become network clients of it.

State. A run has a phase (submitting, starting, running, done, failed,
cancelled, lost) that only moves forward, and an independent contact
(Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh
that is.
Nothing becomes terminal on a timer: only a receipt, a verified cancel, a
changed boot id, a scheduler verdict, a proven-absent launch or Abandon
end a run. NeedsAuth and Unreconciled raise attention notices immediately;
an Unreachable outage raises one after ten minutes.

Submission. The run and its full launch spec are committed before any
side effect, then fenced per attempt. A request that never left the
laptop stays submitting with its contact; a lost reply goes to recovery
(Found, Absent or Unknown per backend). A relaunch that is interrupted
again backs off with its error shown instead of looping.

Transport. One `HostConn` actor per (alias, root) owns one plain `ssh`
child and multiplexes requests over it in priority lanes. Authentication
state is one breaker per alias: after an auth failure no root of that
alias logs in again until the user presses Connect. First contact is one
BatchMode attempt; interactive hosts use a detached master started by
Connect through an askpass bridge. Engine ssh never has a tty.

Logs. The host keeps the whole log; the laptop keeps all of it up to
1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced
marker and retrying until done. Pages outside the kept ranges come from
the host on demand, or a structured 503 while it is away.

Results. Terminal, attention and notify notices land in an outbox and are
delivered to subscribed chats as hidden turns, exactly once per frozen
batch, with backoff and retirement. Agents launch, end their turn, and
are told; `orx exp wait` is one SSE request.

API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs,
paged logs and SSE events; the CLI finds or starts a headless `orx up`
and never polls. A headless server exits after 10 idle minutes; the
desktop app stays alive without a window while runs are active.

Backends. SSH and Slurm use `orx-host`; Slurm splits authority between
the receipt on the shared filesystem, squeue and sacct. Local and Tinker
use orx-host on Unix, including the same receipts and recovery as SSH.
Windows uses job objects to contain and stop local payloads. HF, Modal,
k8s, Ray and Vercel use a generic API handle, one provider table and a
batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop,
writing through the engine and its transport.

Coexistence. Runs held by a live 0.2.x supervisor are only read; free
ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay
for them. Plain-SSH connection sharing stays available through the
server-owned master, with its Include and ControlPath preserved for
other SSH tools.

Also: seeded chaos and property tests against real sshd, a live dashboard
fixture, release CI that builds and embeds the host binaries, and agent
skills documenting the notify file.

Cancellation now separates intent, terminal phase and stop proof. Pending
stops show Cancelling…; complete empty proof shows Cancelled; survivors
show Not fully stopped; incomplete empty proof shows Cancelled · couldn't
confirm. Terminal runs with pending kills stay watched as tombstones.
Abandon records a lost outcome without forgetting that kill, and later
proof settles the contact and sends the affected chats one follow-up.
Slurm allocation teardown settles those tombstones after the wrapper
releases its allocation; scheduler and shared-filesystem evidence keep
their separate authority.

Transport errors distinguish authentication failure from host-key,
network and bootstrap failures before opening the alias breaker.
Plain-SSH sharing is restored without giving background reconnects an
interactive login. macOS host and local process identity use the hardware
UUID, boot identity and native start stamps before trusting a PID.

Notify delivery records durable acknowledgements so reconnects and
restarts preserve pending messages. The API adapter records typed handles
and provider authority, including Vercel Sandbox process execution and
logs, with recovery retaining uncertainty when a process id was lost.

CLI additions include cancelling one run with `orx exp cancel --run`,
`orx exp abandon`, following engine logs with `orx logs -f`, contact-aware
run listings, headless startup, and per-run Slurm `--partition` from Chen
Hsieh's PR #666. Compute tests and interactive Connect use the server's
connections and prompt bridge.

Closes #744
Fixes OR-638
Fixes OR-579
Fixes OR-598
Fixes OR-628
Fixes OR-618
Fixes OR-570
Fixes OR-501
Fixes OR-376
Fixes OR-469
Fixes OR-541
Fixes OR-427
Fixes OR-584
Fixes OR-452
Fixes OR-637
Fixes OR-545
Fixes OR-369
Fixes OR-507
Fixes OR-639
Fixes OR-453
Fixes OR-490
Fixes OR-497
Fixes OR-631
Fixes OR-592
Fixes OR-576
Fixes OR-458
Fixes OR-459
Fixes OR-622
Closes #665

Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com>
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5
paperclover added a commit that referenced this pull request Nov 5, 2026
Until now each run had its own detached `orx supervise` process, which
polled over SSH, wrote SQLite directly and lived or died with the laptop
session. Runs went silently stale, a key-auth host could demand an
interactive login, and background reconnects kept logging in after an
authentication failure (OR-638). This moves launch, observation, cancel,
log keeping and result delivery into one engine inside `orx up`; the
dashboard, the CLI and agents become network clients of it.

State. A run has a phase (submitting, starting, running, done, failed,
cancelled, lost) that only moves forward, and an independent contact
(Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh
that is.
Nothing becomes terminal on a timer: only a receipt, a verified cancel, a
changed boot id, a scheduler verdict, a proven-absent launch or Abandon
end a run. NeedsAuth and Unreconciled raise attention notices immediately;
an Unreachable outage raises one after ten minutes.

Submission. The run and its full launch spec are committed before any
side effect, then fenced per attempt. A request that never left the
laptop stays submitting with its contact; a lost reply goes to recovery
(Found, Absent or Unknown per backend). A relaunch that is interrupted
again backs off with its error shown instead of looping.

Transport. One `HostConn` actor per (alias, root) owns one plain `ssh`
child and multiplexes requests over it in priority lanes. Authentication
state is one breaker per alias: after an auth failure no root of that
alias logs in again until the user presses Connect. First contact is one
BatchMode attempt; interactive hosts use a detached master started by
Connect through an askpass bridge. Engine ssh never has a tty.

Logs. The host keeps the whole log; the laptop keeps all of it up to
1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced
marker and retrying until done. Pages outside the kept ranges come from
the host on demand, or a structured 503 while it is away.

Results. Terminal, attention and notify notices land in an outbox and are
delivered to subscribed chats as hidden turns, exactly once per frozen
batch, with backoff and retirement. Agents launch, end their turn, and
are told; `orx exp wait` is one SSE request.

API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs,
paged logs and SSE events; the CLI finds or starts a headless `orx up`
and never polls. A headless server exits after 10 idle minutes; the
desktop app stays alive without a window while runs are active.

Backends. SSH and Slurm use `orx-host`; Slurm splits authority between
the receipt on the shared filesystem, squeue and sacct. Local and Tinker
use orx-host on Unix, including the same receipts and recovery as SSH.
Windows uses job objects to contain and stop local payloads. HF, Modal,
k8s, Ray and Vercel use a generic API handle, one provider table and a
batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop,
writing through the engine and its transport.

Coexistence. Runs held by a live 0.2.x supervisor are only read; free
ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay
for them. Plain-SSH connection sharing stays available through the
server-owned master, with its Include and ControlPath preserved for
other SSH tools.

Also: seeded chaos and property tests against real sshd, a live dashboard
fixture, release CI that builds and embeds the host binaries, and agent
skills documenting the notify file.

Cancellation now separates intent, terminal phase and stop proof. Pending
stops show Cancelling…; complete empty proof shows Cancelled; survivors
show Not fully stopped; incomplete empty proof shows Cancelled · couldn't
confirm. Terminal runs with pending kills stay watched as tombstones.
Abandon records a lost outcome without forgetting that kill, and later
proof settles the contact and sends the affected chats one follow-up.
Slurm allocation teardown settles those tombstones after the wrapper
releases its allocation; scheduler and shared-filesystem evidence keep
their separate authority.

Transport errors distinguish authentication failure from host-key,
network and bootstrap failures before opening the alias breaker.
Plain-SSH sharing is restored without giving background reconnects an
interactive login. macOS host and local process identity use the hardware
UUID, boot identity and native start stamps before trusting a PID.

Notify delivery records durable acknowledgements so reconnects and
restarts preserve pending messages. The API adapter records typed handles
and provider authority, including Vercel Sandbox process execution and
logs, with recovery retaining uncertainty when a process id was lost.

CLI additions include cancelling one run with `orx exp cancel --run`,
`orx exp abandon`, following engine logs with `orx logs -f`, contact-aware
run listings, headless startup, and per-run Slurm `--partition` from Chen
Hsieh's PR #666. Compute tests and interactive Connect use the server's
connections and prompt bridge.

Closes #744
Fixes OR-638
Fixes OR-579
Fixes OR-598
Fixes OR-628
Fixes OR-618
Fixes OR-570
Fixes OR-501
Fixes OR-376
Fixes OR-469
Fixes OR-541
Fixes OR-427
Fixes OR-584
Fixes OR-452
Fixes OR-637
Fixes OR-545
Fixes OR-369
Fixes OR-507
Fixes OR-639
Fixes OR-453
Fixes OR-490
Fixes OR-497
Fixes OR-631
Fixes OR-592
Fixes OR-576
Fixes OR-458
Fixes OR-459
Fixes OR-622
Closes #665

Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com>
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5
paperclover added a commit that referenced this pull request Nov 5, 2026
Until now each run had its own detached `orx supervise` process, which
polled over SSH, wrote SQLite directly and lived or died with the laptop
session. Runs went silently stale, a key-auth host could demand an
interactive login, and background reconnects kept logging in after an
authentication failure (OR-638). This moves launch, observation, cancel,
log keeping and result delivery into one engine inside `orx up`; the
dashboard, the CLI and agents become network clients of it.

State. A run has a phase (submitting, starting, running, done, failed,
cancelled, lost) that only moves forward, and an independent contact
(Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh
that is.
Nothing becomes terminal on a timer: only a receipt, a verified cancel, a
changed boot id, a scheduler verdict, a proven-absent launch or Abandon
end a run. NeedsAuth and Unreconciled raise attention notices immediately;
an Unreachable outage raises one after ten minutes.

Submission. The run and its full launch spec are committed before any
side effect, then fenced per attempt. A request that never left the
laptop stays submitting with its contact; a lost reply goes to recovery
(Found, Absent or Unknown per backend). A relaunch that is interrupted
again backs off with its error shown instead of looping.

Transport. One `HostConn` actor per (alias, root) owns one plain `ssh`
child and multiplexes requests over it in priority lanes. Authentication
state is one breaker per alias: after an auth failure no root of that
alias logs in again until the user presses Connect. First contact is one
BatchMode attempt; interactive hosts use a detached master started by
Connect through an askpass bridge. Engine ssh never has a tty.

Logs. The host keeps the whole log; the laptop keeps all of it up to
1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced
marker and retrying until done. Pages outside the kept ranges come from
the host on demand, or a structured 503 while it is away.

Results. Terminal, attention and notify notices land in an outbox and are
delivered to subscribed chats as hidden turns, exactly once per frozen
batch, with backoff and retirement. Agents launch, end their turn, and
are told; `orx exp wait` is one SSE request.

API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs,
paged logs and SSE events; the CLI finds or starts a headless `orx up`
and never polls. A headless server exits after 10 idle minutes; the
desktop app stays alive without a window while runs are active.

Backends. SSH and Slurm use `orx-host`; Slurm splits authority between
the receipt on the shared filesystem, squeue and sacct. Local and Tinker
use orx-host on Unix, including the same receipts and recovery as SSH.
Windows uses job objects to contain and stop local payloads. HF, Modal,
k8s, Ray and Vercel use a generic API handle, one provider table and a
batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop,
writing through the engine and its transport.

Coexistence. Runs held by a live 0.2.x supervisor are only read; free
ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay
for them. Plain-SSH connection sharing stays available through the
server-owned master, with its Include and ControlPath preserved for
other SSH tools.

Also: seeded chaos and property tests against real sshd, a live dashboard
fixture, release CI that builds and embeds the host binaries, and agent
skills documenting the notify file.

Cancellation now separates intent, terminal phase and stop proof. Pending
stops show Cancelling…; complete empty proof shows Cancelled; survivors
show Not fully stopped; incomplete empty proof shows Cancelled · couldn't
confirm. Terminal runs with pending kills stay watched as tombstones.
Abandon records a lost outcome without forgetting that kill, and later
proof settles the contact and sends the affected chats one follow-up.
Slurm allocation teardown settles those tombstones after the wrapper
releases its allocation; scheduler and shared-filesystem evidence keep
their separate authority.

Transport errors distinguish authentication failure from host-key,
network and bootstrap failures before opening the alias breaker.
Plain-SSH sharing is restored without giving background reconnects an
interactive login. macOS host and local process identity use the hardware
UUID, boot identity and native start stamps before trusting a PID.

Notify delivery records durable acknowledgements so reconnects and
restarts preserve pending messages. The API adapter records typed handles
and provider authority, including Vercel Sandbox process execution and
logs, with recovery retaining uncertainty when a process id was lost.

CLI additions include cancelling one run with `orx exp cancel --run`,
`orx exp abandon`, following engine logs with `orx logs -f`, contact-aware
run listings, headless startup, and per-run Slurm `--partition` from Chen
Hsieh's PR #666. Compute tests and interactive Connect use the server's
connections and prompt bridge.

Closes #744
Fixes OR-638
Fixes OR-579
Fixes OR-598
Fixes OR-628
Fixes OR-618
Fixes OR-570
Fixes OR-501
Fixes OR-376
Fixes OR-469
Fixes OR-541
Fixes OR-427
Fixes OR-584
Fixes OR-452
Fixes OR-637
Fixes OR-545
Fixes OR-369
Fixes OR-507
Fixes OR-639
Fixes OR-453
Fixes OR-490
Fixes OR-497
Fixes OR-631
Fixes OR-592
Fixes OR-576
Fixes OR-458
Fixes OR-459
Fixes OR-622
Closes #665

Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com>
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5
paperclover added a commit that referenced this pull request Nov 5, 2026
Until now each run had its own detached `orx supervise` process, which
polled over SSH, wrote SQLite directly and lived or died with the laptop
session. Runs went silently stale, a key-auth host could demand an
interactive login, and background reconnects kept logging in after an
authentication failure (OR-638). This moves launch, observation, cancel,
log keeping and result delivery into one engine inside `orx up`; the
dashboard, the CLI and agents become network clients of it.

State. A run has a phase (submitting, starting, running, done, failed,
cancelled, lost) that only moves forward, and an independent contact
(Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh
that is.
Nothing becomes terminal on a timer: only a receipt, a verified cancel, a
changed boot id, a scheduler verdict, a proven-absent launch or Abandon
end a run. NeedsAuth and Unreconciled raise attention notices immediately;
an Unreachable outage raises one after ten minutes.

Submission. The run and its full launch spec are committed before any
side effect, then fenced per attempt. A request that never left the
laptop stays submitting with its contact; a lost reply goes to recovery
(Found, Absent or Unknown per backend). A relaunch that is interrupted
again backs off with its error shown instead of looping.

Transport. One `HostConn` actor per (alias, root) owns one plain `ssh`
child and multiplexes requests over it in priority lanes. Authentication
state is one breaker per alias: after an auth failure no root of that
alias logs in again until the user presses Connect. First contact is one
BatchMode attempt; interactive hosts use a detached master started by
Connect through an askpass bridge. Engine ssh never has a tty.

Logs. The host keeps the whole log; the laptop keeps all of it up to
1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced
marker and retrying until done. Pages outside the kept ranges come from
the host on demand, or a structured 503 while it is away.

Results. Terminal, attention and notify notices land in an outbox and are
delivered to subscribed chats as hidden turns, exactly once per frozen
batch, with backoff and retirement. Agents launch, end their turn, and
are told; `orx exp wait` is one SSE request.

API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs,
paged logs and SSE events; the CLI finds or starts a headless `orx up`
and never polls. A headless server exits after 10 idle minutes; the
desktop app stays alive without a window while runs are active.

Backends. SSH and Slurm use `orx-host`; Slurm splits authority between
the receipt on the shared filesystem, squeue and sacct. Local and Tinker
use orx-host on Unix, including the same receipts and recovery as SSH.
Windows uses job objects to contain and stop local payloads. HF, Modal,
k8s, Ray and Vercel use a generic API handle, one provider table and a
batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop,
writing through the engine and its transport.

Coexistence. Runs held by a live 0.2.x supervisor are only read; free
ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay
for them. Plain-SSH connection sharing stays available through the
server-owned master, with its Include and ControlPath preserved for
other SSH tools.

Also: seeded chaos and property tests against real sshd, a live dashboard
fixture, release CI that builds and embeds the host binaries, and agent
skills documenting the notify file.

Cancellation now separates intent, terminal phase and stop proof. Pending
stops show Cancelling…; complete empty proof shows Cancelled; survivors
show Not fully stopped; incomplete empty proof shows Cancelled · couldn't
confirm. Terminal runs with pending kills stay watched as tombstones.
Abandon records a lost outcome without forgetting that kill, and later
proof settles the contact and sends the affected chats one follow-up.
Slurm allocation teardown settles those tombstones after the wrapper
releases its allocation; scheduler and shared-filesystem evidence keep
their separate authority.

Transport errors distinguish authentication failure from host-key,
network and bootstrap failures before opening the alias breaker.
Plain-SSH sharing is restored without giving background reconnects an
interactive login. macOS host and local process identity use the hardware
UUID, boot identity and native start stamps before trusting a PID.

Notify delivery records durable acknowledgements so reconnects and
restarts preserve pending messages. The API adapter records typed handles
and provider authority, including Vercel Sandbox process execution and
logs, with recovery retaining uncertainty when a process id was lost.

CLI additions include cancelling one run with `orx exp cancel --run`,
`orx exp abandon`, following engine logs with `orx logs -f`, contact-aware
run listings, headless startup, and per-run Slurm `--partition` from Chen
Hsieh's PR #666. Compute tests and interactive Connect use the server's
connections and prompt bridge.

Closes #744
Fixes OR-638
Fixes OR-579
Fixes OR-598
Fixes OR-628
Fixes OR-618
Fixes OR-570
Fixes OR-501
Fixes OR-376
Fixes OR-469
Fixes OR-541
Fixes OR-427
Fixes OR-584
Fixes OR-452
Fixes OR-637
Fixes OR-545
Fixes OR-369
Fixes OR-507
Fixes OR-639
Fixes OR-453
Fixes OR-490
Fixes OR-497
Fixes OR-631
Fixes OR-592
Fixes OR-576
Fixes OR-458
Fixes OR-459
Fixes OR-622
Closes #665

Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com>
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Slurm runs can't choose a partition per run

2 participants