Repository navigation
feat: rewrite the experiment engine - #862
Merged
Merged
Conversation
This was referenced Nov 3, 2026
paperclover
force-pushed
the
clo/run-engine
branch
3 times, most recently
from
November 3, 2026 08:19
b0dca8b to
68f4586
Compare
paperclover
changed the title
paperclover
force-pushed
the
clo/run-engine
branch
2 times, most recently
from
November 5, 2026 01:18
9323dd3 to
1590bbc
Compare
paperclover
marked this pull request as ready for review
November 5, 2026 01:33
Contributor
 [High impact] The PR changes how new experiment runs are launched, tracked, and recovered across compute backends. Fix the mixed-version dashboard log contract and missing macOS host helpers before merging. FindingsFix with agent prompt### Issue 1
src/commands/up/api_v2.rs:588-592
This replaces the dashboard log API’s `offset`/`generation` requests with required `start`/`end` fields and a different response shape, but `DASHBOARD_PROTOCOL` stays at `3`. An older laptop still accepts this remote server, then its log requests fail because those fields are missing. A new laptop connected to an older remote also expects the wrong response shape.
Bump the dashboard protocol so mixed versions receive update guidance instead. The repository guide requires a protocol bump whenever the remote API changes.
### Issue 2
build.rs:100-102
Linux and Windows releases skip the Darwin helper even when the release artifact contains it. With the engine enabled, connecting either release to a macOS SSH host reaches `binary()` with `Darwin`, finds no embedded match, and fails with `No orx-host for darwin-…`. This also prevents monitoring adopted runs on that host.
Embed the Darwin helper regardless of the client build target. It runs on the remote machine, not the client.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryMoves experiment launch, recovery, cancellation, log collection, and result delivery into one engine in
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+588
to
+592
| #[derive(Deserialize)] | ||
| struct LogQuery { | ||
| start: u64, | ||
| end: u64, | ||
| ident: Option<String>, |
Contributor
There was a problem hiding this comment.
This replaces the dashboard log API’s offset/generation requests with required start/end fields and a different response shape, but DASHBOARD_PROTOCOL stays at 3. An older laptop still accepts this remote server, then its log requests fail because those fields are missing. A new laptop connected to an older remote also expects the wrong response shape.
Bump the dashboard protocol so mixed versions receive update guidance instead. The repository guide requires a protocol bump whenever the remote API changes.
Context Used: AGENTS.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/commands/up/api_v2.rs
Line: 588-592
Comment:
**Mixed versions break remote logs**
This replaces the dashboard log API’s `offset`/`generation` requests with required `start`/`end` fields and a different response shape, but `DASHBOARD_PROTOCOL` stays at `3`. An older laptop still accepts this remote server, then its log requests fail because those fields are missing. A new laptop connected to an older remote also expects the wrong response shape.
Bump the dashboard protocol so mixed versions receive update guidance instead. The repository guide requires a protocol bump whenever the remote API changes.
**Context Used:** AGENTS.md ([source](https://github.com/alphaxiv/openresearch/blob/main/AGENTS.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Contributor
Author
There was a problem hiding this comment.
valid. bumped DASHBOARD_PROTOCOL to 4 in beac997 (src/commands/up_remote.rs), so a mixed laptop/remote pair gets the update prompt instead of failing log requests.
Comment on lines
+100
to
+102
| if host_os == "darwin" && os != "macos" { | ||
| continue; | ||
| } |
Contributor
There was a problem hiding this comment.
Linux and Windows releases skip the Darwin helper even when the release artifact contains it. With the engine enabled, connecting either release to a macOS SSH host reaches binary() with Darwin, finds no embedded match, and fails with No orx-host for darwin-…. This also prevents monitoring adopted runs on that host.
Embed the Darwin helper regardless of the client build target. It runs on the remote machine, not the client.
Prompt To Fix With AI
This is a comment left during a code review.
Path: build.rs
Line: 100-102
Comment:
**macOS hosts lack their helper**
Linux and Windows releases skip the Darwin helper even when the release artifact contains it. With the engine enabled, connecting either release to a macOS SSH host reaches `binary()` with `Darwin`, finds no embedded match, and fails with `No orx-host for darwin-…`. This also prevents monitoring adopted runs on that host.
Embed the Darwin helper regardless of the client build target. It runs on the remote machine, not the client.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Contributor
Author
There was a problem hiding this comment.
valid. that was a download-size call, but with canary forcing the engine on it strands Linux and Windows laptops on Mac hosts. build.rs now embeds the Darwin helper for every client target (8e21be9), about 2 MB.
paperclover
force-pushed
the
clo/run-engine
branch
2 times, most recently
from
November 5, 2026 02:24
890ccf3 to
8e21be9
Compare
The run engine talks to hosts over plain OpenSSH, so its transport, auth and log tests need a real server rather than a mock. `orx-test-support` starts an unprivileged sshd per test in a temporary home, with its own host and user keys, and can put a fault proxy in front of it: black-hole, pause, latency, bandwidth limits, refuse and reset. It counts authentication attempts from the sshd log, which is how the tests prove the engine never logs in again after an auth failure, and it models interactive hosts (encrypted keys, keyboard-interactive) for the Connect flow. Each test process group is killed when its handle drops, and `stop_server` signals a headless `orx up` only while its descriptor still names the same process, so a failed test leaves nothing running. The harness also checks missing SSH includes, shared-session isolation, connection limits and hostile proxy input without touching the real home. Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol Assisted-by: claude-fable-5.1 Assisted-by: claude-sonnet-5.5
orx-host is the only thing the run engine puts on a remote machine: one std-only Rust binary, built static for x86_64/aarch64 Linux (musl) and universal Darwin, uploaded once per host root under its sha256. - `orx-host wrap <run dir>` supervises one run in its own session. It becomes subreaper, records its identity (node, boot id, pid, start time), creates `started` exactly once, runs the payload, and publishes an `exit.json` receipt by rename. Completion is the direct child's exit; processes it left behind are listed and killed on request. Cancel is the wrapper's own stop sequence (cancel hook, TERM, KILL, rescan via pidfd), so verification never races its exit. Cancelled survivors keep being killed with fresh cleanup receipts until proof is empty; Slurm releases the allocation after its receipt. - `orx-host serve --root <root>` lives as long as one ssh session and answers length-prefixed JSON frames out of order: Hello, Launch (idempotent per run id), a Changes long-poll keyed on the client's last summary, paged ReadLog, content-addressed source upload and pinning, Slurm submit/queue/cancel, bounded Exec, and read-only ops for 0.2.x run dirs. - `orx-host-proto` is the shared wire format. Byte fields are base64; frames are capped at 8 MiB and pages at 1 MiB. Nothing persists on the host except files and each run's own wrapper: no daemon to install, upgrade or restart. `build.rs` embeds the host binaries from ORX_EMBED_HOST_DIR (required for production builds), and `scripts/build-host.mjs` builds them with cargo-zigbuild. macOS identity uses the hardware UUID and boot time, with native process scans and identity checks before signaling. Durable notify acknowledgements survive reconnects. Linux stop scans exclude unreadable OpenSSH session processes from incomplete-proof candidates. FROZEN.md records the disk and wire compatibility contract, with fixtures covering older receipts and explicit proof lists. Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol Assisted-by: claude-fable-5.1 Assisted-by: claude-sonnet-5.5
Until now each run had its own detached `orx supervise` process, which polled over SSH, wrote SQLite directly and lived or died with the laptop session. Runs went silently stale, a key-auth host could demand an interactive login, and background reconnects kept logging in after an authentication failure (OR-638). This moves launch, observation, cancel, log keeping and result delivery into one engine inside `orx up`; the dashboard, the CLI and agents become network clients of it. State. A run has a phase (submitting, starting, running, done, failed, cancelled, lost) that only moves forward, and an independent contact (Live, Unreachable, NeedsAuth, Unreconciled, Stopping) saying how fresh that is. Nothing becomes terminal on a timer: only a receipt, a verified cancel, a changed boot id, a scheduler verdict, a proven-absent launch or Abandon end a run. NeedsAuth and Unreconciled raise attention notices immediately; an Unreachable outage raises one after ten minutes. Submission. The run and its full launch spec are committed before any side effect, then fenced per attempt. A request that never left the laptop stays submitting with its contact; a lost reply goes to recovery (Found, Absent or Unknown per backend). A relaunch that is interrupted again backs off with its error shown instead of looping. Transport. One `HostConn` actor per (alias, root) owns one plain `ssh` child and multiplexes requests over it in priority lanes. Authentication state is one breaker per alias: after an auth failure no root of that alias logs in again until the user presses Connect. First contact is one BatchMode attempt; interactive hosts use a detached master started by Connect through an askpass bridge. Engine ssh never has a tty. Logs. The host keeps the whole log; the laptop keeps all of it up to 1 GiB, else the first 64 MiB and last 256 MiB, resuming from a synced marker and retrying until done. Pages outside the kept ranges come from the host on demand, or a structured 503 while it is away. Results. Terminal, attention and notify notices land in an outbox and are delivered to subscribed chats as hidden turns, exactly once per frozen batch, with backoff and retirement. Agents launch, end their turn, and are told; `orx exp wait` is one SSE request. API. `/api/v2` (bearer token, HMAC hello for discovery) serves runs, paged logs and SSE events; the CLI finds or starts a headless `orx up` and never polls. A headless server exits after 10 idle minutes; the desktop app stays alive without a window while runs are active. Backends. SSH and Slurm use `orx-host`; Slurm splits authority between the receipt on the shared filesystem, squeue and sacct. Local and Tinker use orx-host on Unix, including the same receipts and recovery as SSH. Windows uses job objects to contain and stop local payloads. HF, Modal, k8s, Ray and Vercel use a generic API handle, one provider table and a batched adapter with a log spool; OpenResearch boxes keep the 0.2.x loop, writing through the engine and its transport. Coexistence. Runs held by a live 0.2.x supervisor are only read; free ones are adopted. The 0.2.x supervisor, ssh-service and helper.py stay for them. Plain-SSH connection sharing stays available through the server-owned master, with its Include and ControlPath preserved for other SSH tools. Also: seeded chaos and property tests against real sshd, a live dashboard fixture, release CI that builds and embeds the host binaries, and agent skills documenting the notify file. Cancellation now separates intent, terminal phase and stop proof. Pending stops show Cancelling…; complete empty proof shows Cancelled; survivors show Not fully stopped; incomplete empty proof shows Cancelled · couldn't confirm. Terminal runs with pending kills stay watched as tombstones. Abandon records a lost outcome without forgetting that kill, and later proof settles the contact and sends the affected chats one follow-up. Slurm allocation teardown settles those tombstones after the wrapper releases its allocation; scheduler and shared-filesystem evidence keep their separate authority. Transport errors distinguish authentication failure from host-key, network and bootstrap failures before opening the alias breaker. Plain-SSH sharing is restored without giving background reconnects an interactive login. macOS host and local process identity use the hardware UUID, boot identity and native start stamps before trusting a PID. Notify delivery records durable acknowledgements so reconnects and restarts preserve pending messages. The API adapter records typed handles and provider authority, including Vercel Sandbox process execution and logs, with recovery retaining uncertainty when a process id was lost. CLI additions include cancelling one run with `orx exp cancel --run`, `orx exp abandon`, following engine logs with `orx logs -f`, contact-aware run listings, headless startup, and per-run Slurm `--partition` from Chen Hsieh's PR #666. Compute tests and interactive Connect use the server's connections and prompt bridge. Closes #744 Fixes OR-638 Fixes OR-579 Fixes OR-598 Fixes OR-628 Fixes OR-618 Fixes OR-570 Fixes OR-501 Fixes OR-376 Fixes OR-469 Fixes OR-541 Fixes OR-427 Fixes OR-584 Fixes OR-452 Fixes OR-637 Fixes OR-545 Fixes OR-369 Fixes OR-507 Fixes OR-639 Fixes OR-453 Fixes OR-490 Fixes OR-497 Fixes OR-631 Fixes OR-592 Fixes OR-576 Fixes OR-458 Fixes OR-459 Fixes OR-622 Closes #665 Co-authored-by: Chen Hsieh <chen.hsieh.uga@gmail.com> Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol Assisted-by: claude-fable-5.1 Assisted-by: claude-sonnet-5.5
The dashboard reads runs from the engine and shows phase and contact together: one badge per run (Cancelling…, Cancelled, Not fully stopped, Cancelled · couldn't confirm, needs auth and abandoned), with the status marked stale and dated whenever contact is not Live. A contact notice explains what is wrong and offers the one action that helps: Connect for a host that needs auth, or Abandon for a run whose host is gone. A finished run whose result never reached its chat says why. Logs are a paged terminal over the engine's log API: it follows the live tail through SSE, can jump anywhere in a multi-gigabyte log, and retries a page held only by an unreachable host when it returns. Compute › SSH shows each host once, in the worst state of its connections, with the cause, Connect and Retry; a run shows the state of its own connection. Cancel evidence names survivors and incomplete scans consistently across run views. Later cleanup proof updates the same badge; translated catalogs and focused model tests cover these projections and contact actions. Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol Assisted-by: claude-fable-5.1 Assisted-by: claude-sonnet-5.5
The arc/ essays landed on main while the run engine was in flight and still described the per-run `orx supervise`, the SSH service and helper.py as how runs work. They are now the home for the engine design: experiments.md covers phase and contact, submission, cancel and Abandon, logs, notices and rentals; remote.md covers orx-host, Slurm authority, transport, the auth breaker and plain-SSH sharing. FROZEN.md stays the canonical freeze list the essays link to. AGENTS.md's run summary now names the engine. docs/runner-stress.md is a runbook of failure and recovery drills beyond the deterministic test tiers. Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol Assisted-by: claude-fable-5.1 Assisted-by: claude-sonnet-5.5
Show cancellation pending until the machine can receive the cancel, and warn before stopping run tracking. Assisted-by: gpt-6.1-sol
Stop OpenResearch rentals through the existing companion API even when SSH is unreachable or rejects the key. Accepted release is authoritative stop proof; failed release keeps the intent retryable. Surface continuing billing and a Stop box action without adding a contact-loss timer. Assisted-by: gpt-6.1-sol
"Cancel" beside a run that is being cancelled reads as either choice. Assisted-by: claude-opus-5.5 Assisted-by: gpt-6.1-sol
Request Cancel after 15 minutes of key rejection on a previously reachable rented box. API disappearance or release proof settles the run; network outages and first contact never qualify. Keep transport and auth history separate for each endpoint and rental so one box cannot supply another run with cancellation evidence. Assisted-by: gpt-6.1-sol
Import each legacy wakeup once, including writes from a downgraded CLI or one still running alongside the engine, without replaying delivered notices. Prefer published kept ranges after adoption: an old generation cursor can otherwise hide the engine-collected log. Bound cursor reads to committed bytes. Assisted-by: gpt-6.1-sol
…directory Windows refuses to delete the data directory while a host watcher task still holds its store, which failed the rental-transport test on CI. Assisted-by: claude-opus-5.5
Keep regular releases on the supervisor launchers while the engine is tested. Canary builds force the engine; stable and dev builds opt in through the "New experiment engine" setting (Settings and Compute) or ORX_RUN_ENGINE=1|0, default off. The choice is made once, where the server accepts a new submission, so flipping it never strands a run in flight. Each mode has one owner for orphaned runs: with the setting off, supervisors are re-spawned and the engine only watches and relays their notices; with it on, the engine adopts a free run only after the same settle window trunk used. On Windows, Connect also logs in the supervisor service when a supervised run or launch needs it. Engine runs now record started_at and ended_at from the source's own timestamps (wrapper, Slurm, provider), so queue and run durations can be derived later without UI changes now. Supervisor runs keep their exit code. Assisted-by: gpt-6.1-sol
…tials Provider authentication rejection was treated as an outage, directing API runs toward host recovery. Preserve HTTP auth status and recognize SDK and kubectl rejections so those runs request credentials, with provider-specific dashboard recovery. Known payload exit codes disappeared at the client/adapter boundary. Carry Ray, Vercel, Modal, Kubernetes and HF status evidence into terminal observations and the stored result. Retain failure detail when HF does not report a numeric outcome. Supervised SSH runs on the old path now read their exit receipt too, not only rented boxes. A provider can report a terminated sandbox as a failed process. Reconcile a nonzero exit against durable cancel intent so an accepted stop ends Cancelled across providers, retaining the code even after restart. Fake provider regressions cover rejected credentials, outages, persisted exit 7 and requested stops. Assisted-by: gpt-6.1-sol Assisted-by: claude-opus-5.5
Keep hosted cancellation semantics separate from wrapper stop proof, restore supervised monitoring notices and Windows login attempts, and retain source timestamp bounds without replaying terminal effects. Assisted-by: gpt-6.1-sol
A fixed log window opened scrolled to its bottom, so "Go to start" showed the last rows of bytes 0 to 1 MB under a position of 0 bytes. The window is now 256 KiB so wrapped lines fit xterm scrollback, and a jump scrolls to its top. Byte sizes under 1 KB read "0 bytes". A run that never launched answered log reads with "Run <id> has no handle"; it now says it never started. The run row and badge show "Not fully stopped" and keep the count and host in the tooltip and the detail notice. Assisted-by: claude-sonnet-5.5
The engine changes remote dashboard routes, including the paged log request and response, so a laptop and a remote of different versions must see a protocol mismatch and get update guidance instead of failing requests. Assisted-by: claude-opus-5.5
A Linux or Windows laptop with the engine on could not run or monitor runs on a Mac SSH host because only macOS builds carried the Darwin helper. It runs on the remote, so every client embeds it, at about 2 MB. Assisted-by: claude-opus-5.5
Route laptop-side engine commands through the terminal environment and keep host-side spawning in the host process environment. Assisted-by: gpt-6.1-sol
Exec kills its command process group after the launcher exits. nohup leaves the rental supervisor in that group, so the engine kills it before the payload can start. Start the supervisor in its own session while keeping the redirected SSH pipes closed. Assisted-by: gpt-6.1-sol
paperclover
force-pushed
the
clo/run-engine
branch
from
November 5, 2026 03:09
8e21be9 to
c7efe51
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SSH experiments could go stale after a laptop disconnect, demand an interactive login on a key-auth host, or keep retrying rejected credentials (ORX-638).
One engine in
orx uplaunches, watches, cancels and reports new runs. A per-runorx-hostpreserves execution and receipts across disconnects. After authentication fails, one breaker per alias blocks further logins until Connect. Managed rentals release on Cancel or Abandon, and request cancellation after 15 minutes of key rejection on a previously reachable box. Engine runs recordstarted_atandended_atfrom the source's own clock, so queue and run time can be shown later.The engine is behind a "New experiment engine" setting (Settings and Compute), off by default;
ORX_RUN_ENGINE=1|0overrides it, and canary builds always use the engine. With it off, new runs launch through trunk's supervisors, which also own orphaned runs, while the engine only watches them; existing 0.2.x supervisors keep their runs either way. The Fixes below apply when the engine is on. Run results now arrive as their own hidden chat turn; delegate news keeps trunk's visible batching.Architecture: experiments, remote transport, storage. Validation on this tree: fmt, clippy with warnings denied, the full Rust suite with the setting off and with
ORX_RUN_ENGINE=1, and 694 UI tests. Real runs on a macOS SSH host covered both settings, flips with runs in flight, restarts, a killed supervisor, offline finishes and cancels of payloads that resist TERM. Earlier trees passed seven real Slurm scenarios, an 85-minute macOS GPU run, a published 0.2.23 upgrade replay, five real Claude result-delivery scenarios, and provider smoke runs on Ray, Vercel, Modal and Hugging Face Jobs. Windows is covered by unit tests only.Closes #744
Fixes ORX-638
Fixes ORX-559
Fixes ORX-579
Fixes ORX-598
Fixes ORX-628
Fixes ORX-618
Fixes ORX-570
Fixes ORX-501
Fixes ORX-376
Fixes ORX-469
Fixes ORX-541
Fixes ORX-427
Fixes ORX-584
Fixes ORX-452
Fixes ORX-637
Fixes ORX-545
Fixes ORX-369
Fixes ORX-507
Fixes ORX-639
Fixes ORX-453
Fixes ORX-490
Fixes ORX-497
Fixes ORX-631
Fixes ORX-592
Fixes ORX-576
Fixes ORX-458
Fixes ORX-459
Fixes ORX-622
Closes #665
Fixes ORX-587
Fixes ORX-546
Fixes ORX-641
Fixes ORX-640
Fixes ORX-363
Fixes ORX-495
Fixes ORX-588
Fixes ORX-538
Fixes ORX-644
Fixes ORX-489
Fixes ORX-619
Fixes ORX-400
Fixes ORX-650
Closes #790
Related, not closed by this PR:
#824, ORX-652, ORX-642: Windows ACL code on the old path; waiting on the Windows test
ORX-543: Windows connection count; waiting on the Windows test
ORX-648: Windows login lasts one session; waiting on the Windows test
ORX-485, ORX-580: OpenResearch boxes still use the 0.2.x box loop; follow-up moves them onto orx-host
ORX-568: remote runs survive a closed lid; local runs need a keep-awake follow-up
Assisted-by: claude-opus-5.5
Assisted-by: gpt-6.1-sol
Assisted-by: claude-fable-5.1
Assisted-by: claude-sonnet-5.5