exe.dev driver: the second real provider

Parent

Part of #154 (remote agent sessions). Phase 4. Design doc docs/designs/remote-workrooms.md: Provider Decision, premise 8, the Success Criteria item "The same workroom behaves identically on two different providers", and Next Steps item 5 ("the container fixture alone does not establish parity across two real providers"). Provider survey: docs/designs/oq20-remote-git-credentials.md, "exe.dev" and "Other providers" (PR #248).

The provider: exe.dev (decided 2026-09-27, owner)

What is already known:

Persistent Linux VMs reached over ssh (ssh exe.dev), with the same commands over an HTTPS API (POST https://exe.dev/exec). The owner has an account.
No git config to override. The published image (boldsoftware/exeuntu) and a live VM carry no credential helper and no insteadOf: the only git config at any level is init.defaultbranch=main in ~/.gitconfig, and there is no /etc/gitconfig. A VM's first-boot script is the user's own --setup-script.
Outbound HTTPS is open, so the broker at codaset.dev is reachable.
cp <vm> copies a VM, and copies its tags by default. Whether it copies memory or disk only is not documented; find out, because it decides what a copy inherits from the base.
exe.dev's own GitHub integration and its integrations generally go unused under OQ20. Its Identity Federation (VM-minted OIDC tokens) is per integration, not per VM, so the enrolment code stays the way a VM proves who it is.
It is ssh-shaped like boxd, so it tests the driver seam less hard than an SDK-transport provider would. Accepted.

What to build

A HostDriver for exe.dev through the portable derivation path (#252), so the agnosticism decision is tested against two real providers rather than one provider and a fixture.

Drive exe.dev's ssh API, wrapping its --json output in the driver's own result, as the boxd driver wraps boxd --json.
The base machine and deriveFromBase on exe.dev: derive with cp or a fresh VM, whichever is correct first; re-init machine identity; enrol after the copy; fetch and check out; serve.
A supervisor that starts the agent at boot with --screens on durable disk.
Driver traits declared honestly: transport kind, derivation speed and whether it carries live processes, disk durability, maximum lifetime, how exe.dev defines idle, egress to the broker, and machine identity.
A lifecycle shim if exe.dev's idle behaviour needs one (see Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257), or a recorded reason why it does not.

Acceptance criteria

A remote workroom can be created, used and destroyed on exe.dev from the app.
The same workroom behaves identically on boxd and on exe.dev, differing only in creation latency: terminal, Changes, History, the diff viewer, the Files tree, the badges, forwarded ports, and a push with no client attached.
Two workrooms derived from one base on exe.dev share neither an enrolment key nor an ssh host key nor /etc/machine-id.
After an exe.dev restart, the workroom's panes show their last screen.
The driver's traits, including what cp copies, are recorded in the design doc.

Blocked by

#252 (portable derivation on the container driver). The parity criterion also needs #256 (boxd driver).

Filed by Claude Code at the owner's request, 2026-09-27. Provider decided at the owner's request, 2026-09-27.

docs(remote-workrooms): mark #254, #255 and #257 as merged

🤖 Left by Claude Code (at the owner's request). Updated 2026-10-08 after /plan-eng-review: changes are marked (review).

Status (2026-10-08): built on branch feat/259-exedev-driver, not merged. Unit suite green; the exe.dev live parity suite passes all 7 cases, rerun after the switch to normal ssh (6a3ede0). The boxd live suite passes 8 of 9: testAnIdleBoxStillSleepsWithTheAppAttached fails: the app's own 10 s status poll keeps a connected box BUSY, so it never sleeps (#380). The same test passed on 2026-10-06, so it is a regression; this branch's boxd code behaves the same as master, and whether the cause is on master or on boxd's side is not yet bisected. The soak passed: an idle VM and a busy one ran 9 h with no sleep (same boot_id, no tick gap over 61 s), so sleepsWhenIdle stays false. Pool load is measured: two workrooms building at once share the plan's 2 vCPU, each build about 1.7 times slower than one alone (3715064). Left: the manual check of New Workroom › Remote › exe.dev from a Finder-launched app. Details: "As built (#259)" in docs/designs/remote-workrooms.md. Changes from this plan are marked (built) below.

Spike results (2026-10-08, live account, region `lon`)

Six throwaway VMs (four in the spike, two for the review's tag probe), all removed afterwards. Every claim below was measured, not read from the docs.

Derivation: cp is a cold copy of the flushed disk.

cp <base> <name> --json returns in 0.4-0.5 s ({"elapsed":0.42,…,"state":"starting"}). The copy answers ssh about 2 s later.
The copy keeps the base's disk only. A process running on the base (sleep) and a file in /dev/shm were both gone. The copy has its own boot_id.
Only flushed data is copied. A 50 MB file written without sync (48 MB still dirty in the page cache) was missing from the copy. So the derive must sync the base first, as on boxd. The base keeps running throughout.
The platform gives each VM its own identity at first boot. hostname equals the VM name. /etc/machine-id is new (written by exe-init at boot, same mtime as /etc/hostname). The gateway's per-VM sshd host key under /exe.dev/etc/ssh is new.
No reboot step is needed, unlike boxd: nothing from the base's memory survives the copy.

boxd.sh works unchanged on exe.dev (sudo sh -s -- exedev <socket> <screens> exits 0, both units active).

On a copy, workroom-identity saw the new hostname and minted within about 2 s. It wiped the copied broker.json and screens, and workroom-agent came up.
After restart, the marker matched, so the copy kept its machine-id, its broker.json and its screens.
The script's hostname-keyed marker holds on exe.dev, so it can become one shared systemd setup script.

The VM itself.

Ubuntu 24.04, x86_64, with systemd as PID 1.
The login user is exedev, with passwordless sudo.
/run is tmpfs and / is ext4, so the agent's state goes under /home/exedev/.local/state/workroom, as on boxd.
The default size is 2 vCPU, 8 GB of memory and a 25 GB disk, all drawn from the account's pool.
There is no stock /etc/ssh host key. exe.dev runs its own sshd (/exe.dev/bin/sshd, config /exe.dev/etc/ssh).

Transport.

ssh ends at exe.dev's gateway. exe.dev and every *.exe.xyz VM present one RSA key, SHA256:JJOP/lwiBGOMilfONPWZCXUrfK154cnJFXcqlsi6lPo (the fingerprint exe.dev's docs publish).
A locked-down config works: User exedev, IdentitiesOnly, and a known_hosts holding only *.exe.xyz <gateway key> with StrictHostKeyChecking yes.
Over that config, stdin EOF works (printf hello | ssh … wc -c printed 5), -tt gets a pty, and a remote exit status comes back (3 came back as 3).
whoami --json lists ssh_keys[].fingerprint, which matched ~/.ssh/id_ed25519.pub here.
(review) That key has a passphrase. Under a locked config (-F /dev/null, IdentitiesOnly yes) exe.dev answered Permission denied (publickey,keyboard-interactive) with neither the agent nor the Keychain. It authenticated with UseKeychain yes alone, with IdentityAgent $SSH_AUTH_SOCK alone, and with both.
The HTTPS API is not a transport: it caps each call at 30 s, with no stdin and no pty.

CLI shapes.

new --name workroom-<uuid> --json accepts the 45-character name. It returns vm_name and ssh_host.
ls --json returns {"vms":[…]}, each entry with vm_name, status (running), created_at, allocated_cpus, memory_capacity_bytes and disk_capacity_bytes.
restart --json returns {"status":"restarted",…}. The VM came back in about 4 s, with a new boot_id and its disk kept.
Not found:
- restart: {"error":"VM \"<name>\" not found"}, exit 1.
- cp: {"error":"vm \"<name>\" not found"} (lowercase vm), exit 1.
- rm exits 0 even for a VM that doesn't exist. It prints the error and then {"deleted":[],"failed":["<name>"]}, so destroy must read deleted/failed, never the exit status.
whoami --json has email, region and ssh_keys, and no user ID, so the email is the account check.
The CLI has no stop, start or idle timer.
(review) new --tag <t> tags a VM, ls --json lists tags when a VM has any, and cp --copy-tags=false gives a copy none.
(review) The account is on the Individual (Small) plan: max_cpus 2, max_memory_gb 8, max_vms 50, shared by every VM.

Idle. Three VMs sat idle for 40 minutes (12:55 to 13:35 BST), with the control plane polled every 60 s and no traffic to the VMs. All three read running throughout. Afterwards the base still had the same boot_id, its sleep process and its /dev/shm file, so it was never suspended or restarted. The CLI has no idle timer to set, and the docs say nothing about idle suspension. Not measured: anything longer than 40 minutes.

Decisions (owner, 2026-10-08)

Workrooms go in the account's pool, not --standalone.
(review) The driver authenticates with the user's own ssh credentials. (built, 6a3ede0) It runs the user's own ssh, as ssh exe.dev in Terminal: no -F and no key of its own, so the user's ~/.ssh/config, agent and Keychain pick the key. A second exe.dev account is a Host exe.dev block in ~/.ssh/config, as for any ssh, and the account check still refuses another account's VMs. (The first build chose a key itself, the first of ~/.ssh/*.pub exe.dev accepted under IdentitiesOnly; that could pick the wrong account's key, and the owner asked for ssh to behave as close as possible to normal ssh.) Nothing is added to the exe.dev account. A Workroom-registered key was considered and dropped: it adds an account-wide key, and the planned generic SSH driver (Generic SSH driver: remote workrooms on any VM or server over plain ssh #378) can't register keys on arbitrary servers, so both share this path.
(review) No HostPresenceDriver protocol yet: exe.dev VMs don't sleep, so it would have one adopter. boxd's sleep, presence and let-go code stays boxd-specific until a second sleeping provider exists.

Plan

1. Driver seam refactor (boxd only, no behaviour change, its own PR)

Rule: provider-specific behaviour lives on the driver, as traits or optional protocols. RemoteHosts asks only whether a host is a local container or a remote provider. No registry and no factory.

Today	Becomes
`DriverKey.isBoxd`, meaning "broker only, no relay, reports steps"	`isRemoteProvider` (anything not `.container`); add `.exeDev(account:)`
`clonePath`: `isBoxd ? BoxdHostDriver.user : user`	the login user comes from the key
`existingDriver(holding:)`, `runtimeIsMissing`, `environment`'s `gitHubToken`, AppStore's `codasetRequired` and step reporting (review)	"remote provider", so an exe.dev host is reached after a relaunch, never reads as missing without Docker, and gets no Mac `gh` token
`agentSocket(of:)`: `as? BoxdHostDriver` (review)	the socket and login user come from the driver key
`BoxdHostDriver.poll` (review)	moved to `extension HostDriver`, shared by both drivers; the check strings stay per driver
`driverName`: `runtime?.rawValue ?? boxdDriver`	a switch over the key
`Place` and the literal `[.remote(.boxd)]` in the picker and the sidebar	a `Place.remoteProviders` list
`HostDescriptor.org` / `account`	reused: exe.dev records `account` (the `whoami` email) and no `org`; no new fields
`Resources/host-setup/boxd.sh`	renamed to a shared systemd setup script (the spike shows it is provider-neutral)

(review) First commit: three table-driven tests in RemoteHostsTests over the boxd and exedev keys, shown to fail when their branch breaks: a remote-provider create without Codaset throws codasetRequired; a remote-provider environment has no gitHubToken; a remote-provider agent socket is the driver's own home-disk path. Then the refactor.

After this, a new provider means one driver file, one DriverKey case and one Place entry. A driver that can't provision (#378) answers create and deriveFromBase with notImplemented. The CLI helpers in BoxdHostDriver (cli, CLIFailure, decode) stay where they are: exe.dev's error and exit conventions differ (above), so there is nothing shared to extract yet.

2. `ExeDevHostDriver`

create: new --name workroom-<uuid> --tag workroom-base --no-email --json, then (built) wait for systemd to finish booting (the live suite found new returns, and ssh answers, while the VM still boots: the setup's systemctl failed with "Failed to connect to bus" on four of seven creates), run the setup script with sudo sh -s, then wait for the identity marker.
deriveFromBase: (review) first refuse a source whose ls --json doesn't list workroom-base, since every driver must refuse to derive from a workroom instance. Then sync on the base, then cp <base> <name> --copy-tags=false --json. Then wait for the identity marker (hostname = name) and for the agent. There is no reboot. A failure removes the copy by name.
destroy: rm <name> --json. It decodes the last JSON object on stdout and reads deleted/failed. It counts the not-found line (VM "<name>" not found, or vm lowercase from cp), matched whole, as already removed.
Account guard: check the whoami email before every create, derive, destroy and rollback, as boxd's checkOrg does.
Transport: the control plane runs as ssh exe.dev … --json, and each VM is reached as exedev@<name>.exe.xyz. (built, 6a3ede0) Both run /usr/bin/ssh over the user's own configuration with only what a background link needs set as -o (ExeDevHostDriver.sshOptions): BatchMode yes, no agent, X11 or port forwarding, no multiplexing, RemoteCommand none, RequestTTY no (a pane's -t outranks it), EscapeChar none, keepalives (Persistent remote agent: supervised serve, stdio relay and the container fixture #228), ConnectTimeout and LogLevel ERROR. They go through a new ContainerHostDriver.Route (options, destination, environment); container and boxd keep their own -F config. Moving boxd onto this path with Generic SSH driver: remote workrooms on any VM or server over plain ssh #378 is boxd driver: reach machines through the user's normal ssh, shared with exe.dev and the generic SSH driver #379.
- (review, built) The user's SSH_AUTH_SOCK (the app's, else the login shell's) is passed to ssh; the user's own IdentityAgent wins over it. Measured: with a bare environment, ssh signed in through launchd's agent and through the Keychain alone.
- (review, built) Host keys are the user's own known_hosts: a VM's key is checked against the exe.dev entry (HostKeyAlias exe.dev), since the gateway presents one key for both. Measured: without that entry, "Host key verification failed". Nothing is compared with the published fingerprint: a fingerprint built into the app would need an app update after a rotation.
- (review, built) When ssh can't sign in or verify, the create fails before anything is made, with ssh's own message and the fix: run ssh exe.dev once in Terminal (it must sign in without asking), naming the published fingerprint.

Traits:

Trait	Value
transport	`sshStdio`
`deriveSpeed`	about 5 s (built: copy, then the identity and agent waits)
`deriveCarriesLiveProcesses`	`false`
`durableDisk`	`true` (writes stream off-machine continuously)
`maxLifetime`	`nil`
`keepAwakeHoldsCredential`	`false`
`sleepsWhenIdle`	`false` (40 minutes measured; overnight soak before merge)

Lifecycle shim: none, and recorded as a reason in the design doc: a VM is never put to sleep while idle, so nothing needs keeping awake. The Far-side lifecycle shim: keep a busy box awake through the provider (boxd first) #257 heartbeat is harmless here.
(review) Before merge: an overnight soak (8 h or more) of one idle VM and one running a detached CPU job with no client, polling the control plane only. If either sleeps, sleepsWhenIdle flips and the shim question reopens.
Debug builds: fix(macapp): Debug builds talk only to a local Codaset; requests name the build #278's broker reverse forward runs over the agent's service connection (ReverseForwardRegistry), not ssh -R, so it needs nothing per provider.

3. App wiring

New Workroom › Remote › exe.dev. It is unavailable when signed out of Codaset; a create checks whoami against the host's account.
Credentials: broker only. Progress steps: shown during create, as for boxd.

4. Tests

ExeDevHostDriverTests: a fake runner. It covers JSON decoding, the rm with exit 0 and a failed list, both not-found spellings, (built) the ssh it runs (no -F or key, the forced -o set, HostKeyAlias, the agent socket on a pane's attach), the error when ssh can't sign in, refusing an untagged source, sync before cp, and rollback.
Parity: make BoxdIntegrationTests run per provider (WR_EXEDEV_TESTS=1 beside WR_BOXD_TESTS=1, the same cases) instead of copying it. It covers create, use and delete; distinct identities; restart keeping the last screens; and a push with no client attached. The boxd-only keep-awake cases stay boxd-only.
- (review) The identity probe takes each provider's sshd host key path (/etc/ssh on boxd, /exe.dev/etc/ssh on exe.dev), and asserts the gateway key separately as the one key shared on purpose.
- (review) exe.dev gets its own failure-at-each-step case: cp, the identity wait, the agent wait.
- (review) During the parity run, measure a base plus two workrooms building at once against one alone, and record the result with the plan tier.

5. Docs

Add an "As built (exe.dev driver: the second real provider #259)" entry to the design doc, with the traits table, what cp copies (an acceptance criterion), and the soak and load results.
Add a line under Provider Decision noting that the pool objection still stands: the Personal pool is shared by the base and every workroom.

Acceptance mapping

Criterion	Covered by
Create, use and destroy from the app	step 3 and the parity suite
Identical to boxd except creation latency	the parity suite
No shared enrolment key, ssh host key or `/etc/machine-id`	the platform mints `machine-id` and the gateway host key; `workroom-identity` wipes enrolment (spike); the parity suite checks `/exe.dev/etc/ssh`
Panes show their last screen after a restart	the parity suite (`restart`)
Traits recorded, including what `cp` copies	step 5

Not in scope (review)

A HostPresenceDriver protocol, until a second provider sleeps.
Sweeping exe.dev VMs a crash leaves unrecorded: TODOS.md (P3), with boxd: sweep machines a crashed create left running #373.
The generic SSH driver: Generic SSH driver: remote workrooms on any VM or server over plain ssh #378.

test(macapp): exe.dev error text, a VM left behind, a refused key, th…

fix: pre-landing review fixes (#259)

fix(macapp): exe.dev's account commands get the login shell's ssh age…

docs: a second exe.dev account is a Host exe.dev *.exe.xyz block (#259)

fix(macapp): an exe.dev VM's spawned ssh gets the login shell's PATH (#…

docs: exe.dev's ssh agent and PATH rules, the provider tooltip and st…

test(macapp): pin the remote-provider credential and socket rules bef…

refactor(macapp): a driver seam for remote providers, boxd only, no b…

fix(macapp): an exe.dev base waits for systemd, and a failed listing …

test(macapp): one parity suite for every real provider, run on boxd a…

docs(remote-workrooms): as built for the exe.dev driver (#259)

perf(macapp): a new exe.dev driver signs in with the key found before (…

fix(macapp): exe.dev runs the user's own ssh, not a key Workroom picks (

docs(remote-workrooms): the boxd live suite after the #259 refactor, …

chore(macapp): exe.dev comments match the user's own ssh; drop an unu…

docs(remote-workrooms): exe.dev soak and pool load results (#259)

test(macapp): exe.dev error text, a VM left behind, a refused key, th…

fix: pre-landing review fixes (#259)

fix(macapp): exe.dev's account commands get the login shell's ssh age…

docs: a second exe.dev account is a Host exe.dev *.exe.xyz block (#259)

fix(macapp): an exe.dev VM's spawned ssh gets the login shell's PATH (#…

docs: exe.dev's ssh agent and PATH rules, the provider tooltip and st…

docs: the exe.dev seam note after #380 removed the let-go (#259)

Navigation Menu

Repository navigation

exe.dev driver: the second real provider #259

Parent

The provider: exe.dev (decided 2026-09-27, owner)

What to build

Acceptance criteria

Blocked by

Spike results (2026-10-08, live account, region `lon`)

Decisions (owner, 2026-10-08)

Plan

1. Driver seam refactor (boxd only, no behaviour change, its own PR)

2. `ExeDevHostDriver`

3. App wiring

4. Tests

5. Docs

Acceptance mapping

Not in scope (review)

20 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

exe.dev driver: the second real provider #259

Description

Parent

The provider: exe.dev (decided 2026-09-27, owner)

What to build

Acceptance criteria

Blocked by

Activity

joelmoss commented on Oct 8, 2026

Spike results (2026-10-08, live account, region lon)

Decisions (owner, 2026-10-08)

Plan

1. Driver seam refactor (boxd only, no behaviour change, its own PR)

2. ExeDevHostDriver

3. App wiring

4. Tests

5. Docs

Acceptance mapping

Not in scope (review)

20 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Spike results (2026-10-08, live account, region `lon`)

2. `ExeDevHostDriver`