diff --git a/AGENTS.md b/AGENTS.md index 52525d357..944b1a94d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,6 +38,16 @@ Go uses `testing` with `*_test.go` files and `Test…` functions. Swift uses XCT Use concise, descriptive commits. History commonly uses `fix(macapp): …`, `refactor(macapp): …`, and `docs(…): …`; plain imperative subjects also occur. PRs should explain behavior changes, link issues (`Fixes #177`), report validation and skipped checks, and include screenshots for visible UI changes. +## Reviews & Reported Bugs + +A finding from a review, a reviewer bot or an agent is not a bug until it is checked against how the app is actually used, or could plausibly be used. Before fixing or filing it, answer three questions: + +- **What triggers it?** Name the real sequence: what the user does, what the environment does, and what timing it needs. +- **How likely is that?** It needs a realistic path in ordinary use, or a failure that does happen (an outage, a quit mid-operation, a corrupt config). A race that needs a click inside one actor hop, or input the code never produces, does not qualify. +- **What does it cost when it happens?** Lost or corrupted data, a paid resource left running, a broken or misleading UI, or nothing that persists. + +Fix it only when it is likely enough to happen, or costly enough when it does. Otherwise reply on the thread with that reasoning and close it. File an issue only when a future change could make it reachable. Comment wording, test-timing hypotheticals and style suggestions never justify another review round. + ## Architecture & Configuration Preserve the CLI `--json` contract; breaking changes require a `schema_version` bump in `cmd/json.go`. Read `CONTRIBUTING.md` and [repository notes](docs/repository-notes.md) for architecture and operations. For app work, also follow [macapp/AGENTS.md](macapp/AGENTS.md). Never use personal workrooms as destructive test fixtures. diff --git a/TODOS.md b/TODOS.md index f54d25fce..d53c2550b 100644 --- a/TODOS.md +++ b/TODOS.md @@ -7,99 +7,30 @@ ## P0 — before the next release -## P1 — before GA - -### A second Mac can find a remote workroom and reach its host (macapp, Codaset) — #255 blocker - -**What:** Let a Mac that did not create a remote workroom discover it and get an ssh key its host -accepts. - -**Why:** #255 (cross-machine reattach) cannot be tested on a real host until this exists. Today a -remote workroom is listed only in the creating Mac's CLI config, and its host accepts only that -Mac's key (`RemoteWorkrooms.swift` mints its `id_ed25519` once per Mac per build). Container -workrooms on one Mac's Docker are unreachable from another Mac at all, so the live test runs on -boxd. Design: `docs/designs/oq8-cross-machine-reattach.md`, premise 10. - -**How to start:** The Codaset broker already keys grants by `workroomID` per user (#250, #251). -List the signed-in user's grants to discover workrooms, and enrol this Mac's public key on the host -through the broker rather than by hand. - -**Depends on:** #250 and #251 (done). Blocks #255's live acceptance test. +### `AgentWakefulnessTests.testAReplyCrossingAKeepDoesNotResurrectTheCard` flakes (macapp) — #216 test -**Priority:** P1 (chosen in the #255 eng review, 2026-10-05) +**What:** Find why the crossing-`keep` test sometimes fails and fix the test or the race it exposes. -### Before wiring boxd into the app: a wake-on-connect resets the awake ceiling (wr-agent) — #257 review +**Why:** It failed once in a targeted `/ship` QA run on `feat/boxd-in-app` (2026-10-07, 2 of 80 +`AgentWakefulnessTests` red: "the failure brought the retry back", `XCTAssertTrue failed` at the +retry-flag assert), then passed in every full suite after it (3267/0, four runs). The test came with +#216 on master; the boxd branch didn't touch the keep/retry path. A flake in the prompt card's retry +logic can hide a real resurrection bug, which is what the test exists to catch. -**What:** Decide what a resume does to the awake ceiling once a remote host can sleep, before -`BoxdHostDriver` is wired into `RemoteHosts`. +**How to start:** Loop it alone (`make app-test "APP_TEST_FLAGS=-only-testing:WorkroomAppTests/AgentWakefulnessTests/testAReplyCrossingAKeepDoesNotResurrectTheCard"`) +under CPU load until it fails. Then check whether the crossing reply and the failed `keep` can land in +either order on the main actor; the assert assumes one. -**Why:** `Ceiling::resumed()` resets `busy_since` and the state to Below on every resume, which was -harmless while nothing held a box awake. With the #257 heartbeat it is not: a box whose ceiling -prompt went unanswered sleeps with its job still running; the next connect wakes it (the status -sweep connects to every remote workroom); the ceiling restarts; the box reads BUSY again; and the -heartbeat holds it for another full ceiling. So "ask, then sleep" becomes "awake in 4 h chunks" on -a paid box. Today no host that sleeps is wired in (containers never sleep), so nothing is exposed. - -Three more things the same reviews found, all moot until a host that sleeps is wired in: - -- **A base gets ask mode with nobody watching.** Every `AgentBootstrap.connect` sends - `ask && sleepsWhenIdle`, including provisioning's base connects, but only `RemoteHosts.connect` - starts a prompt watch. A long base job past the ceiling would sleep unasked. Send `ask: false` - from connects that start no watch. -- **An idle window under 60 s defeats the heartbeat silently.** boxd's auto-suspend is - user-configurable; read the machine's timers at connect and warn below ~90 s. -- **Prompt cards and stale verdicts.** Many hosts asking at once stack cards with no scroll, so one - can be pushed off-screen while its deadline runs; and a service that stalls after publishing IDLE - shows "idle", not unprotected (the badge's stall rule needs `busy`). On a sleeping host treat a - stalled reading as unknown, and bound the stack. +**Depends on:** nothing. -**How to start:** Keep the ceiling's awake time across a resume when the box was `Suppressed`, or -make a resume after `Suppressed` stay suppressed until the user answers. Pin it with a -`new_settings_apply_from_the_next_tick`-style table in `wakefulness/tests.rs`. +**Priority:** P0 (chosen in /ship for #356, invocation 4, D6, 2026-10-07) -**Depends on:** wiring `BoxdHostDriver` into `RemoteHosts` (#356). Land it with the -`BoxdIntegrationTests` keep-awake case (P2, part of #356). +## P1 — before GA -**Priority:** P1, gating boxd in the app (chosen in /ship's adversarial review of #257, 2026-10-06) +None open. The second-Mac discovery entry was dropped on 2026-10-07 (#348, closed as not needed). ## P2 — perf, correctness, and the next VCS phase -### Put the keep-awake acceptance run in `BoxdIntegrationTests` (macapp) — part of #356 - -**What:** Encode #257's live acceptance as a gated case in -`macapp/WorkroomAppTests/BoxdIntegrationTests.swift`, so one command reruns it. Two machines: - -- **Control** (no agent): proves the job alone does not keep the box awake, which is what lets the - agent's result mean anything. -- **Agent**: the claim under test. - -The hand run's third machine (ask mode, nobody answering) is dropped: "an unanswered prompt stops -the heartbeat" is pinned by unit tests that run on every push, so the live case only re-proved -boxd's timer. - -**Why:** It ran once by hand on 2026-10-06 and passed (results in "As built (#257)" in -`docs/designs/remote-workrooms.md`). Nothing reruns it, so a boxd change to how it counts idle -network time would quietly let busy boxes sleep. That costs nothing while boxd is not wired into the -app, and costs users' work once it is. exe.dev (#259) needs the same proof. - -**How to start:** Two throwaway machines with 120 s suspend and hibernate timers -(`--auto-suspend-timeout=120 --auto-hibernate-timeout=120`; `BoxdHostDriver.create()` takes no -timers, so the case calls the CLI or the driver gains a test-only option). Run the same 6-minute -job, with no network use, on each: the control with no agent, and the agent supervised with -`--idle-timeout never`. Poll each machine's state every ~20 s from the Mac, and have the job log -its ticks on the box. Pass: the control hibernates mid-job; the agent's machine logs every tick and -hibernates within the idle window after the job ends. Destroy both, whatever the outcome (the -derive tests' `remove(machine:)` pattern). About 15 minutes per run. - -**Depends on:** wiring `BoxdHostDriver` into `RemoteHosts` (#356). Runs only with -`TEST_RUNNER_WR_BOXD_TESTS=1`, a `WR_AGENT_LINUX=1` build and the sandbox off, like the other -cases there. - -**Priority:** build it with the boxd wiring (#356; eng review decision D10, 2026-10-06), and run it -before any release that ships boxd. Moved out of P1 on 2026-10-06 because nothing that sleeps is -wired in yet; originally deferred from plan `~/.gstack/projects/joelmoss-workroom/master-eng-review-20261005-233111.md` -(T6), at /ship of #257. - ### Give the keep-awake heartbeat its own thread (wr-agent) — #257 review **What:** Send the heartbeat from a thread of its own, fed the published verdict, instead of from diff --git a/docs/designs/remote-workrooms.md b/docs/designs/remote-workrooms.md index ba3a90e76..c3992a6cf 100644 --- a/docs/designs/remote-workrooms.md +++ b/docs/designs/remote-workrooms.md @@ -1369,8 +1369,12 @@ these are the subsystems that actually gate "a remote workroom is a real workroo instance. If undoing a step fails too, `Failure.rollbackIncomplete` names the host and grant still live, for the caller to record and finish, including a grant the enrolment itself could not cancel (`AgentEnrolment.GrantStillLive`); `destroy` reports the same way. The - driver's own failed cleanup is `HostDriverError.leftBehind`, naming what is still there, and - destroying a host twice succeeds, so a caller can retry what came after it. A failed grant cancel does not keep the box: + driver's own failed cleanup is `HostDriverError.leftBehind`, naming what is still there (and, + on boxd, the host it made), and destroying a host twice succeeds, so a caller can retry what + came after it. A create that leaves a machine running, from the driver's undo or a build's own + failed rollback, records it `failed`: a workroom's on its entry, a project's first base on the + project, which then refuses a second base until the project is deleted (#370). A boxd delete + also removes the derive snapshot named after the machine, which a failed derive can leave. A failed grant cancel does not keep the box: the key that would mint against the grant goes with it. An instance is never derived from, because its disk holds its key and credential helper. The workroom branches from `refs/remotes/origin/HEAD` after `git remote set-head origin --auto`, since `fetch` never @@ -1892,6 +1896,71 @@ these are the subsystems that actually gate "a remote workroom is a real workroo its own. An IPv6-only box has no IPv4 default route and reports the error, and so does a default route with no gateway (`default dev wg0`), which a provider VM is not known to have. A changed setting reaches a remote agent on the app's next connect, not while connected. +- **As built (#356, boxd in the app).** The app now makes, opens and deletes remote workrooms on + boxd, in Nightly and Dev builds behind `remoteWorkroomsPreview`, from New Workroom › Remote › + boxd. Eng review 2026-10-06 (decisions D1-D11). + - **One driver table.** `RemoteHosts.DriverKey` is an enum over a container runtime (with its + Docker context) and a boxd org and account, and `RemoteHosts` holds any `HostTerminalDriver`. + Before, a boxd host had no key: a delete fell back to Docker, adopt dropped it, and a missing + container CLI read as its host being gone. A descriptor naming an unknown driver is refused, + never taken down with another. A boxd host records `driver: "boxd"`, its `org` and its + `account` (`boxd auth`'s `user_id`); the CLI keeps both as opaque fields. + - **Account, not only org.** Every personal account's org is nil, so after an account switch the + org check passed and a machine read as "not found" was taken for gone. The driver now checks + the account too before every create, derive, destroy and rollback. + - **Credentials.** Broker only (OQ20): a boxd workroom never borrows the Mac's `gh`, so the menu + asks for Codaset, and a create without it stops before anything is made. Whether boxd itself is + signed in is a CLI call, so the create checks it, naming `boxd auth login`. + - **Sleeping boxes.** An ssh login wakes a box, so a background read of a boxd host asks boxd + first (`machine get`, `standby` or `hibernated`) and leaves an asleep box alone; its row shows a + moon. Opening the workroom wakes it. boxd is asked at most once per host per 30 s about a box it + called asleep. When boxd cannot say (no CLI, a timeout, output the app does not expect), the + read leaves the box alone too and logs why, so a broken CLI cannot wake every box. + - **The ceiling across a wake.** A resume after an unanswered prompt carries the awake time from + before the sleep, so the agent asks again at once rather than granting a running job another + whole ceiling; the resume mask's IDLE ticks are ignored by the ceiling. Provisioning's connects + send the agent no settings (its `ask: false` could race a watched connection's `ask: true`), the + badge warns when a box's boxd idle timer is under 90 s, host prompt cards scroll past two, and a + stalled IDLE reading on a sleeping host shows "status unknown". + - **Not swept.** Unrecorded boxd machines are left alone: names carry no build, so one build's + sweep would take another's live workrooms (#284 has the boxd case). + - **Letting go of an idle box.** boxd's idle meter counts inbound traffic, and the first live run + (2026-10-06) found an attached box never slept: the service connection's ssh keepalives (15 s) and + the badge's polls (10 s) held it, idle or past an unanswered prompt. So a boxd host whose agent + reports IDLE (its published verdict, which an unanswered prompt also makes IDLE), whose workroom + is not selected, which forwards no port and has no pane attached is let go of: its connection + closed, no background read reconnecting it, its badge sending nothing, and its row showing "idle, + not connected" (an open moon) until boxd confirms it asleep. Selecting or opening the workroom, a + click (closing a pane, Keep awake), a pane attaching, or a port forward or BUSY reading that + arrives while the app is letting it go, takes it back. A remote pane's own ssh + keeps its box awake whatever the app does, so a host with a pane attached is never let go of; a + pane restored at launch attaches only once it is shown, and until then holds nothing. The app + polls each connected boxd host itself, so a row scrolled out of view still lets its box go; it + ignores IDLE readings for 10 s after a connect, so the read that connected the host is answered + first; and a host whose agent's wakefulness service has stopped is let go of too, as nothing on it + holds the box awake, unless its last reading was BUSY (the connection is then all that keeps the + job's box awake): that reading never changes and nothing will ask about it, so it holds the box + for one ceiling at most. A stalled service's reading, and a read that reached the agent and + failed, hold the box the same way: for one ceiling from the first such reading (a trusted reading + starts the count again), then it is let go. A read refused before it left, as for a host already + let go of, counts for nothing, and a connect starts a new count. The box's idle window is read on + every connect: a failed read keeps the last one, and an answer with no timer set clears it, so the + badge stops warning about a timer that was turned off. A host let go of is asked about again once + its boxd idle window (plus 30 s) has passed: if boxd says it is still awake, something on the box + holds it, so the app reconnects and hears its ceiling prompts. + - **Limit: selected workrooms and attached panes.** The selected workroom's host is never let + go of, nor is one with a pane attached, and a pane's own ssh holds its box awake regardless. + So a job past its ceiling, with its prompt left unanswered, keeps the box awake (and billed) + while its workroom is selected or any of its panes is attached. A pane counts as attached only + while its ssh lives: one whose ssh exited, or whose window closed, holds nothing, and a + reconnect attaches it again. The prompt's "let it sleep" takes effect only once that + workroom's panes are closed and another is selected. + - **Live runs (2026-10-06).** `BoxdIntegrationTests`, all nine cases, pass: the app's create and + delete (base in `/home/boxd`, records naming boxd with org and account, nothing left), #257's + keep-awake acceptance (the control slept mid-job, the agent's box logged every tick then + slept), and, after the change above, both app-attached checks: an idle box slept (the whole case, + machine creation included, took 318 s), and a box past a 60 s ceiling with its 30 s prompt unanswered slept mid-job. `machine + get --json` carries `status` and the timers as strings (`"auto_suspend": "120s"`). ## Phase 0 Results @@ -2594,8 +2663,8 @@ disagreement passes every test on either side alone while presenting as an empty two open Macs is not built. *Not covered.* Local workrooms keep their layouts in `session.json` (TODOS, P3). A second Mac - still has to find the workroom and hold a key its host accepts (TODOS, P1), so the live - two-Mac test runs on the ssh fixture for now. + finding the workroom and holding a key its host accepts is not planned (#348, closed + 2026-10-07), so the two-Mac test runs on the ssh fixture only. 9. **What machine identity does a fork need re-minted?** **ANSWERED — Phase 0 item 6.** Measured duplicated across two live forks: `/etc/machine-id`, **`/proc/sys/kernel/random/boot_id`** (not on the list below, and it should be), both ssh host keys, and any on-disk secret. Re-minted by boxd: hostname and IP. Live processes carry across with their original pids. Not session UUIDs — those are client-minted and a template base has no sessions. The real list is ssh host key, machine-id, the agent's instance/socket identity, and any base-resident pty. Phase 0 item 6 measures which of these @@ -2938,10 +3007,10 @@ disagreement passes every test on either side alone while presenting as an empty idle policy would otherwise have stopped it. That is the wakefulness service doing its job. - Start a long-running agent in it, quit Workroom entirely, reopen hours later, and land in the same session with the alternate-screen program correctly repainted. -- Do the same from a **different Mac**. Open question 8 is answered and built (#255): the - layout and the sessions' workrooms live on the host, proven with two clients on the ssh fixture. - What still gates it on a real host is a second Mac finding the workroom and holding a key the - host accepts (TODOS, P1). +- ~~Do the same from a **different Mac**.~~ **Removed (2026-10-07):** dropped from the gate + (#260), together with the second-Mac discovery work it needed (#348, closed as not needed). Open + question 8 is still answered and built (#255): the layout and the sessions' workrooms live on the + host, proven with two clients on the ssh fixture. - Close the laptop; the remote agent completes a `git fetch` and a `git push` with no client attached. (The concrete test that premise 6's revision was necessary.) - Changes, History, the diff viewer and the Files tree all work against a remote workroom and agree diff --git a/macapp/AGENTS.md b/macapp/AGENTS.md index dc2b5c4a1..81b51e2a9 100644 --- a/macapp/AGENTS.md +++ b/macapp/AGENTS.md @@ -221,11 +221,16 @@ CLI manages in `~/.ssh/config` and `~/.ssh/known_hosts` (`SSHDetails`). A base i `Resources/host-setup/boxd.sh` (an identity unit and the agent's supervisor, both systemd). A derive is snapshot, restore, then reboot, and the reboot is not optional: a restored snapshot runs the base's processes until it happens. The CLI acts in boxd's active org, so the driver -is told its org (`Configuration.org`) and refuses to act while another is active. Machines and -snapshots are named +is told its org and account (`Configuration.org`, `Configuration.account`; every personal +account's org is nil, so the org alone cannot tell two accounts apart) and refuses to act while +another is active (#356). Machines and snapshots are named `-`, so cleanup and a relaunched app find them by ID. `BoxdIntegrationTests` make real machines: they run only with `TEST_RUNNER_WR_BOXD_TESTS=1`, a `WR_AGENT_LINUX=1` build, -and the sandbox off. See "As built (#256)" in `docs/designs/remote-workrooms.md`. +and the sandbox off. The app reaches it from New Workroom › Remote › boxd (#356): `RemoteHosts` +holds any driver by `DriverKey` (a container runtime or a boxd org and account), a background read +of a boxd host asks boxd first and leaves an asleep box asleep, and an idle host's connection is +let go of so boxd can sleep it. See "As built (#256)" and "As built (#356)" in +`docs/designs/remote-workrooms.md`. ## Remote workrooms in config (#249) diff --git a/macapp/WorkroomApp/Core/AgentPortForwarding.swift b/macapp/WorkroomApp/Core/AgentPortForwarding.swift index 0d6e483c8..88323972b 100644 --- a/macapp/WorkroomApp/Core/AgentPortForwarding.swift +++ b/macapp/WorkroomApp/Core/AgentPortForwarding.swift @@ -908,6 +908,11 @@ final class PortForwardingModel: ObservableObject { /// forwards with its connection (`watch()`), leaving an empty model behind. private static var models: [HostID: PortForwardingModel] = [:] + /// Whether `host` has a forward up: its connection is carrying it, so it must not be dropped. + static func hasForwards(_ host: HostID) -> Bool { + models[host]?.forwards.isEmpty == false + } + static func model(for host: HostID) -> PortForwardingModel { if let model = models[host] { return model } let model = PortForwardingModel(host: host, transport: .live(host)) diff --git a/macapp/WorkroomApp/Core/AgentWakefulness.swift b/macapp/WorkroomApp/Core/AgentWakefulness.swift index b9f00a517..4b5ba0017 100644 --- a/macapp/WorkroomApp/Core/AgentWakefulness.swift +++ b/macapp/WorkroomApp/Core/AgentWakefulness.swift @@ -110,22 +110,38 @@ struct AgentWakefulness: Decodable, Sendable, Equatable { || (running && busy && keepAwake == nil) } - /// What the badge shows. Only these four, because only these four are actionable: the classifier's - /// raw verdict and the CPU cost are diagnostics. - enum Display: Equatable { case idle, busy, busyPastCeiling, busyUnprotected } + /// What the badge shows. Only these, because only these are actionable: the classifier's raw + /// verdict and the CPU cost are diagnostics. `unknown` is a stalled service whose last reading + /// was IDLE, on a host that sleeps (#356): that reading may be stale while work runs, and nothing + /// is keeping the box awake either way. + enum Display: Equatable { case idle, busy, busyPastCeiling, busyUnprotected, unknown } /// What the badge shows for a host that `sleeps` when idle or not. One that never sleeps (a /// container) cannot be slept under a job, so nothing there is "not kept awake": an old agent, /// a stalled one or a failing heartbeat would only be a false alarm, and its advice (restart the /// agent) would end the user's sessions. - func display(hostSleeps sleeps: Bool) -> Display { + /// + /// On a host its provider sleeps after `idleWindow` seconds idle on the network, when known + /// (#356), a window shorter than the heartbeat can beat leaves busy work unprotected however + /// healthy the heartbeat is. + func display(hostSleeps sleeps: Bool, idleWindow: TimeInterval? = nil) -> Display { + if sleeps, busy, Self.isTooShort(idleWindow) { return .busyUnprotected } if sleeps { return display } if awakeCeilingExceeded { return .busyPastCeiling } return busy ? .busy : .idle } + /// The heartbeat sends once a minute (`heartbeat.rs`), so a provider that sleeps a box after + /// less idle time than this can sleep it between two sends. + static let minimumIdleWindow: TimeInterval = 90 + + static func isTooShort(_ idleWindow: TimeInterval?) -> Bool { + idleWindow.map { $0 < minimumIdleWindow } ?? false + } + var display: Display { if unprotected { return .busyUnprotected } + if stalled == true { return .unknown } if awakeCeilingExceeded { return .busyPastCeiling } return busy ? .busy : .idle } @@ -504,6 +520,9 @@ final class WakefulnessModel: ObservableObject { /// Whether this model's host is put to sleep when idle (`HostDriverTraits.sleepsWhenIdle`), set /// from its driver on connect. Assumed until then; nothing shows before a connection anyway. var hostSleeps = true + /// How long the host's provider lets it sit idle on the network before sleeping it, in seconds, + /// read on connect where the provider says (boxd, #356); nil when unknown. + var idleWindow: TimeInterval? init(transport: Transport = .live, host: UUID? = nil) { self.transport = transport @@ -532,7 +551,8 @@ final class WakefulnessModel: ObservableObject { if let model = Hosts.shared.models[id] { return model } let model = WakefulnessModel( transport: .on(.remote(id), manager: .shared) { - try await RemoteHosts.shared.ensureConnected($0) + // Only "Keep awake" connects: a click, which wakes a box let go of or asleep (#356). + try await RemoteHosts.shared.ensureConnected($0, wake: true) }, host: id) Hosts.shared.models[id] = model @@ -552,7 +572,9 @@ final class WakefulnessModel: ObservableObject { /// The host is gone (deleted): its model, its watch and any prompt card with it. static func forgetHost(_ id: UUID) { - Hosts.shared.models.removeValue(forKey: id)?.stopWatchingPrompts() + let model = Hosts.shared.models.removeValue(forKey: id) + model?.stopWatchingPrompts() + model?.stopPollingWhileConnected() Hosts.shared.showing.remove(id) } @@ -595,8 +617,9 @@ final class WakefulnessModel: ObservableObject { } /// Modest on purpose. The verdict changes on a 30 s hysteresis window, so anything faster only - /// costs round trips, and this runs only while something is showing the result. - static let pollInterval: Duration = .seconds(10) + /// costs round trips, and this runs only while something is showing the result, or while a boxd + /// host is connected (`pollWhileConnected`). + nonisolated static let pollInterval: Duration = .seconds(10) private var pollers = 0 private var pollTask: Task? @@ -613,17 +636,24 @@ final class WakefulnessModel: ObservableObject { /// `prompt_pending: false` cannot undo a newer reply's raise. private var requestsIssued = 0 private var newestApplied = 0 + /// When this host's readings stopped being ones to trust (#356): its service stopped, stalled, or + /// a `status` that reached its agent failed. Such a reading holds the box for at most one ceiling + /// from then, as nothing on the box will ask the user about it. A connect starts a new count. + private var untrustedSince: ContinuousClock.Instant? + /// The ceiling the last applied reply named, for a failed read, which names none. + private var lastCeiling: TimeInterval? /// Polls for as long as the caller's task lives. Driven by a SwiftUI `.task`, so closing the /// inspector, the card or the window ends it — there is no polling while nothing is showing the - /// result. One loop however many callers: N windows with the inspector open are N callers of a - /// single poll, not N polls stomping one `status` (one's timeout would blank every window's badge). + /// result, except for a connected boxd host, which `pollWhileConnected` polls (#356). One loop + /// however many callers: N windows with the inspector open are N callers of a single poll, not N + /// polls stomping one `status` (one's timeout would blank every window's badge). /// /// A cancelled poll leaves the last verdict standing rather than clearing it. Clearing on exit /// read as tidy and was wrong in two ways: this model is shared by every window, so one window /// closing its inspector blanked the others' badge; and the value is only ever rendered by a badge /// whose own `.task` refreshes it before its first sleep, so a stale value cannot be displayed. - /// A FAILED poll still clears it — that assignment is the `try?` in `refresh`. + /// A FAILED poll still clears it, in `observe`'s failed-read branch. func poll() async { pollers += 1 if pollTask == nil { @@ -672,21 +702,73 @@ final class WakefulnessModel: ObservableObject { issued: (generation: Int, request: Int), status: @Sendable () async throws -> AgentWakefulness ) async -> AgentWakefulness? { - let next = try? await status() + let next: AgentWakefulness? + var reachedAgent = false + do { next = try await status() } catch { + next = nil + // A read refused before it reached the agent (the host let go of, asleep, not connected, the + // request refused here, or its connection replaced) says nothing about the box, so it + // neither holds it nor starts the hold's clock. + switch error { + case is RepositoryRoutingError, HostConnectionError.notDispatched, + HostConnectionError.staleGeneration, HostConnectionError.mismatchedContext: + reachedAgent = false + default: reachedAgent = true + } + } guard !Task.isCancelled else { return next } guard let next else { - if issued.request == requestsIssued, self.status != nil { self.status = nil } + guard issued.request == requestsIssued else { return nil } + if self.status != nil { self.status = nil } + // A failed read says nothing about a job, so it holds the box, for one ceiling at most. + if reachedAgent { report(busy: holding()) } return nil } guard issued.request > newestApplied else { return next } newestApplied = issued.request if next != self.status { self.status = next } + // Only a usable ceiling is taken: garbage (negative, not finite) is dropped, so the hold keeps + // the last one or the setting rather than ending at once, and a huge one is capped, as every + // wire duration here is, since `Duration.seconds` traps out of range. + if let ceiling = next.ceilingSeconds, ceiling.isFinite, ceiling >= 0 { + lastCeiling = wakefulnessSeconds(ceiling) + } + // A stalled service's reading may be stale while work runs (`display` shows it as unknown), so + // it holds the box. One not running at all holds nothing awake, heartbeat included, so its box + // is let go of too, unless its last reading was BUSY: a service that died under a job leaves + // the connection as the one thing keeping that job's box awake. Neither reading changes and + // nothing will ask about it, so each holds the box for one ceiling at most. + if next.running && next.stalled != true { + untrustedSince = nil + report(busy: next.busy) + } else { + report(busy: holding() && (next.running || next.busy)) + } if issued.generation == promptGeneration { prompt.reconcile(next, now: Date()) } return next } + /// Whether a reading the app can't trust still holds the box: for its last known ceiling (the + /// one in preferences, which the app hands the agent, before any is known) from the first such + /// reading. + private func holding() -> Bool { + // A monotonic clock: moving the Mac's clock must neither cut the hold short nor stretch it. + let now = ContinuousClock.now + let since = untrustedSince ?? now + untrustedSince = since + return since.duration(to: now) + < wakefulnessDuration(lastCeiling ?? AgentWakefulnessSettings.current.ceiling) + } + + /// A remote box that is not busy may be let go of, so the app's own traffic stops holding it + /// awake (#356); `RemoteHosts` decides whether to (`observed`). + private func report(busy: Bool) { + guard let host else { return } + Task { await RemoteHosts.shared.observed(.remote(host), busy: busy) } + } + private var watchTask: Task? /// Starts the single ceiling-prompt watch. Idempotent, and deliberately NOT bound to any view's @@ -701,6 +783,28 @@ final class WakefulnessModel: ObservableObject { var isWatchingPrompts: Bool { watchTask != nil } + /// A boxd host's poll while its connection is up (#356), started by `RemoteHosts.connect`. An + /// IDLE reading is what lets the box go, and the badges poll only while they are on screen, so + /// without this a box whose row was scrolled away stayed connected, and awake, until quit. + /// ponytail: runs until the host is let go of or forgotten; a connection that drops otherwise + /// leaves it failing one local request every `pollInterval` until the next connect. + private var connectionPoll: Task? + + /// Starts the connection poll. `connected` is true from a connect, which is new evidence, so it + /// also restarts the untrusted hold's clock; a let-go taken back only resumes the poll. + func pollWhileConnected(connected: Bool = true) { + if connected { untrustedSince = nil } + guard connectionPoll == nil else { return } + connectionPoll = Task { [weak self] in await self?.poll() } + } + + var isPollingWhileConnected: Bool { connectionPoll != nil } + + func stopPollingWhileConnected() { + connectionPoll?.cancel() + connectionPoll = nil + } + func stopWatchingPrompts() { watchTask?.cancel() watchTask = nil diff --git a/macapp/WorkroomApp/Core/AppStore.swift b/macapp/WorkroomApp/Core/AppStore.swift index 613b3abe4..583a6ac71 100644 --- a/macapp/WorkroomApp/Core/AppStore.swift +++ b/macapp/WorkroomApp/Core/AppStore.swift @@ -147,6 +147,8 @@ final class AppStore: ObservableObject { deinit { for token in frameObservers { NotificationCenter.default.removeObserver(token) } + // A closed window no longer holds its workroom's host connected (#356). + RemoteHosts.shared.select(nil, in: ObjectIdentifier(self)) } /// Only the active (last-key) window writes the shared sidebar prefs to UserDefaults, so multiple @@ -208,8 +210,11 @@ final class AppStore: ObservableObject { // instead of lingering on the previous workroom's commits until the panel's `.task` catches // up. `focus` is idempotent, so the panel's later re-focus to the same root no-ops. if selectedTargetID != oldValue { - // Opening a container workroom starts its container if it was stopped (#309). - if let host = selectedWorkroom?.reachableHost { RemoteHosts.shared.activate(.remote(host)) } + // Opening a container workroom starts its container if it was stopped (#309). Its host is + // never let go of as idle while selected (#356). + let selected = selectedWorkroom?.reachableHost.map(HostID.remote) + RemoteHosts.shared.select(selected, in: ObjectIdentifier(self)) + if let selected { RemoteHosts.shared.activate(selected) } focusHistoryIfShown() focusRemoteStateIfShown() // A confirmation asking about the OLD workroom's dirty tree is meaningless once the selection @@ -3118,6 +3123,7 @@ final class AppStore: ObservableObject { // every reload passes here, and opening again on each would restart a container the user // stopped. if resolved == nil, let host = selectedWorkroom?.reachableHost { + RemoteHosts.shared.select(.remote(host), in: ObjectIdentifier(self)) RemoteHosts.shared.activate(.remote(host)) } // Prune shared caches only when publishing an accepted snapshot. @@ -3546,6 +3552,28 @@ final class AppStore: ObservableObject { /// shows it. @Published var imagePulls: [String: Double] = [:] + /// How far each project's boxd workroom create has got (#356): the step it is on, and that as a + /// fraction of the steps it will take. The row's spinner shows it, as it does an image pull. + @Published var createSteps: [String: CreateStep] = [:] + + struct CreateStep: Equatable { + let fraction: Double + let label: String + } + + /// `step` of a boxd create that builds the project's base first or not, as the row shows it, or + /// nil for a step that create does not take. + nonisolated static func createStep(_ step: RemoteProvisioning.Step, buildsBase: Bool) + -> CreateStep? + { + let all = RemoteProvisioning.Step.allCases + let steps = buildsBase ? all : Array(all.drop { $0 != .snapshot }) + guard let index = steps.firstIndex(of: step) else { return nil } + return CreateStep( + fraction: Double(index) / Double(steps.count), + label: "\(step.rawValue) (step \(index + 1) of \(steps.count))") + } + /// Why no workroom can be created in `project` right now, as the New Workroom menu shows it beside /// an entry (#309), or nil when one can: another create holds the project, or it is being deleted. func createBlockedReason(in project: Project) -> String? { @@ -3570,11 +3598,13 @@ final class AppStore: ObservableObject { && !deletingProjects.contains(project.path) } - /// Creates a container workroom for `project` on `runtime` (#253, #309), derived from the - /// project's base machine (built first if it has none), then selects it, which opens its first - /// pane on the far side. `splitAnchor` lands it beside that workroom instead, as `createWorkroom`. + /// Creates a remote workroom for `project` at `place` (#253, #309, #356): a container on this Mac + /// or a boxd machine, derived from the project's base machine there (built first if it has none), + /// then selects it, which opens its first pane on the far side. `splitAnchor` lands it beside + /// that workroom instead, as `createWorkroom`. func createRemoteWorkroom( - in project: Project, runtime: RemoteWorkrooms.Runtime = .docker, splitAnchor: SidebarID? = nil + in project: Project, place: RemoteWorkrooms.Place = .container(.docker), + splitAnchor: SidebarID? = nil ) async { // As it is now, not as a menu or the picker caught it: a create that finished meanwhile may // have recorded the project's first base, which this one must derive from, not overwrite. @@ -3585,10 +3615,14 @@ final class AppStore: ObservableObject { do { // A project keeps a base per runtime and Docker context (#309): the workroom derives from the // one where it is asked for, made there first if there is none. - let wanted = try await RemoteHosts.shared.key(for: runtime) + let wanted = try await RemoteHosts.shared.key(for: place) let base = RemoteWorkrooms.base(in: project.host, for: wanted) let key = base.flatMap(RemoteHosts.DriverKey.init) ?? wanted let (driver, environment) = try RemoteHosts.shared.environment(key) + // A remote host has no relay to fall back to (OQ20): say so before anything is made. + if key.isBoxd, environment.client == nil { + throw RemoteWorkrooms.Failure.codasetRequired + } let resolution = await WorkroomStatusResolver().resolveRepository(in: project.path) guard case .found(let repository) = resolution else { throw RemoteWorkrooms.Failure.notOnGitHub("Couldn't find its GitHub repository with gh.") @@ -3597,7 +3631,19 @@ final class AppStore: ObservableObject { throw RemoteWorkrooms.Failure.notOnGitHub("Its origin is on \(repository.host).") } let path = project.path - defer { imagePulls[path] = nil } + defer { + imagePulls[path] = nil + createSteps[path] = nil + } + let buildsBase = base == nil + let stepped: @Sendable (RemoteProvisioning.Step) -> Void = { step in + Task { @MainActor [weak self] in + guard let self, self.isBusyProject(path), + let shown = Self.createStep(step, buildsBase: buildsBase) + else { return } + self.createSteps[path] = shown + } + } let report: @Sendable (Double?) -> Void = { fraction in Task { @MainActor [weak self] in // A report that arrives after the create ended shows nothing. @@ -3605,12 +3651,17 @@ final class AppStore: ObservableObject { self.imagePulls[path] = fraction } } - let created = try await ContainerHostDriver.$pullProgress.withValue(report) { - try await RemoteWorkrooms.create( - repository: repository, cloneURL: RemoteWorkrooms.cloneURL(for: repository), - base: base, project: project.host, runtime: runtime, driver: driver, - environment: environment, - recorder: remoteRecorder(project: project.path)) + // Only a boxd create reports steps: a container's are quick, bar its image pull. + let created = try await RemoteProvisioning.$reportStep.withValue( + key.isBoxd ? stepped : nil + ) { + try await ContainerHostDriver.$pullProgress.withValue(report) { + try await RemoteWorkrooms.create( + repository: repository, cloneURL: RemoteWorkrooms.cloneURL(for: repository), + base: base, project: project.host, key: key, driver: driver, + environment: environment, + recorder: remoteRecorder(project: project.path)) + } } // Its panes attach through their own ssh; Changes and the rest connect for themselves. await created.instance.connection.close() @@ -3629,10 +3680,10 @@ final class AppStore: ObservableObject { signedIn: environment.client != nil, hadBase: base != nil, made: made) { errorTitle = - "\(runtime.displayName) workrooms of \(project.displayName) use your own GitHub access" + "\(place.displayName) workrooms of \(project.displayName) use your own GitHub access" errorMessage = "The Codaset App isn't installed for \(repository.owner), so git in this project's " - + "\(runtime.displayName) workrooms asks your gh sign-in on this Mac, with all of its " + + "\(place.displayName) workrooms asks your gh sign-in on this Mac, with all of its " + "access, as a workroom on this Mac does. They keep doing so after the App is installed." } } catch { diff --git a/macapp/WorkroomApp/Core/BoxdHostDriver.swift b/macapp/WorkroomApp/Core/BoxdHostDriver.swift index 8ee61a5c7..2234fcca7 100644 --- a/macapp/WorkroomApp/Core/BoxdHostDriver.swift +++ b/macapp/WorkroomApp/Core/BoxdHostDriver.swift @@ -1,4 +1,5 @@ import Foundation +import os /// The first real provider driver (#256): boxd.sh machines, driven through the `boxd` CLI and /// reached over ssh, with the **portable** derivation (design doc, Phase 4: "the slowest correct @@ -19,7 +20,7 @@ import Foundation /// base's processes with the base's machine-id and boot_id (measured); after it, the instance /// has only the base's disk, a kernel of its own, and the identity its identity unit mints at /// boot. The snapshot is removed once the machine exists (a machine outlives its snapshot, -/// measured). +/// measured); a failed derive's snapshot goes with its machine's `destroy`. /// - **Names.** A host is the machine `-`, and a derive's snapshot is named the /// same. A failed step removes by name, since the CLI's own output may never have arrived, and /// a driver made after a relaunch reaches a recorded host by its ID alone. @@ -39,11 +40,17 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { /// machines read as "not found", which `destroy` would take for gone and leave running. /// Whoever records a host records this beside it. var org: String? + /// The boxd account this driver's machines belong to, `boxd auth --json`'s `user_id`, or nil + /// to accept any (#356). Every personal account's org is nil, so after a switch to another + /// account the org check alone passes, and a machine of the first account reads as "not + /// found", which `destroy` would take for gone. Whoever records a host records this beside it. + var account: String? /// The agent's socket on every host. On the home disk, never the tmpfs `/run`: the agent keeps /// its broker enrolment beside it (`broker.rs`), and a stopped machine would lose it. - var agentSocket = "/home/boxd/.local/state/workroom/agent/agent.sock" + var agentSocket = "\(BoxdHostDriver.stateDirectory)/agent/agent.sock" + /// Where the supervisor has the agent keep each session's last screen (#232). - var screens = "/home/boxd/.local/state/workroom/screens" + var screens = "\(BoxdHostDriver.stateDirectory)/screens" /// The files the CLI writes each machine's ssh stanza and host key into. var sshConfig = FileManager.default.homeDirectoryForCurrentUser .appendingPathComponent(".ssh/config") @@ -51,6 +58,11 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { .appendingPathComponent(".ssh/known_hosts") } + /// The user every boxd machine logs in as, whose home holds the agent's state and the clone. + static let user = "boxd" + /// Where Workroom keeps its state on a boxd machine: the agent's socket and the sessions' screens. + static let stateDirectory = "/home/\(user)/.local/state/workroom" + /// Estimated from its measured parts, not timed end to end: the snapshot is most of it (9-11 s /// for a stock machine's disk), the restore half a second, and the reboot a few seconds more. /// A derive keeps no process: the reboot ends every one. @@ -75,6 +87,9 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { self.runner = runner } + private static let logger = Logger( + subsystem: "com.developwithstyle.workroom", category: "BoxdHostDriver") + func name(of id: UUID) -> String { "\(configuration.prefix)-\(id.uuidString.lowercased())" } private func id(of host: HostID) throws -> UUID { @@ -95,7 +110,9 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { let id = UUID() let name = name(of: id) do { + RemoteProvisioning.reportStep?(.machine) _ = try await cli(["machine", "new", name]) + RemoteProvisioning.reportStep?(.setup) let script = try Self.setupScript() let user = try ssh(id).user let (status, output) = try await exec( @@ -144,9 +161,11 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { throw HostDriverError.provisioning("syncing \(baseName) failed: \(output)") } // `snapshots save` prints nothing until it is done, and copying a base's disk takes a while. + RemoteProvisioning.reportStep?(.snapshot) _ = try await cli(["snapshots", "save", baseName, name], timeout: 900) // From here until the reboot the instance runs the base's processes (a restore resumes // them), so nothing waits in between that does not have to. + RemoteProvisioning.reportStep?(.restore) _ = try await cli(["machine", "new", name, "--from-snapshot", name]) // boxd names the machine on restore; minting before it had would key the identity on the // base's hostname. Measured to be there at once. @@ -164,6 +183,7 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { throw HostDriverError.provisioning("minting \(name)'s identity failed: \(said)") } let boot = try await bootID(of: id) + RemoteProvisioning.reportStep?(.reboot) _ = try await cli(["machine", "reboot", name]) _ = try await cli(["snapshots", "remove", name, "-y"]) try await awaitIdentity(of: id, rebootedFrom: boot) @@ -185,6 +205,21 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { try Task.checkCancellation() throw HostDriverError.provisioning(failure) } + // A derive that failed partway can leave its snapshot, named as its machine is (#356): it goes + // too. Best-effort: the machine is gone, which is what a delete reports, so a failure here is + // logged rather than made to read as a running box. Usually "not found", which this org (just + // checked) makes final, so no second org check. + do { + _ = try await cli(["snapshots", "remove", name(of: id), "-y"]) + } catch let failure as CLIFailure where failure.notFound { + } catch is CancellationError { + // A cancelled delete isn't done: it fails, so the record of the snapshot stays. + throw CancellationError() + } catch { + Self.logger.error( + "snapshot \(self.name(of: id), privacy: .public) not removed: \(error.localizedDescription, privacy: .public)" + ) + } try? FileManager.default.removeItem(at: hostDirectory(id)) } @@ -209,7 +244,8 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { (error as? CLIFailure).map { HostDriverError.provisioning($0.localizedDescription) } ?? error guard !leftover.isEmpty else { throw error } - throw HostDriverError.leftBehind(cause: error.localizedDescription, leftover: leftover) + throw HostDriverError.leftBehind( + cause: error.localizedDescription, leftover: leftover, host: .remote(id)) } /// Nil once the machine is gone, whether this removed it or it was never there. @@ -290,13 +326,14 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { ) { "\(self.name(of: id))'s agent never answered: \($0)" } } - /// Refuses to act while boxd's active org is not this driver's (`Configuration.org`). + /// Refuses to act while boxd's active org or signed-in account is not this driver's + /// (`Configuration.org`, `Configuration.account`). private func checkOrg() async throws { - let account: Account - do { - account = try decode(Account.self, await cli(["auth"]), "the signed-in account") - } catch let failure as CLIFailure { - throw HostDriverError.provisioning(failure.localizedDescription) + let account = try await signedIn() + if let expected = configuration.account, account.userID != expected { + throw HostDriverError.invalidConfiguration( + "boxd is signed in to another account than the one this workroom's machines are in; sign" + + " back in to that account with `boxd auth login`") } guard account.activeOrg == configuration.org else { throw HostDriverError.invalidConfiguration( @@ -305,6 +342,17 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { } } + /// The signed-in account, as `boxd auth --json` reports it. Throws, naming `boxd auth login`, + /// when nobody is signed in. + func signedIn() async throws -> Account { + do { + return try decode(Account.self, await cli(["auth"]), "the signed-in account") + } catch let failure as CLIFailure { + throw HostDriverError.provisioning( + "\(failure.localizedDescription). Sign in with `boxd auth login`.") + } + } + private func poll( _ id: UUID, _ check: String, tries: Int, failure: (String) -> String ) async throws { @@ -331,22 +379,130 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { private struct CLIFailure: Error, LocalizedError { let command: String let said: String + /// The CLI ran and answered with its own `error:` line, as opposed to timing out or failing to + /// run at all. + var answered = false var notFound: Bool { BoxdHostDriver.isNotFound(said) } var errorDescription: String? { "boxd \(command): \(said)" } } /// `auth --json`, as far as the driver reads it. - private struct Account: Decodable { + struct Account: Decodable, Equatable { let activeOrg: String? - enum CodingKeys: String, CodingKey { case activeOrg = "active_org" } + let userID: String? + enum CodingKeys: String, CodingKey { + case activeOrg = "active_org" + case userID = "user_id" + } + + /// The account's id, which a new host records (#356): without it, deleting the host would skip + /// the check that the same account is signed in, and another account's "not found" would read + /// as gone. + func requiredUserID() throws -> String { + guard let userID else { + throw HostDriverError.provisioning( + "boxd didn't say which account is signed in. Sign in again with `boxd auth login`.") + } + return userID + } } /// `machine get --json`, as far as the driver reads it. struct Machine: Decodable { - /// `standalone`, `fork/` or `snapshot/:`. - let source: String + /// `standalone`, `fork/` or `snapshot/:`. Optional, so a presence read + /// that needs only `status` does not fail open on it. + var source: String? = nil + /// `running`, `standby` (suspended; `get` normalises the raw `suspended`), `hibernated`, + /// `stopped`, and others in passing (boxd CLI docs). + var status: String? = nil + } + + /// Whether `host` is asleep, from boxd's own record, without reaching the machine: an ssh login + /// would wake it (#356). Nil when the CLI could not answer (missing, timed out, output it did not + /// expect, a machine without a status); `RemoteHosts` takes that as asleep for a background read. + /// + /// `gone` when boxd answers that the machine is not found: deleted outside the app, or in an org + /// that isn't active. Any other error the CLI answers with (signed out, another account) counts + /// as asleep: the read can't tell, and connecting would wake a box the user may be paying for, + /// so a background read leaves it alone; opening the workroom still connects. + func presence(_ host: HostID) async -> Presence? { + guard case .remote(let id) = host else { return nil } + do { + let output = try await cli( + ["machine", "get", name(of: id)], timeout: Self.statusTimeout) + let machine = try decode(Machine.self, output, name(of: id)) + // A reply without a status says nothing either way: taken as awake, it would wake the box. + guard let status = machine.status else { return nil } + return Self.asleepStatuses.contains(status) ? .asleep : .awake + } catch let failure as CLIFailure { + if failure.notFound { return .gone } + return failure.answered ? .asleep : nil + } catch { return nil } + } + + /// What boxd says of a machine, for a background read (#356). + enum Presence: Equatable, Sendable { case awake, asleep, gone } + + /// How long boxd lets `host` sit idle on the network before suspending or hibernating it, in + /// seconds: the shorter of the two that are set, or nil when neither is (#356). Throws when boxd + /// can't say, which a caller must not take for "no timer". + func idleWindow(_ host: HostID) async throws -> TimeInterval? { + guard case .remote(let id) = host else { return nil } + let timers = try decode( + Timers.self, await cli(["machine", "get", name(of: id)], timeout: Self.statusTimeout), "") + return [timers.autoSuspend, timers.autoHibernate].compactMap { $0?.seconds }.filter { $0 > 0 } + .min() + } + + /// `machine get --json`'s idle timers. Their encoding isn't documented, so each is read as + /// seconds from a number, or from a string such as `120`, `120s`, `2m` or `1h`; anything else is + /// nil, which never warns. + struct Timers: Decodable { + let autoSuspend: Seconds? + let autoHibernate: Seconds? + enum CodingKeys: String, CodingKey { + case autoSuspend = "auto_suspend" + case autoHibernate = "auto_hibernate" + } + + init(from decoder: Decoder) throws { + let fields = try decoder.container(keyedBy: CodingKeys.self) + autoSuspend = try? fields.decodeIfPresent(Seconds.self, forKey: .autoSuspend) + autoHibernate = try? fields.decodeIfPresent(Seconds.self, forKey: .autoHibernate) + } } + struct Seconds: Decodable, Equatable { + let seconds: TimeInterval? + + init(from decoder: Decoder) throws { + let value = try decoder.singleValueContainer() + let read = try (try? value.decode(Double.self)) ?? Self.parse(value.decode(String.self)) + // A timer past `maximum` (or not finite) is nothing boxd sets, and as a `Duration` it + // would trap: it reads as unknown instead. + seconds = read.flatMap { $0.isFinite && $0 <= Self.maximum ? $0 : nil } + } + + /// A week: longer than any idle timer boxd offers. + static let maximum: TimeInterval = 7 * 24 * 3600 + + static func parse(_ text: String) -> TimeInterval? { + let text = text.trimmingCharacters(in: .whitespaces).lowercased() + let units: [(String, Double)] = [("h", 3600), ("m", 60), ("s", 1)] + for (suffix, scale) in units where text.hasSuffix(suffix) { + return Double(text.dropLast()).map { $0 * scale } + } + return Double(text) + } + } + + /// How long a status read (`presence`, `idleWindow`) waits for the CLI: a status sweep and a + /// connect wait on it, so a hung CLI must not hold them for the CLI's usual 120 s. + static let statusTimeout: TimeInterval = 10 + + /// The statuses a connection would wake from: suspended and hibernated. + static let asleepStatuses: Set = ["standby", "hibernated"] + private func decode(_ type: T.Type, _ output: String, _ name: String) throws -> T { do { return try JSONDecoder().decode(T.self, from: Data(output.utf8)) @@ -363,10 +519,11 @@ final class BoxdHostDriver: HostTerminalDriver, @unchecked Sendable { // The runner kills the CLI when the task is cancelled, and that reads as a failure. try Task.checkCancellation() guard result.ok else { + let line = Self.errorLine(result.stderr) let said = - Self.errorLine(result.stderr) - ?? (result.timedOut ? "timed out after \(Int(timeout))s" : "exited \(result.exitCode)") - throw CLIFailure(command: arguments.prefix(2).joined(separator: " "), said: said) + line ?? (result.timedOut ? "timed out after \(Int(timeout))s" : "exited \(result.exitCode)") + throw CLIFailure( + command: arguments.prefix(2).joined(separator: " "), said: said, answered: line != nil) } return result.stdout } diff --git a/macapp/WorkroomApp/Core/ContainerHostDriver.swift b/macapp/WorkroomApp/Core/ContainerHostDriver.swift index dfaefbee8..4b207d45f 100644 --- a/macapp/WorkroomApp/Core/ContainerHostDriver.swift +++ b/macapp/WorkroomApp/Core/ContainerHostDriver.swift @@ -258,7 +258,7 @@ final class ContainerHostDriver: HostTerminalDriver, @unchecked Sendable { }.value // A run that could not remove its container says so; its image goes on that list too. var (cause, leftover) = (error.localizedDescription, removals) - if case HostDriverError.leftBehind(let inner, let left) = error { + if case HostDriverError.leftBehind(let inner, let left, _) = error { (cause, leftover) = (inner, left + removals) } guard !leftover.isEmpty else { throw error } diff --git a/macapp/WorkroomApp/Core/GhosttySurfaceView.swift b/macapp/WorkroomApp/Core/GhosttySurfaceView.swift index b5432bb1c..c40f9b845 100644 --- a/macapp/WorkroomApp/Core/GhosttySurfaceView.swift +++ b/macapp/WorkroomApp/Core/GhosttySurfaceView.swift @@ -68,9 +68,20 @@ final class GhosttySurfaceView: NSView { /// #7); nil for every ordinary terminal. The host passes a shell-wrapped string (`$SHELL -lic …`). private let runCommand: String? + /// Names this pane's attach (`PersistentSessionService.attachCommand(by:)`), so only this pane + /// clears it. Not its object identity, which a later view can reuse once this one is freed. + let attachToken = UUID() /// Daemon session this surface attaches to. When set and the helper is available, `createSurface` /// launches `workroom-session attach` instead of a login shell. - var persistentSessionID: UUID? + var persistentSessionID: UUID? { + // Pointed at another session (the detached-sessions list): the old one's ssh goes with the + // surface it is about to lose, so it holds its remote host no more (#356). + didSet { + if let oldValue, oldValue != persistentSessionID { + PersistentSessionService.shared.paneDetached(oldValue, by: attachToken) + } + } + } /// Whether `persistentSessionID` was carried over from a previous launch rather than minted for /// this pane. Only a restored id can name a session that has since died, and only a restored id /// is re-checked before attaching — asking about a fresh one would refuse every new pane, since @@ -310,6 +321,8 @@ final class GhosttySurfaceView: NSView { surface = ghostty_surface_new(app, &config) guard let surface else { freeSurfaceCStrings() + // No surface, no ssh: the attach its command marked holds nothing (#356). + detachFromHost() return } @@ -360,6 +373,8 @@ final class GhosttySurfaceView: NSView { private func destroySurface() { if let surface { ghostty_surface_free(surface) } surface = nil + // Its ssh went with it, so it holds its remote host no more (#356). + detachFromHost() lastSetFocus = nil // a new surface starts unfocused — don't let a stale value skip the re-sync freeSurfaceCStrings() } @@ -399,6 +414,12 @@ final class GhosttySurfaceView: NSView { for observer in keyWindowObservers { NotificationCenter.default.removeObserver(observer) } if let surface { ghostty_surface_free(surface) } freeSurfaceCStrings() + // A window closing frees its panes without a teardown; their ssh goes with them (#356). Only + // this pane's own mark goes, so the hop to the main actor cannot undo a later pane's attach. + if surface != nil, let id = persistentSessionID { + let pane = attachToken + Task { @MainActor in PersistentSessionService.shared.paneDetached(id, by: pane) } + } } private func freeSurfaceCStrings() { @@ -460,7 +481,8 @@ final class GhosttySurfaceView: NSView { // the owning backend has no binary, so the global check bought nothing on this path. guard let attach = PersistentSessionService.shared.attachCommand( - forSession: persistentSessionID, restored: persistentSessionIsRestored), + forSession: persistentSessionID, restored: persistentSessionIsRestored, + by: attachToken), let attachPointer = strdup(attach) else { // A pane that expected a persisted session fell back to a plain login shell. Whatever the @@ -917,7 +939,16 @@ final class GhosttySurfaceView: NSView { /// attach twice, and by closing the pane. private(set) var pendingReconnect: DispatchWorkItem? + /// This pane's ssh is gone: its attach no longer holds its remote host awake (#356). + private func detachFromHost() { + if let persistentSessionID { + PersistentSessionService.shared.paneDetached(persistentSessionID, by: attachToken) + } + } + func handleChildExited(exitCode: UInt32) { + // A dead ssh holds its remote host no more, whether or not it reconnects (#356). + detachFromHost() reconnectIfTheLinkDropped(exitCode: exitCode) onChildExited?(exitCode) } diff --git a/macapp/WorkroomApp/Core/HostConnectionManager.swift b/macapp/WorkroomApp/Core/HostConnectionManager.swift index 147268d64..6a2aca87d 100644 --- a/macapp/WorkroomApp/Core/HostConnectionManager.swift +++ b/macapp/WorkroomApp/Core/HostConnectionManager.swift @@ -218,6 +218,15 @@ actor HostConnectionManager { invalidate(host: lease.host, error: HostConnectionError.connectionLost) } + /// Disconnects `lease` unless `keep` says to keep it, asked here so nothing on this actor runs in + /// between: a caller that took the host back before this keeps the connection it found up (#356). + /// Returns whether it disconnected. + func disconnect(_ lease: Lease, unless keep: @Sendable () -> Bool) -> Bool { + guard !keep() else { return false } + disconnect(lease) + return true + } + func perform( on lease: Lease, operation: @escaping @Sendable () async throws -> Value ) async throws -> Value { diff --git a/macapp/WorkroomApp/Core/HostDriver.swift b/macapp/WorkroomApp/Core/HostDriver.swift index f1861ebfa..5dfa2c2e8 100644 --- a/macapp/WorkroomApp/Core/HostDriver.swift +++ b/macapp/WorkroomApp/Core/HostDriver.swift @@ -110,8 +110,9 @@ enum HostDriverError: Error, Equatable, Sendable, LocalizedError { /// host as it was, still known to the driver, so it can be tried again. case provisioning(String) /// A failed `create` or `deriveFromBase` could not remove what it had made: each of `leftover` - /// is still there, named by what the driver called it. - case leftBehind(cause: String, leftover: [String]) + /// is still there, named by what the driver called it. `host`, when the driver can name it, is + /// the host it made, so the caller can record it for a later delete to take down (#356). + case leftBehind(cause: String, leftover: [String], host: HostID? = nil) var errorDescription: String? { switch self { @@ -119,7 +120,7 @@ enum HostDriverError: Error, Equatable, Sendable, LocalizedError { case .notImplemented(let what): return "\(what) is not implemented yet." case .invalidConfiguration(let detail): return "Invalid host configuration: \(detail)" case .provisioning(let detail): return "Couldn't provision the host: \(detail)" - case .leftBehind(let cause, let leftover): + case .leftBehind(let cause, let leftover, _): return "Couldn't provision the host: \(cause) Still there: \(leftover.joined(separator: "; "))" } diff --git a/macapp/WorkroomApp/Core/Models.swift b/macapp/WorkroomApp/Core/Models.swift index f058c9c2e..5ac069c35 100644 --- a/macapp/WorkroomApp/Core/Models.swift +++ b/macapp/WorkroomApp/Core/Models.swift @@ -19,7 +19,8 @@ struct Warning: Codable, Hashable { /// remote workroom away from every local action instead of failing the whole listing. struct HostDescriptor: Codable, Hashable { var state: String? = nil - /// Which `HostDriver` made the host: `container` (`ContainerHostDriver`). + /// Which `HostDriver` made the host: `container` or `apple-container` (`ContainerHostDriver`), + /// or `boxd` (`BoxdHostDriver`, #356). var driver: String? = nil /// The bundle ID of the build that made the host, whose key and labels it carries: a Dev and a /// Nightly app share config but not keys, so each adopts only its own hosts. @@ -44,6 +45,12 @@ struct HostDescriptor: Codable, Hashable { /// `relay` for a local container base or workroom whose git credentials come from the Mac's /// relay instead of the broker (#309); nil for the broker. var credentials: String? = nil + /// A boxd host's org, as `boxd auth` named the active one when it was made; nil is the account's + /// own (#356). + var org: String? = nil + /// A boxd host's account, `boxd auth`'s `user_id` when it was made (#356). Every personal + /// account's org is nil, so the org alone can't tell two accounts apart. + var account: String? = nil /// What kind of host this is, for the workroom's icon in the sidebar (#309): a container on this /// Mac, on Docker (naming its context when it has one) or Apple's runtime, or a remote provider by @@ -69,7 +76,8 @@ struct HostDescriptor: Codable, Hashable { var isDestroyed: Bool { state == "destroyed" } enum CodingKeys: String, CodingKey { - case state, driver, provisioner, id, repository, path, container, bases, credentials + case state, driver, provisioner, id, repository, path, container, bases, credentials, org + case account case grantID = "grant_id" case workroomID = "workroom_id" case cloneURL = "clone_url" @@ -79,7 +87,8 @@ struct HostDescriptor: Codable, Hashable { state: String? = nil, driver: String? = nil, provisioner: String? = nil, id: UUID? = nil, grantID: String? = nil, workroomID: UUID? = nil, repository: String? = nil, cloneURL: String? = nil, path: String? = nil, container: ContainerHostDriver.Record? = nil, - bases: [HostDescriptor]? = nil, credentials: String? = nil + bases: [HostDescriptor]? = nil, credentials: String? = nil, org: String? = nil, + account: String? = nil ) { self.state = state self.driver = driver @@ -93,6 +102,8 @@ struct HostDescriptor: Codable, Hashable { self.container = container self.bases = bases self.credentials = credentials + self.org = org + self.account = account } init(from decoder: Decoder) throws { @@ -109,6 +120,8 @@ struct HostDescriptor: Codable, Hashable { container = try? fields?.decodeIfPresent(ContainerHostDriver.Record.self, forKey: .container) bases = try? fields?.decodeIfPresent([HostDescriptor].self, forKey: .bases) credentials = try? fields?.decodeIfPresent(String.self, forKey: .credentials) + org = try? fields?.decodeIfPresent(String.self, forKey: .org) + account = try? fields?.decodeIfPresent(String.self, forKey: .account) } /// A project's base, when the descriptor records a whole one. diff --git a/macapp/WorkroomApp/Core/RemoteProvisioning.swift b/macapp/WorkroomApp/Core/RemoteProvisioning.swift index ee44fa6a2..b1af44f43 100644 --- a/macapp/WorkroomApp/Core/RemoteProvisioning.swift +++ b/macapp/WorkroomApp/Core/RemoteProvisioning.swift @@ -13,6 +13,24 @@ import Foundation /// /// Repository work runs through the agent's exec service, and the Go CLI stays on the Mac. enum RemoteProvisioning { + /// A step of a remote create that takes long enough to say (#356): the boxd driver's machine work + /// and the sequence's own git and enrolment. + enum Step: String, CaseIterable, Sendable { + case machine = "Creating the base machine" + case setup = "Setting up the base machine" + case clone = "Cloning the repository" + case snapshot = "Copying the base machine" + case restore = "Starting the workroom's machine" + case reboot = "Restarting it with its own identity" + case enrol = "Signing it in to Codaset" + case checkout = "Checking out the workroom's branch" + } + + /// Told each `Step` as it starts, for a create's progress on its project row. Set with + /// `$reportStep.withValue`: a task-local, so the driver's methods keep their shape, as + /// `ContainerHostDriver.$pullProgress` does for an image pull. + @TaskLocal static var reportStep: (@Sendable (Step) -> Void)? + /// What the sequence reaches a provider and the broker through. struct Environment: Sendable { let driver: any HostDriver @@ -43,7 +61,10 @@ enum RemoteProvisioning { self.agentBroker = agentBroker self.connect = connect ?? { host in - try await AgentBootstrap.connect(host: host, driver: driver, socket: agentSocket) + // Nothing watches a provisioning connection's ceiling prompts, so it hands the agent + // no settings (#356). + try await AgentBootstrap.connect( + host: host, driver: driver, socket: agentSocket, watched: false) } self.revoke = revoke } @@ -116,6 +137,7 @@ enum RemoteProvisioning { guard case .remote(let id) = host else { throw HostDriverError.unknownHost(host) } let connection = try await environment.connect(host) defer { Task { await connection.close() } } + reportStep?(.clone) let relayed = try await withCloneToken(repository, relayed: nil, in: environment) { header in _ = try await git( ["clone", "--quiet", "--origin", "origin", "--", cloneURL, path], in: "/", @@ -274,6 +296,7 @@ enum RemoteProvisioning { } else { guard let client = environment.client else { throw RemoteWorkrooms.Failure.signedOut } let grantID: String + reportStep?(.enrol) do { grantID = try await AgentEnrolment.enrol( client: client, driver: environment.driver, host: host, @@ -288,6 +311,7 @@ enum RemoteProvisioning { try await checkpoint(host, grantID) } // With the instance's own credentials: `wr-agent enrol` made its helper git's. + reportStep?(.checkout) try await fetch(base.path, environment: header, on: connected) // Fully qualified: a tag or branch named `origin/HEAD` would make the short form ambiguous. _ = try await git( diff --git a/macapp/WorkroomApp/Core/RemoteWorkrooms.swift b/macapp/WorkroomApp/Core/RemoteWorkrooms.swift index 870af7983..3e2d0baad 100644 --- a/macapp/WorkroomApp/Core/RemoteWorkrooms.swift +++ b/macapp/WorkroomApp/Core/RemoteWorkrooms.swift @@ -35,8 +35,54 @@ enum RemoteWorkrooms { var containerPhrase: String { self == .docker ? "a Docker container" : "an Apple container" } } + /// Where a remote workroom is made (#309, #356): a container runtime on this Mac, or boxd. + enum Place: Hashable, Sendable { + case container(Runtime) + case boxd + + var displayName: String { + switch self { + case .container(let runtime): runtime.displayName + case .boxd: "boxd" + } + } + + /// One of its hosts, in a sentence: "a Docker container on this Mac", "a boxd machine". + var hostPhrase: String { + switch self { + case .container(let runtime): "\(runtime.containerPhrase) on this Mac" + case .boxd: "a boxd machine" + } + } + } + + /// Why a workroom can't be created at `place` now, as the New Workroom menu and picker show it + /// beside the entry, or nil when it can. + @MainActor + static func unavailability(of place: Place) -> String? { + switch place { + case .container(let runtime): unavailability(of: runtime) + case .boxd: + // Only what is cheap to read: this runs in menus' and the picker's bodies. Whether boxd is + // signed in is a CLI call, so a create checks that (`RemoteHosts.key(for:)`). + unavailability( + ofBoxdInstalled: RemoteHosts.boxdExecutable() != nil, + codasetSignedIn: BrokerSession.shared.isSignedIn) + } + } + + /// A boxd workroom fetches and pushes with the broker's tokens, never the Mac's `gh` (OQ20): it + /// must keep working with the Mac closed. So Codaset is required, where a container is not. + static func unavailability(ofBoxdInstalled installed: Bool, codasetSignedIn: Bool) -> String? { + if !installed { return "not installed" } + if !codasetSignedIn { return "sign in to Codaset" } + return nil + } + /// The descriptor's `driver` for a Docker host. static let containerDriver = Runtime.docker.rawValue + /// The descriptor's `driver` for a boxd host (#356). + static let boxdDriver = "boxd" /// Why a workroom can't be created on `runtime` now, as the New Workroom menu shows it beside the /// entry, or nil when it can (#309). @@ -98,9 +144,12 @@ enum RemoteWorkrooms { "https://\(repository.host)/\(repository.owner)/\(repository.name).git" } - /// Where a project's base clones its repository on the host. - static func clonePath(for repository: GitHubRepository) -> String { - "/home/\(user)/\(repository.name)" + /// Where a project's base clones its repository on a host of `key`: the home of the user its + /// driver logs in as (`workroom` in a container, `boxd` on boxd, #356). + static func clonePath( + for repository: GitHubRepository, on key: RemoteHosts.DriverKey = .init() + ) -> String { + "/home/\(key.isBoxd ? BoxdHostDriver.user : user)/\(repository.name)" } enum Failure: Error, LocalizedError, Equatable { @@ -113,6 +162,8 @@ enum RemoteWorkrooms { case anotherBuildsHost(String) case baseRepositoryChanged(base: String, origin: String) case incompleteBase + case codasetRequired + case boxdNotInstalled var errorDescription: String? { switch self { @@ -137,6 +188,12 @@ enum RemoteWorkrooms { return "This project's base machine record is incomplete, so it can't be reused, and " + "building another would leave it running unrecorded. Delete the project (its remote " + "workrooms and base go with it) and add it again." + case .codasetRequired: + return "A boxd workroom fetches and pushes with Codaset's repository tokens, so it keeps " + + "working with this Mac closed. Sign in to Codaset in Settings → Remote workrooms." + case .boxdNotInstalled: + return "The boxd command wasn't found. Install it from boxd.sh, then sign in with " + + "`boxd auth login`." case .noDocker: return "No docker command was found. Install Docker Desktop, OrbStack or Colima." case .noAppleContainer: @@ -154,8 +211,10 @@ enum RemoteWorkrooms { /// no context and follows the current one, as `key` does. static func base(in host: HostDescriptor?, for key: RemoteHosts.DriverKey) -> HostDescriptor? { let bases = host?.allBases ?? [] - return bases.first { RemoteHosts.DriverKey($0) == key } - ?? bases.first { RemoteHosts.DriverKey($0) == RemoteHosts.DriverKey(runtime: key.runtime) } + if let exact = bases.first(where: { RemoteHosts.DriverKey($0) == key }) { return exact } + // Only Docker has an unpinned form; a boxd base belongs to exactly one org and account. + guard let runtime = key.runtime else { return nil } + return bases.first { RemoteHosts.DriverKey($0) == RemoteHosts.DriverKey(runtime: runtime) } } /// A project's descriptor `host` with `base` recorded in it, in place of any base on the same @@ -220,15 +279,15 @@ enum RemoteWorkrooms { /// Creates a remote workroom for a project (#253): its base first if it has none, then a name /// in config, then the derived instance, then the instance's descriptor. The name is taken /// before the derive, so the branch is named for it and a crash part-way leaves an entry the - /// user can see and delete; anything the crash left on the host is the sweep's - /// (`RemoteHosts.adopt`). + /// user can see and delete; anything the crash left on a container host is the sweep's + /// (`RemoteHosts.adopt`). A boxd machine a crash left is nobody's until #284. /// /// When the derive fails and undid itself, the entry is dropped. When undoing it failed too, /// the entry keeps what is still live (host, grant) so deleting it can finish the job. static func create( repository: GitHubRepository, cloneURL: String, base existing: HostDescriptor?, - project projectHost: HostDescriptor? = nil, runtime: Runtime = .docker, - driver: ContainerHostDriver, + project projectHost: HostDescriptor? = nil, key: RemoteHosts.DriverKey = .init(), + driver: any HostTerminalDriver, environment: RemoteProvisioning.Environment, recorder: Recorder ) async throws -> Created { let base: RemoteProvisioning.Base @@ -236,11 +295,13 @@ enum RemoteWorkrooms { // Its key and labels are another build's, so this one can neither reach nor replace it. throw Failure.anotherBuildsBase(existing.provisioner ?? "an unknown build") } - // A workroom is derived from its base, on the base's runtime. - if let existing, existing.id != nil, - let other = Runtime(rawValue: existing.driver ?? Runtime.docker.rawValue), other != runtime - { - throw Failure.baseOnOtherRuntime(other.displayName) + // A workroom is derived from its base, on the base's driver: a base of another runtime, or of + // boxd when the workroom isn't (or the reverse), is never derived from (#356). + if let existing, existing.id != nil { + let other = (try? RemoteHosts.deletionKey(existing)) ?? .init() + if other.runtime != key.runtime { + throw Failure.baseOnOtherRuntime(other.place.displayName) + } } if let recorded = existing?.base { // Reused only for the repository it cloned: a changed origin would otherwise get workrooms of @@ -256,22 +317,32 @@ enum RemoteWorkrooms { // live with nothing pointing at it. throw Failure.incompleteBase } else { - base = try await RemoteProvisioning.buildBase( - repository: "\(repository.owner)/\(repository.name)", - cloneURL: cloneURL, - path: clonePath(for: repository), in: environment - ) { base in - // Beside the project's bases on other runtimes and contexts (#309). - try await recorder.record( - nil, - recording( - HostDescriptor( - driver: runtime.rawValue, provisioner: provisioner, id: base.host, - repository: base.repository, - cloneURL: base.cloneURL, path: base.path, - container: driver.record(of: .remote(base.host)), - credentials: base.relayed == true ? "relay" : nil), - in: projectHost)) + do { + base = try await RemoteProvisioning.buildBase( + repository: "\(repository.owner)/\(repository.name)", + cloneURL: cloneURL, + path: clonePath(for: repository, on: key), in: environment + ) { base in + // Beside the project's bases on other runtimes and contexts (#309). + var descriptor = describing( + .remote(base.host), key: key, driver: driver, credentials: base.relayed) + descriptor.repository = base.repository + descriptor.cloneURL = base.cloneURL + descriptor.path = base.path + try await recorder.record(nil, recording(descriptor, in: projectHost)) + } + } catch { + // A base machine left running stays recorded, `failed`, so the project knows it (a later + // create refuses to make a second one beside it) and a delete finds it: whether the driver + // could not undo its create, or the build's own rollback could not destroy it. + if let host = leftBehindHost(error) { + var descriptor = describing(host, key: key, driver: driver, credentials: nil) + descriptor.state = "failed" + await recordLeftBehind(host) { + try await recorder.record(nil, recording(descriptor, in: projectHost)) + } + } + throw error } } @@ -279,8 +350,8 @@ enum RemoteWorkrooms { let name = try await recorder.reserve( base.path, HostDescriptor( - state: "creating", driver: runtime.rawValue, provisioner: provisioner, - workroomID: workroomID)) + state: "creating", driver: driverName(key), provisioner: provisioner, + workroomID: workroomID, org: key.org, account: key.account)) let instance: RemoteProvisioning.Instance do { instance = try await RemoteProvisioning.derive( @@ -290,46 +361,61 @@ enum RemoteWorkrooms { try await recorder.record( name, remaining( - state: "creating", workroomID: workroomID, host: host, grant: grant, runtime: runtime, + state: "creating", workroomID: workroomID, host: host, grant: grant, key: key, driver: driver)) } } catch RemoteProvisioning.Failure.rollbackIncomplete(let cause, let host, let grant, let left) { // `destroyed` only with nothing live: the CLI deletes a destroyed entry, and a live grant // must keep its record until the app's delete cancels it. - try? await recorder.record( - name, - remaining( - state: host == nil && grant == nil ? "destroyed" : "failed", workroomID: workroomID, - host: host, grant: grant, runtime: runtime, driver: driver)) + await recordLeftBehind(host, grant: grant) { + try await recorder.record( + name, + remaining( + state: host == nil && grant == nil ? "destroyed" : "failed", workroomID: workroomID, + host: host, grant: grant, key: key, driver: driver)) + } throw RemoteProvisioning.Failure.rollbackIncomplete( cause: cause, host: host, grantID: grant, cleanup: left) + } catch HostDriverError.leftBehind(let cause, let left, let host?) { + // The driver could not remove the machine it made: the entry keeps it, `failed`, so deleting + // the workroom takes it down. Forgotten, a paid machine would run on with nothing to find it. + await recordLeftBehind(host) { + try await recorder.record( + name, + remaining( + state: "failed", workroomID: workroomID, host: host, grant: nil, key: key, + driver: driver)) + } + throw RemoteProvisioning.Failure.rollbackIncomplete( + cause: cause, host: host, grantID: nil, cleanup: left) } catch { try? await recorder.forget(name) throw error } - guard case .remote(let id) = instance.host else { + guard case .remote = instance.host else { throw HostDriverError.unknownHost(instance.host) } do { - try await recorder.record( - name, - HostDescriptor( - driver: runtime.rawValue, provisioner: provisioner, id: id, grantID: instance.grantID, - workroomID: workroomID, container: driver.record(of: instance.host), - credentials: base.relayed == true ? "relay" : nil)) + var descriptor = describing( + instance.host, key: key, driver: driver, credentials: base.relayed) + descriptor.grantID = instance.grantID + descriptor.workroomID = workroomID + try await recorder.record(name, descriptor) } catch { // Unrecorded, the instance would be found by nothing but the sweep, and its grant by nothing. do { try await RemoteProvisioning.destroy(instance, workroom: workroomID, in: environment) } catch RemoteProvisioning.Failure.rollbackIncomplete(_, let host, let grant, _) { // Something is still live: the entry stays, `failed`, for delete to finish. - try? await recorder.record( - name, - remaining( - state: "failed", workroomID: workroomID, host: host, grant: grant, runtime: runtime, - driver: driver)) + await recordLeftBehind(host, grant: grant) { + try await recorder.record( + name, + remaining( + state: "failed", workroomID: workroomID, host: host, grant: grant, key: key, + driver: driver)) + } throw error } catch { // `destroy` throws nothing else: everything it made is gone. @@ -342,19 +428,37 @@ enum RemoteWorkrooms { /// The descriptor of a workroom whose undoing left `host` and `grant` live. private static func remaining( - state: String, workroomID: UUID, host: HostID?, grant: String?, runtime: Runtime, - driver: ContainerHostDriver + state: String, workroomID: UUID, host: HostID?, grant: String?, key: RemoteHosts.DriverKey, + driver: any HostTerminalDriver ) -> HostDescriptor { var descriptor = HostDescriptor( - state: state, driver: runtime.rawValue, provisioner: provisioner, grantID: grant, - workroomID: workroomID) + state: state, driver: driverName(key), provisioner: provisioner, grantID: grant, + workroomID: workroomID, org: key.org, account: key.account) if case .remote(let id) = host { descriptor.id = id - descriptor.container = driver.record(of: .remote(id)) + descriptor.container = (driver as? ContainerHostDriver)?.record(of: .remote(id)) } return descriptor } + /// A live host's descriptor on `key`: its driver, this build, and how `key`'s driver finds it + /// again (a container record, or a boxd org and account). + private static func describing( + _ host: HostID, key: RemoteHosts.DriverKey, driver: any HostTerminalDriver, credentials: Bool? + ) -> HostDescriptor { + var descriptor = HostDescriptor( + driver: driverName(key), provisioner: provisioner, + container: (driver as? ContainerHostDriver)?.record(of: host), + credentials: credentials == true ? "relay" : nil, org: key.org, account: key.account) + if case .remote(let id) = host { descriptor.id = id } + return descriptor + } + + /// The descriptor's `driver` for `key`. + static func driverName(_ key: RemoteHosts.DriverKey) -> String { + key.runtime?.rawValue ?? boxdDriver + } + /// Whether `host` records anything still up: a box or a grant. A destroyed one has nothing, and /// neither has one a create left at `creating` before its derive made a box, whose container (if /// any) the sweep takes once no record names it. @@ -410,7 +514,7 @@ enum RemoteWorkrooms { if let id = host.id { await MainActor.run { WakefulnessModel.forgetHost(id) } } (remaining.id, remaining.container) = (nil, nil) } - try? await recorder.record(name, remaining) + await recordLeftBehind(liveHost, grant: grant) { try await recorder.record(name, remaining) } throw RemoteProvisioning.Failure.rollbackIncomplete( cause: cause, host: liveHost, grantID: grant, cleanup: left) } @@ -418,6 +522,36 @@ enum RemoteWorkrooms { try await forget() } + /// The machine a failed create left running, when the failure names one: the driver's own undo + /// (`leftBehind`) or a build's rollback (`rollbackIncomplete`) that could not take it down. + private static func leftBehindHost(_ error: any Error) -> HostID? { + switch error { + case HostDriverError.leftBehind(_, _, let host?): return host + case RemoteProvisioning.Failure.rollbackIncomplete(_, let host?, _, _): return host + default: return nil + } + } + + /// Records what a failure left live (a machine, a grant). A record that fails too is logged with + /// it named, since nothing else in the app will remember it (#356). + private static func recordLeftBehind( + _ host: HostID?, grant: String? = nil, _ write: () async throws -> Void + ) async { + do { try await write() } catch { + let live = [host.map { String(describing: $0) }, grant.map { "grant \($0)" }].compactMap { + $0 + } + let what = + live.isEmpty ? "the workroom as destroyed" : "\(live.joined(separator: ", ")) left live" + logger.error( + "could not record \(what, privacy: .public): \(error.localizedDescription, privacy: .public)" + ) + } + } + + private static let logger = Logger( + subsystem: "com.developwithstyle.workroom", category: "RemoteWorkrooms") + /// Destroys a project's base (#253), for an explicit project delete, then clears its record /// (`clear`). Its workrooms go first: config keeps a project with a base when its last workroom is /// removed, and only while it has one. @@ -433,18 +567,24 @@ enum RemoteWorkrooms { /// The app's remote hosts (#253): a `ContainerHostDriver` per container runtime and Docker context /// on this Mac (#309), the hosts config records adopted into theirs at each reload, and one sweep -/// per launch for what no record names. +/// per launch for what no record names. A boxd host is reached by name through a `BoxdHostDriver`, +/// never adopted or swept (#356). final class RemoteHosts: @unchecked Sendable { static let shared = RemoteHosts() private let lock = NSLock() - /// Which driver a host is in: its runtime, and for Docker the context it names (nil names none). - struct DriverKey: Hashable, Sendable { - var runtime: RemoteWorkrooms.Runtime = .docker - var context: String? = nil + /// Which driver a host is in (#309, #356): a container runtime on this Mac, and for Docker the + /// context it names (nil names none); or a boxd org (nil is the account's own) and account. + enum DriverKey: Hashable, Sendable { + case container(RemoteWorkrooms.Runtime, context: String?) + case boxd(org: String?, account: String?) - /// The driver a recorded host is in, or nil for a descriptor of no container runtime. + /// The driver a recorded host is in, or nil for a descriptor of no driver this build knows. init?(_ descriptor: HostDescriptor) { + if descriptor.driver == RemoteWorkrooms.boxdDriver { + self = .boxd(org: descriptor.org, account: descriptor.account) + return + } guard let runtime = RemoteWorkrooms.Runtime(rawValue: descriptor.driver ?? "") else { return nil } @@ -452,16 +592,46 @@ final class RemoteHosts: @unchecked Sendable { } init(runtime: RemoteWorkrooms.Runtime = .docker, context: String? = nil) { - self.runtime = runtime // Apple's runtime has no contexts. - self.context = runtime == .docker ? context : nil + self = .container(runtime, context: runtime == .docker ? context : nil) + } + + /// Whether the key names a boxd account rather than a container runtime. + var isBoxd: Bool { + if case .boxd = self { return true } + return false + } + + /// The container runtime, or nil for boxd. + var runtime: RemoteWorkrooms.Runtime? { + if case .container(let runtime, _) = self { return runtime } + return nil + } + + /// The Docker context a container key names; nil for none, and for boxd. + var context: String? { + if case .container(_, let context) = self { return context } + return nil + } + + /// A boxd key's org and account; nil for a container key. + var org: String? { + if case .boxd(let org, _) = self { return org } + return nil } + var account: String? { + if case .boxd(_, let account) = self { return account } + return nil + } + + /// Where this key's workrooms are made. + var place: RemoteWorkrooms.Place { runtime.map(RemoteWorkrooms.Place.container) ?? .boxd } } /// The drivers made so far. - private var made: [DriverKey: ContainerHostDriver] = [:] - /// The runtime each host config records is on, whether or not a driver holds it. - private var runtimes: [UUID: RemoteWorkrooms.Runtime] = [:] + private var made: [DriverKey: any HostTerminalDriver] = [:] + /// The driver each host config records it in, whether or not that driver has been made. + private var keys: [UUID: DriverKey] = [:] /// Workroom hosts whose git credentials come through the Mac's relay (#309). private var relayed: Set = [] private var swept = false @@ -472,6 +642,24 @@ final class RemoteHosts: @unchecked Sendable { private var connecting: [HostID: (task: Task, starts: Bool)] = [:] /// When each host's last attempt failed, which answers for it for `retryAfter`. private var failedAt: [HostID: ContinuousClock.Instant] = [:] + /// When boxd last said each host was asleep, which answers for it for `retryAfter` (#356). + private var asleepAt: [HostID: ContinuousClock.Instant] = [:] + /// boxd hosts the app let go of because they reported IDLE while not selected (#356): boxd counts + /// inbound traffic as activity, so a connection's ssh keepalives and the badge's polls kept an + /// idle box awake for good (measured, eng review D8). Each with when it was let go of: a background + /// read never reconnects one inside `revisitAfter(_:)` of that; selecting or opening its + /// workroom, or a click (`wake`), does. + private var letGoAt: [HostID: ContinuousClock.Instant] = [:] + /// When the app last connected each host, which an IDLE reading must outlast to let it go. + private var connectedAt: [HostID: ContinuousClock.Instant] = [:] + /// How many BUSY readings each host has reported, so a let-go that began on an older IDLE one + /// gives way to a BUSY reported while it waited. + private var busyReports: [HostID: Int] = [:] + /// Each boxd host's idle window as its provider last said, read on connect. + private var idleWindows: [HostID: TimeInterval] = [:] + /// The host of the workroom selected in each window (one `AppStore` per window), none of which + /// is ever let go of. + private var selectedHosts: [AnyHashable: HostID] = [:] /// Hosts of workrooms the user has opened this launch (`activate`), whose connect starts a /// stopped container first. private var activated: Set = [] @@ -484,12 +672,14 @@ final class RemoteHosts: @unchecked Sendable { private let relayHost: (@Sendable (HostID) async throws -> Void)? /// `connect`'s agent connection, bootstrap included; nil in the app (`AgentBootstrap.connect`). private let connectAgent: - (@Sendable (HostID, ContainerHostDriver) async throws -> AgentVCSConnection)? + (@Sendable (HostID, any HostTerminalDriver) async throws -> AgentVCSConnection)? private let isConnected: (@Sendable (HostID) async -> Bool)? + /// What boxd says of a boxd host (`BoxdHostDriver.presence`); nil in the app. + private let presenceSeam: (@Sendable (HostID) async -> BoxdHostDriver.Presence?)? private let startHost: (@Sendable (HostID) async throws -> Void)? private let now: @Sendable () -> ContinuousClock.Instant /// `adopt`'s seams, nil in the app: making a key's driver, and sweeping one. - private let makeDriver: (@Sendable (DriverKey) throws -> ContainerHostDriver)? + private let makeDriver: (@Sendable (DriverKey) throws -> any HostTerminalDriver)? private let sweepDriver: (@Sendable (ContainerHostDriver, Set, Set) async -> [String])? @@ -498,12 +688,14 @@ final class RemoteHosts: @unchecked Sendable { isConnected: (@Sendable (HostID) async -> Bool)? = nil, startHost: (@Sendable (HostID) async throws -> Void)? = nil, now: @escaping @Sendable () -> ContinuousClock.Instant = { .now }, - makeDriver: (@Sendable (DriverKey) throws -> ContainerHostDriver)? = nil, + makeDriver: (@Sendable (DriverKey) throws -> any HostTerminalDriver)? = nil, sweepDriver: (@Sendable (ContainerHostDriver, Set, Set) async -> [String])? = nil, relayHost: (@Sendable (HostID) async throws -> Void)? = nil, - connectAgent: (@Sendable (HostID, ContainerHostDriver) async throws -> AgentVCSConnection)? = - nil + connectAgent: (@Sendable (HostID, any HostTerminalDriver) async throws -> AgentVCSConnection)? = + nil, + presence: (@Sendable (HostID) async -> BoxdHostDriver.Presence?)? = nil ) { + self.presenceSeam = presence self.connectHost = connectHost self.relayHost = relayHost self.connectAgent = connectAgent @@ -522,22 +714,44 @@ final class RemoteHosts: @unchecked Sendable { private static let logger = Logger( subsystem: "com.developwithstyle.workroom", category: "RemoteWorkrooms") - /// The driver for `key`, made on first use: it needs that runtime's CLI and this Mac's ssh key. - /// The default key is Docker naming no context, as every driver did before #309. Each driver - /// writes its hosts' ssh files under their own IDs, so they share one directory. - func driver(_ key: DriverKey = DriverKey()) throws -> ContainerHostDriver { + /// The driver for `key`, made on first use. A container driver needs that runtime's CLI and this + /// Mac's ssh key; a boxd driver reaches its hosts through the ssh details the boxd CLI wrote, so + /// making one never fails (#356). The default key is Docker naming no context, as every driver + /// did before #309. Each driver writes its hosts' ssh files under their own IDs, so they share + /// one directory. + func driver(_ key: DriverKey = DriverKey()) throws -> any HostTerminalDriver { try lock.withLock { if let made = made[key] { return made } - let driver = - try makeDriver?(key) - ?? ContainerHostDriver( - hosts: [:], directory: Self.directory.appendingPathComponent("hosts", isDirectory: true), - provisioning: try Self.provisioning(key)) + let hosts = Self.directory.appendingPathComponent("hosts", isDirectory: true) + let driver: any HostTerminalDriver + if let makeDriver { + driver = try makeDriver(key) + } else if case .boxd(let org, let account) = key { + let cli = Self.boxdExecutable() + driver = BoxdHostDriver( + configuration: BoxdHostDriver.Configuration( + cli: URL(fileURLWithPath: cli ?? Self.boxdCandidates[0]), org: org, account: account), + directory: hosts) + // Not kept while the CLI is missing, so installing it later takes without a relaunch. + guard cli != nil else { return driver } + } else { + driver = ContainerHostDriver( + hosts: [:], directory: hosts, provisioning: try Self.provisioning(key)) + } made[key] = driver return driver } } + /// `key`'s driver, which must be a container runtime's: adopting, sweeping, starting and naming + /// contexts are theirs alone. + private func containerDriver(_ key: DriverKey = DriverKey()) throws -> ContainerHostDriver { + guard let driver = try driver(key) as? ContainerHostDriver else { + throw HostDriverError.invalidConfiguration("\(key) is not a container runtime") + } + return driver + } + /// Whether this call runs the launch's one sweep. A call that isn't allowed leaves it for the /// next. func claimSweep(allowed: Bool) -> Bool { @@ -556,9 +770,14 @@ final class RemoteHosts: @unchecked Sendable { var sweepHeld: Bool { lock.withLock { held && !swept } } /// The driver that holds host `id`, for a pane, which must not probe Docker: nil when no driver - /// made so far has it. - func existingDriver(holding id: UUID) -> ContainerHostDriver? { - lock.withLock { made.values.first { $0.record(of: .remote(id)) != nil } } + /// made so far has it. A boxd host's driver is made here if need be: it reaches the host by name, + /// with nothing to adopt and no runtime to probe (#356). + func existingDriver(holding id: UUID) -> (any HostTerminalDriver)? { + let key = lock.withLock { keys[id] } + if let key, case .boxd = key { return try? driver(key) } + return lock.withLock { + made.values.first { ($0 as? ContainerHostDriver)?.record(of: .remote(id)) != nil } + } } /// Takes on every host `projects` record, each into the driver for its Docker context, so their @@ -578,7 +797,7 @@ final class RemoteHosts: @unchecked Sendable { let bases = Set(projects.flatMap { $0.host?.allBases.compactMap(\.id) ?? [] }) let already = lock.withLock { for descriptor in recorded { - if let id = descriptor.id, let key = DriverKey(descriptor) { runtimes[id] = key.runtime } + if let id = descriptor.id, let key = DriverKey(descriptor) { keys[id] = key } // A base never has git ask for credentials; only its workrooms are relayed. if let id = descriptor.id, descriptor.isRelayed, !bases.contains(id) { relayed.insert(id) } } @@ -587,11 +806,17 @@ final class RemoteHosts: @unchecked Sendable { guard !recorded.isEmpty || !already.isEmpty else { return } // A descriptor with no container record yet goes to its runtime's driver that names no // context, where the one driver before #309 would have had it. - let keys = already.union(recorded.compactMap(DriverKey.init)) + // A boxd host needs no adopting: its driver reaches it by name. Nor is boxd swept: machine + // names carry no build, so one build's sweep would take another's live workrooms (#284). + // gstack-shortcut(dec-boxd-no-sweep): a crashed create can leave a paid boxd machine running, + // upgrade when #284 makes sweeps safe and boxd machine names carry the build. + let containerKeys = already.union(recorded.compactMap(DriverKey.init)).filter { + $0.runtime != nil + } var drivers: [ContainerHostDriver] = [] - for key in keys { + for key in containerKeys { let driver: ContainerHostDriver - do { driver = try self.driver(key) } catch { + do { driver = try containerDriver(key) } catch { // That runtime isn't installed; another's may be. Self.logger.error("remote hosts: \(error.localizedDescription, privacy: .public)") continue @@ -626,26 +851,43 @@ final class RemoteHosts: @unchecked Sendable { } } + /// The driver key a new workroom at `place` wants: a container runtime's (`key(for:)` below), or + /// for boxd the org and account it is signed in to now (#356). + func key(for place: RemoteWorkrooms.Place) async throws -> DriverKey { + switch place { + case .container(let runtime): return try await key(for: runtime) + case .boxd: + // The org and account boxd is signed in to now, which a new base is made in. Throws, naming + // `boxd auth login`, when nobody is signed in. + guard let cli = Self.boxdExecutable() else { throw RemoteWorkrooms.Failure.boxdNotInstalled } + let account = try await BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: cli)), directory: Self.directory + ).signedIn() + return .boxd(org: account.activeOrg, account: try account.requiredUserID()) + } + } + /// The driver key a new workroom on `runtime` wants (#309): Apple's runtime has one, and Docker's /// is the context the CLI uses now, which a new base is pinned to. `RemoteWorkrooms.base(in:for:)` /// then finds the project's base there, if it has one. func key(for runtime: RemoteWorkrooms.Runtime) async throws -> DriverKey { guard runtime == .docker else { return DriverKey(runtime: runtime) } - return DriverKey(runtime: .docker, context: try await driver().currentContext()) + return DriverKey(runtime: .docker, context: try await containerDriver().currentContext()) } /// The sequence's environment over `key`'s driver, signed in as this Mac. @MainActor func environment(_ key: DriverKey = DriverKey()) throws -> ( - ContainerHostDriver, RemoteProvisioning.Environment + any HostTerminalDriver, RemoteProvisioning.Environment ) { // Signed out of Codaset, a local container workroom's git credentials come from the Mac's own - // `gh` instead (#309): every runtime here is a local one. + // `gh` instead (#309). A remote host's never do: it must fetch and push with the Mac closed, + // so it takes the broker's repo-scoped tokens only (OQ20, #356). let client = BrokerSession.shared.client() let driver = try driver(key) var environment = RemoteProvisioning.Environment( - driver: driver, agentSocket: RemoteWorkrooms.agentSocket, client: client, - gitHubToken: CredentialRelay.gitHubToken) + driver: driver, agentSocket: Self.agentSocket(of: driver), client: client, + gitHubToken: key.isBoxd ? nil : CredentialRelay.gitHubToken) #if DEBUG // A Debug agent reaches this Mac's Codaset through a listener on its host, which lives on // the host's service connection (`BrokerReverseForwards`), so that connection comes first. @@ -667,11 +909,12 @@ final class RemoteHosts: @unchecked Sendable { guard try RemoteWorkrooms.checkDeletable(hosts) else { return nil } var environments: [DriverKey: RemoteProvisioning.Environment] = [:] for host in hosts where RemoteWorkrooms.isLive(host) { - let key = DriverKey(host) ?? DriverKey() + let key = try Self.deletionKey(host) if environments[key] == nil { environments[key] = try environment(key).1 } // Taking a box down needs it on its driver, whatever a reload adopted: with previews off it - // adopted nothing, and the box would read as unknown. - let driver = try driver(key) + // adopted nothing, and the box would read as unknown. A boxd driver needs nothing adopted. + guard key.runtime != nil else { continue } + let driver = try containerDriver(key) guard let id = host.id, let record = host.container, driver.record(of: .remote(id)) == nil else { continue } // Whether it is a base sticks for the launch: a later create derives from it. @@ -682,24 +925,36 @@ final class RemoteHosts: @unchecked Sendable { return environments.isEmpty ? nil : Deletion(environments: environments) } - /// The environments a delete takes hosts down with, one per runtime and Docker context. + /// The driver a host to take down is in. A descriptor that names no driver predates the field + /// and is Docker's; one naming a driver this build doesn't know is refused, never taken down + /// with another driver (a boxd host once fell back to Docker here). + static func deletionKey(_ host: HostDescriptor) throws -> DriverKey { + if let key = DriverKey(host) { return key } + guard host.driver == nil else { + throw HostDriverError.invalidConfiguration( + "this build doesn't know how to take down a \(host.driver ?? "") host") + } + return DriverKey() + } + + /// The environments a delete takes hosts down with, one per driver key. struct Deletion { let environments: [DriverKey: RemoteProvisioning.Environment] /// The environment for `host`, or nil when it isn't live and needs none. func environment(for host: HostDescriptor) -> RemoteProvisioning.Environment? { - environments[DriverKey(host) ?? DriverKey()] + (try? RemoteHosts.deletionKey(host)).flatMap { environments[$0] } } } /// The app's service connection to `host` (`HostConnectionManager`), with its agent bootstrapped /// first. One already there is kept. - func connect(_ host: HostID, driver: ContainerHostDriver) async throws { + func connect(_ host: HostID, driver: any HostTerminalDriver) async throws { _ = try await HostConnectionManager.shared.connectIfDisconnected(host: host) { [connectAgent] in if let connectAgent { return try await connectAgent(host, driver) } return try await AgentBootstrap.connect( - host: host, driver: driver, socket: RemoteWorkrooms.agentSocket) + host: host, driver: driver, socket: Self.agentSocket(of: driver)) } // Its ceiling prompts are watched from here (#257): the agent was just handed this Mac's // ask-at-ceiling setting, and a prompt nobody shows lets the box sleep under a running job. @@ -707,6 +962,33 @@ final class RemoteHosts: @unchecked Sendable { await MainActor.run { WakefulnessModel.model(forHost: id).hostSleeps = driver.traits.sleepsWhenIdle } + // A provider timer shorter than the heartbeat can beat sleeps a box under a running job + // however healthy the heartbeat is, so the badge says so (#356). Read on every connect, as + // a machine's timers can change; not awaited, as the CLI call must not hold up the connect. + if let boxd = driver as? BoxdHostDriver { + // Its verdict is polled while it is connected, whatever is on screen, so an IDLE one lets + // it go (`observed`). + // Stamped before the poll starts, so its first reading already sees the grace. + lock.withLock { connectedAt[host] = now() } + let model = await MainActor.run { () -> WakefulnessModel in + let model = WakefulnessModel.model(forHost: id) + model.pollWhileConnected() + return model + } + Task { + // A failed read keeps the window last read, rather than forgetting it; an answer with no + // timer set (or none boxd would use) clears it. + let window: TimeInterval? + do { window = try await boxd.idleWindow(host) } catch { return } + lock.withLock { idleWindows[host] = window } + // Given to the model this connect made, while it is still the host's: a delete across + // the read forgot it, and looking the host up again would make it anew, watching for + // prompts for good. + await MainActor.run { + if WakefulnessModel.Hosts.shared.models[id] === model { model.idleWindow = window } + } + } + } } // A relayed workroom's git asks this Mac for credentials (#309): its listener goes with the // connection, and its secret with this launch, so it is set up again on every connect. A @@ -719,7 +1001,7 @@ final class RemoteHosts: @unchecked Sendable { /// Sets up `host`'s credential relay on its connection, up to `attempts` times. One that never /// takes is left pending, for a later connection check to try again. func installRelay( - _ host: HostID, attempts: Int, driver: ContainerHostDriver? = nil + _ host: HostID, attempts: Int, driver: (any HostTerminalDriver)? = nil ) async { guard case .remote(let id) = host else { return } for attempt in 1...attempts { @@ -727,9 +1009,10 @@ final class RemoteHosts: @unchecked Sendable { if let relayHost { try await relayHost(host) } else { + let held = try driver ?? heldDriver(host) try await CredentialRelay.shared.install( - on: host, driver: try driver ?? heldDriver(host), - agentBinary: AgentBootstrap.binary(besideSocket: RemoteWorkrooms.agentSocket)) + on: host, driver: held, + agentBinary: AgentBootstrap.binary(besideSocket: Self.agentSocket(of: held))) } _ = lock.withLock { relayPending.removeValue(forKey: id) } return @@ -760,8 +1043,22 @@ final class RemoteHosts: @unchecked Sendable { /// Any failure is `RepositoryRoutingError.unavailable`, which every panel shows as the host being /// out of reach. Untyped, a stopped host's ssh failure read as "Not a repository". The cause is /// logged. - func ensureConnected(_ host: HostID) async throws { + /// + /// `wake` for a click (closing a pane, Keep awake): the user asked for this host, so a boxd box + /// let go of or asleep is connected, and woken, rather than refused as a background read is. + func ensureConnected(_ host: HostID, wake: Bool = false) async throws { guard case .remote = host else { return } + // A click is tried whatever came of the last attempt, as an opened host is: waking a box can + // fail once while it resumes, and a second click must not wait out the window. Cleared before + // the connection is looked at, so a click that lands while a let-go's disconnect is in flight + // still takes the host back. + if wake { + lock.withLock { + letGoAt[host] = nil + asleepAt[host] = nil + failedAt[host] = nil + } + } let up: Bool if let isConnected { up = await isConnected(host) @@ -780,11 +1077,35 @@ final class RemoteHosts: @unchecked Sendable { return } // An opened host is tried whatever a probe of it found: its connect starts it. - if !lock.withLock({ activated.contains(host) }), - let failed = lock.withLock({ failedAt[host] }), now() - failed < Self.retryAfter + let opened = lock.withLock { activated.contains(host) } + // A selected workroom's box is reached as an opened one is: it is on screen, and a Mac waking + // from sleep must not leave it reading as asleep until it is selected again. + let selected = lock.withLock { selectedHosts.values.contains(host) } + if !wake, !opened, let failed = lock.withLock({ failedAt[host] }), + now() - failed < Self.retryAfter { throw RepositoryRoutingError.unavailable(host) } + // A boxd box wakes on an ssh login, so a background read must not connect to one that is + // asleep: every status sweep would wake every box and keep it billing (#356). Only opening the + // workroom, or a click (`wake`), wakes it. When boxd can't say, the read leaves it be too. + // Nor does one reconnect a box let go of as idle until its idle window has passed, which would + // hold it awake again: then boxd is asked again, and a box it says is still awake has something + // on it holding it (work resumed), so it is connected again and its ceiling prompts are heard. + // One boxd says is gone is reported as such, not as asleep. + if !opened, !wake, !selected, let since = lock.withLock({ letGoAt[host] }) { + guard now() - since >= revisitAfter(host) else { throw RepositoryRoutingError.idle(host) } + _ = lock.withLock { letGoAt.removeValue(forKey: host) } + } + if !opened, !wake, !selected { + switch try await boxdPresence(host) { + case .asleep: throw RepositoryRoutingError.asleep(host) + case .gone: + lock.withLock { failedAt[host] = now() } + throw RepositoryRoutingError.gone(host) + case .awake, nil: break // nil: not a boxd host + } + } do { let connect = try connectHost ?? { [driver = try heldDriver(host)] in @@ -794,7 +1115,9 @@ final class RemoteHosts: @unchecked Sendable { // every remote workroom, and would otherwise start them all. let start = lock.withLock({ activated.contains(host) }) - ? try startHost ?? { [driver = try heldDriver(host)] in try await driver.startIfStopped($0) + ? try startHost ?? { [driver = try heldDriver(host)] in + // A boxd box needs no start: the connect's ssh login wakes it. + try await (driver as? ContainerHostDriver)?.startIfStopped($0) } : nil let task = lock.withLock { () -> Task in @@ -815,7 +1138,10 @@ final class RemoteHosts: @unchecked Sendable { } defer { lock.withLock { if connecting[host]?.task == task { connecting[host] = nil } } } try await task.value - lock.withLock { failedAt[host] = nil } + lock.withLock { + failedAt[host] = nil + connectedAt[host] = now() + } } catch is CancellationError { throw CancellationError() } catch { @@ -827,27 +1153,159 @@ final class RemoteHosts: @unchecked Sendable { } /// The user opened a workroom on `host` (#309): its next connect starts its container if it is - /// stopped, and a failure from before, such as a status probe of the stopped container, no longer - /// holds that attempt back. + /// stopped (or wakes its boxd box), and a failure from before, such as a status probe of the + /// stopped container, no longer holds that attempt back. func activate(_ host: HostID) { lock.withLock { activated.insert(host) failedAt[host] = nil + asleepAt[host] = nil + letGoAt[host] = nil + } + } + + /// The workroom selected in `window` is on `host` (nil for one on this Mac, or a window closing). + /// A host selected in any window is never let go of (#356). + func select(_ host: HostID?, in window: AnyHashable = "default") { + lock.withLock { + selectedHosts[window] = host + if let host { letGoAt[host] = nil } + } + } + + /// A connected host's agent reported its published verdict (`WakefulnessModel`). A boxd host + /// that is not busy (IDLE, its service stopped, or an untrusted reading past its ceiling), not + /// selected, forwarding no port and with no pane attached is let go of: its connection is closed, + /// so nothing the app sends reaches the box and boxd's idle timer can sleep it (#356). An IDLE + /// that comes of an unanswered prompt lets it sleep the same way. A pane's own ssh holds its box + /// awake whatever the app does, so letting go of that one would only blank its status. Nor is a + /// host let go of within `letGoGrace` of its connect, so the read that connected it is answered + /// first. Returns whether it was let go of. + @discardableResult + func observed(_ host: HostID, busy: Bool) async -> Bool { + guard case .remote(let id) = host else { return false } + guard !busy else { + lock.withLock { busyReports[host, default: 0] += 1 } + return false + } + let busySeen = lock.withLock { busyReports[host, default: 0] } + // What holds the box whatever the app does: a pane's ssh, or a port forward. Asked twice, before + // and after the waits, by the one question. + @Sendable func held() async -> Bool { + await MainActor.run { + PortForwardingModel.hasForwards(host) + || PersistentSessionService.shared.hasAttachedPane(on: host) + } + } + let eligible = lock.withLock { () -> Bool in + guard let key = keys[id], case .boxd = key else { return false } + if let connected = connectedAt[host], now() - connected < Self.letGoGrace { return false } + return !selectedHosts.values.contains(host) && letGoAt[host] == nil + } + guard eligible, await !held() else { return false } + // Checked again across the wait: a window may have selected it meanwhile. + let letGo = lock.withLock { () -> Bool in + guard !selectedHosts.values.contains(host), letGoAt[host] == nil else { return false } + letGoAt[host] = now() + return true + } + guard letGo else { return false } + let lease = await HostConnectionManager.shared.snapshot(for: host).lease + await MainActor.run { WakefulnessModel.Hosts.shared.models[id]?.stopPollingWhileConnected() } + // A pane or a port forward may have come across those waits. + let heldNow = await held() + // The last check before the disconnect, after every wait: a selection, a click, a pane, a + // forward or a newer BUSY reading across them takes the host back, which keeps its connection + // and resumes its poll. + let committed = lock.withLock { () -> Bool in + if heldNow || busyReports[host, default: 0] != busySeen { letGoAt[host] = nil } + return letGoAt[host] != nil && !selectedHosts.values.contains(host) + } + guard committed else { + await MainActor.run { + WakefulnessModel.Hosts.shared.models[id]?.pollWhileConnected(connected: false) + } + return false + } + // Checked once more on the manager, where the disconnect happens: a selection, a click or a pane + // that took the host back across the hop to it keeps the connection it already found up. + let takenBack: @Sendable () -> Bool = { [self] in + lock.withLock { letGoAt[host] == nil || selectedHosts.values.contains(host) } } + if let lease, + await !HostConnectionManager.shared.disconnect(lease, unless: takenBack) + { + await MainActor.run { + WakefulnessModel.Hosts.shared.models[id]?.pollWhileConnected(connected: false) + } + return false + } + Self.logger.notice("let go of idle boxd host \(id, privacy: .public)") + return true + } + + /// Whether the app let go of `host` as idle (`observed`). + func isLetGo(_ host: HostID) -> Bool { lock.withLock { letGoAt[host] != nil } } + + /// How long after a connect an IDLE reading is ignored: one `WakefulnessModel.pollInterval`, so + /// the read that connected the host finishes before its connection is closed. + static let letGoGrace = WakefulnessModel.pollInterval + + /// How long a host let go of is left before boxd is asked about it again: its idle window, by + /// when boxd sleeps a box nothing holds, plus a `retryAfter`. ponytail: an unknown window is + /// taken as an hour; a box with a longer one is reconnected, and so held awake, once an hour. + func revisitAfter(_ host: HostID) -> Duration { + .seconds(lock.withLock { idleWindows[host] } ?? Self.unknownIdleWindow) + Self.retryAfter + } + + static let unknownIdleWindow: TimeInterval = 3600 + + /// What boxd says of `host` (awake, asleep, or gone), or nil only when it isn't a boxd host. A + /// boxd that can't say is taken as asleep, logged, so a background read leaves the box be. boxd + /// is asked at most once a `retryAfter` about a box it called asleep. + // ponytail: one `machine get` per host per window; one `machine list` per sweep is the upgrade + // if many boxd workrooms make it slow. + private func boxdPresence(_ host: HostID) async throws -> BoxdHostDriver.Presence? { + guard case .remote(let id) = host, let key = lock.withLock({ keys[id] }), case .boxd = key + else { return nil } + if let slept = lock.withLock({ asleepAt[host] }), now() - slept < Self.retryAfter { + return .asleep + } + let presence: BoxdHostDriver.Presence? + if let presenceSeam { + presence = await presenceSeam(host) + } else { + presence = await (try driver(key) as? BoxdHostDriver)?.presence(host) ?? nil + } + lock.withLock { asleepAt[host] = presence == .asleep || presence == nil ? now() : nil } + // boxd could not say (no CLI, a timeout, output it did not expect, the read cancelled): a + // background read leaves the box be rather than wake it, and says why. Opening the workroom, + // selecting it or a click still connects. Remembered as asleep is, so an outage asks boxd once a + // `retryAfter` rather than on every read. + guard let presence else { + Self.logger.notice( + "boxd could not say whether host \(id, privacy: .public) is asleep; left it be") + return .asleep + } + return presence } /// Whether `host`'s runtime is missing from this Mac, so nothing can be running on it: the /// runtime config records it on, or with none recorded, every runtime. + /// + /// Never for a boxd host (#356): it runs on boxd whether or not this Mac has the boxd CLI, and its + /// panes reach it over ssh without one. func runtimeIsMissing(for host: HostID) -> Bool { guard case .remote(let id) = host else { return false } - let runtime = lock.withLock { runtimes[id] } - return (runtime.map { [$0] } ?? RemoteWorkrooms.Runtime.allCases).allSatisfy { + let key = lock.withLock { keys[id] } + if let key, case .boxd = key { return false } + return (key?.runtime.map { [$0] } ?? RemoteWorkrooms.Runtime.allCases).allSatisfy { Self.executable(for: $0) == nil } } /// The driver holding `host`, which a reload adopted it into (`adopt`) or which made it. - private func heldDriver(_ host: HostID) throws -> ContainerHostDriver { + private func heldDriver(_ host: HostID) throws -> any HostTerminalDriver { guard case .remote(let id) = host, let driver = existingDriver(holding: id) else { throw HostDriverError.unknownHost(host) } @@ -872,6 +1330,25 @@ final class RemoteHosts: @unchecked Sendable { /// Where Apple's `container` is: its installer's, or Homebrew's. static let appleCandidates = ["/usr/local/bin/container", "/opt/homebrew/bin/container"] + /// Where the boxd CLI is (#356): its installer's (`~/.local/bin`), or Homebrew's. + static var boxdCandidates: [String] { + [ + FileManager.default.homeDirectoryForCurrentUser.appendingPathComponent(".local/bin/boxd") + .path, + "/opt/homebrew/bin/boxd", "/usr/local/bin/boxd", + ] + } + + /// The boxd CLI on this Mac, or nil when it isn't installed. + static func boxdExecutable() -> String? { + boxdCandidates.first { FileManager.default.isExecutableFile(atPath: $0) } + } + + /// The agent's socket on `driver`'s hosts. + static func agentSocket(of driver: any HostDriver) -> String { + (driver as? BoxdHostDriver)?.configuration.agentSocket ?? RemoteWorkrooms.agentSocket + } + /// `runtime`'s CLI on this Mac, or nil when it isn't installed. static func executable(for runtime: RemoteWorkrooms.Runtime) -> String? { (runtime == .docker ? runtimeCandidates : appleCandidates).first { @@ -895,9 +1372,12 @@ final class RemoteHosts: @unchecked Sendable { } private static func provisioning(_ target: DriverKey) throws -> ContainerHostDriver.Provisioning { - let resources = resources(for: target.runtime) - guard let runtime = executable(for: target.runtime) else { - throw target.runtime == .docker + guard let runtimeKind = target.runtime else { + throw HostDriverError.invalidConfiguration("\(target) is not a container runtime") + } + let resources = resources(for: runtimeKind) + guard let runtime = executable(for: runtimeKind) else { + throw runtimeKind == .docker ? RemoteWorkrooms.Failure.noDocker : RemoteWorkrooms.Failure.noAppleContainer } let key = try clientKey() @@ -909,7 +1389,7 @@ final class RemoteHosts: @unchecked Sendable { agentSocket: RemoteWorkrooms.agentSocket, // Per build, so a Dev app's sweep never takes a Nightly app's hosts, nor the other way. labels: ["workroom.provisioner=\(RemoteWorkrooms.provisioner)"], context: target.context, - dialect: target.runtime == .docker ? .docker : .apple, cpus: resources.cpus, + dialect: runtimeKind == .docker ? .docker : .apple, cpus: resources.cpus, memory: resources.memory) } diff --git a/macapp/WorkroomApp/Core/RepositoryLocation.swift b/macapp/WorkroomApp/Core/RepositoryLocation.swift index 0c6c8dfb9..db18f15e0 100644 --- a/macapp/WorkroomApp/Core/RepositoryLocation.swift +++ b/macapp/WorkroomApp/Core/RepositoryLocation.swift @@ -10,15 +10,34 @@ enum RepositoryRoutingError: Error, Equatable, Sendable, LocalizedError, CustomS case invalidPath(String) case mixedHosts case unavailable(HostID) + /// A boxd box asleep, or one boxd refused to answer about (signed out, another account), which a + /// background read leaves be: connecting would wake it (#356). + case asleep(HostID) + /// A boxd box the app let go of as idle, which may still be awake until boxd sleeps it (#356). + case idle(HostID) + /// A boxd machine boxd says is not found: deleted, or in another org or account (#356). + case gone(HostID) case registrationRequired var description: String { errorDescription ?? "Repository unavailable" } + /// What a read of an asleep boxd host says (#356), here and on the Changes panel. + static let asleepMessage = + "The box is asleep, or boxd could not say (check the boxd CLI and `boxd auth`). Open the " + + "workroom to reach it." + /// What a read of a boxd host the app let go of as idle says (#356). + static let idleMessage = + "The box is idle, so Workroom disconnected to let it sleep. Open the workroom to reach it." + var errorDescription: String? { switch self { case .invalidPath(let path): return "Invalid repository path: \(path)" case .mixedHosts: return "Working and shared repositories must be on the same host." case .unavailable: return "Repository service unavailable." + case .asleep: return Self.asleepMessage + case .idle: return Self.idleMessage + case .gone: + return "This workroom's boxd machine is gone, or in a boxd org or account that isn't active." case .registrationRequired: return "Reload projects to register this repository before changing it." } diff --git a/macapp/WorkroomApp/Core/Session/AgentBootstrap.swift b/macapp/WorkroomApp/Core/Session/AgentBootstrap.swift index b5a67b878..8ed3f848c 100644 --- a/macapp/WorkroomApp/Core/Session/AgentBootstrap.swift +++ b/macapp/WorkroomApp/Core/Session/AgentBootstrap.swift @@ -115,10 +115,17 @@ enum AgentBootstrap { } /// The bootstrap, then the connection: the one way to a remote host's services. + /// + /// `watched` for a connection whose host has its ceiling prompts watched (`RemoteHosts.connect`), + /// which hands the agent this Mac's settings. An unwatched one (provisioning's) sends none: the + /// settings are the whole agent's, so its `ask: false` could land after a watched connection's + /// `ask: true` and switch prompting off under it (#356). A base that only provisioning ever + /// connects keeps the agent's default, which never asks. static func connect( host: HostID, driver: any HostDriver, socket: String, agent: (String) -> URL? = PersistentSessionPaths.linuxAgentURL(architecture:), - handOff: Bool = AgentHandOff.isEnabled, resources: URL? = GhosttyResources.bundledURL + handOff: Bool = AgentHandOff.isEnabled, resources: URL? = GhosttyResources.bundledURL, + watched: Bool = true ) async throws -> AgentVCSConnection { let outcome = try await ensure( host: host, driver: driver, socket: socket, agent: agent, handOff: handOff, @@ -126,6 +133,7 @@ enum AgentBootstrap { logger.notice("agent bootstrap: \(String(describing: outcome), privacy: .public)") let connection = try await AgentVCSConnection.connect( host: host, stream: try await driver.openStream(to: host)) + guard watched else { return connection } // Not awaited: a slow reply (its timeout is 5 s) must not hold up the connect, the cost // `AgentVCSConnection.wakefulness()` declined a connect-time probe over. // Ask mode only where the box can sleep: its prompt says it may. diff --git a/macapp/WorkroomApp/Core/Session/PersistentSessionService.swift b/macapp/WorkroomApp/Core/Session/PersistentSessionService.swift index 9d52177e5..e049f562d 100644 --- a/macapp/WorkroomApp/Core/Session/PersistentSessionService.swift +++ b/macapp/WorkroomApp/Core/Session/PersistentSessionService.swift @@ -103,6 +103,11 @@ final class PersistentSessionService { /// it. Registered by whoever makes a pane for a remote workroom; nothing persists it yet, so a /// relaunch re-registers it (Phase 4). private var remoteSessions: [UUID: RemoteSession] = [:] + /// Remote sessions a pane has attached to, which its ssh holds open (#356), by the pane that + /// attached: each pane clears only its own mark, so one freed late cannot clear a live pane's of + /// the same session. A pane registers its session when it is made, but a restored one spawns + /// nothing until it enters a window, so a registration alone says nothing about the host. + private var attachedRemote: [UUID: Set] = [:] private struct RemoteSession { let host: HostID @@ -138,7 +143,8 @@ final class PersistentSessionService { // would defeat. Off the main actor: it looks at the file system. let noRuntime = await Task.detached { RemoteHosts.shared.runtimeIsMissing(for: host) }.value if noRuntime { return true } - try await RemoteHosts.shared.ensureConnected(host) + // A click: a box let go of as idle, or asleep, is woken to end the session (#356). + try await RemoteHosts.shared.ensureConnected(host, wake: true) return try await HostConnectionManager.shared.endSession(session, on: host) } } @@ -438,23 +444,57 @@ final class PersistentSessionService { /// Drops a remote session's registration without asking its host anything: for a pane whose /// workroom is being deleted, whose host goes with it (#283). - func forgetRemoteSession(_ sessionID: UUID) { remoteSessions.removeValue(forKey: sessionID) } + func forgetRemoteSession(_ sessionID: UUID) { + remoteSessions.removeValue(forKey: sessionID) + attachedRemote.removeValue(forKey: sessionID) + } /// The host a remote session's pane attaches to, or nil for a local session. func remoteHost(of sessionID: UUID) -> HostID? { remoteSessions[sessionID]?.host } + /// `pane`'s ssh to `sessionID` ended: its surface was freed or its process exited (#356). A + /// window closing does both without ending the session, so this, not `endSession`, is what + /// stops a pane holding its host. A reconnect attaches it again (`attachCommand`). + func paneDetached(_ sessionID: UUID, by pane: UUID) { + attachedRemote[sessionID]?.remove(pane) + if attachedRemote[sessionID]?.isEmpty == true { attachedRemote[sessionID] = nil } + } + + /// Whether a pane has attached to a session on `host`: its ssh holds the host's box awake (#356). + func hasAttachedPane(on host: HostID) -> Bool { + attachedRemote.keys.contains { remoteSessions[$0]?.host == host } + } + /// Whether a remote session's last attach was refused by its host for good (#241). func remoteHostRefusedLastAttach(_ sessionID: UUID) -> Bool { guard let remote = remoteSessions[sessionID] else { return false } return remote.driver.hostRefusedLastAttach(of: sessionID, on: remote.host) } - func attachCommand(forSession sessionID: UUID, restored: Bool = false) -> String? { + /// `pane`, when given, is the pane whose ssh will run the command: it holds the host until it + /// detaches (`paneDetached`). Nil means no pane runs it (a probe), so nothing is held. + func attachCommand( + forSession sessionID: UUID, restored: Bool = false, by pane: UUID? = nil + ) -> String? { if let remote = remoteSessions[sessionID] { do { - return try remote.driver.attachCommand( + let command = try remote.driver.attachCommand( to: remote.host, session: sessionID, workingDirectory: remote.workingDirectory, restored: restored, metadata: remote.metadata) + guard let pane else { return command } + attachedRemote[sessionID, default: []].insert(pane) + // A pane attaching is the user at that box, as a click is: a host the app let go of is + // taken back, so its badge and ceiling prompts are heard while the pane's ssh holds it. + let host = remote.host + if RemoteHosts.shared.isLetGo(host) { + Task { + // Only while a pane still holds it: one that exited or closed meanwhile holds nothing. + guard self.hasAttachedPane(on: host) else { return } + self.logger.notice("a pane attached to a host let go of; taking it back") + try? await RemoteHosts.shared.ensureConnected(host, wake: true) + } + } + return command } catch { logger.error( "no attach command for remote session \(sessionID.uuidString, privacy: .public): \(error)" @@ -629,6 +669,8 @@ final class PersistentSessionService { // running there, and without it a retry would ask the local agent (which "kills" an id it never // held) and a reattach would open a new session on this Mac. if let remote = remoteSessions[sessionID] { + // Its pane is closing, so nothing here holds the host any more, whatever the kill finds. + attachedRemote.removeValue(forKey: sessionID) let id = sessionID.uuidString logger.notice("ending remote session \(id, privacy: .public)") let killed: Bool diff --git a/macapp/WorkroomApp/Core/WorkroomStatus.swift b/macapp/WorkroomApp/Core/WorkroomStatus.swift index 87e25b21c..9fbb1a880 100644 --- a/macapp/WorkroomApp/Core/WorkroomStatus.swift +++ b/macapp/WorkroomApp/Core/WorkroomStatus.swift @@ -87,6 +87,8 @@ enum VCSStatusFailure: Equatable, Sendable { case timeout // probe exceeded its deadline (slow disk, index.lock contention) case busy // the repository refused the read as busy (lock contention) case staleWorkingCopy // the working tree changed while it was being read; a retry settles it + case asleep // a boxd box its provider put to sleep; a probe leaves it be (#356) + case idle // a boxd box the app let go of as idle, awake until boxd sleeps it (#356) } /// One changed path in the working tree, with its change kind (for the detail panel grouping). @@ -273,7 +275,7 @@ struct WorkroomStatus: Equatable, Sendable { case .unavailable, .registrationRequired, .missingPath, .notRepository, .busy, .staleWorkingCopy: return 1 - case .timeout, nil: break + case .timeout, .asleep, .idle, nil: break } if dirty == true { return 2 } return 0 @@ -305,6 +307,14 @@ enum VCSStatusPresentation { return StatusDot( symbol: "exclamationmark.triangle.fill", semantic: .conflict, accessibility: "conflicted") } + if s.failure == .asleep { + // Not a failure: the box is asleep, and a probe would have woken it. + return StatusDot(symbol: "moon.zzz", semantic: .neutral, accessibility: "asleep") + } + if s.failure == .idle { + // Nor this: the app disconnected from an idle box so it can sleep, which it may not have yet. + return StatusDot(symbol: "moon", semantic: .neutral, accessibility: "idle, not connected") + } if s.isUnknown { // gh-style "absent" never reaches here; only genuine probe failures render unknown. let why: String diff --git a/macapp/WorkroomApp/Core/WorkroomStatusResolver.swift b/macapp/WorkroomApp/Core/WorkroomStatusResolver.swift index a4b3e00c7..0888fcf2a 100644 --- a/macapp/WorkroomApp/Core/WorkroomStatusResolver.swift +++ b/macapp/WorkroomApp/Core/WorkroomStatusResolver.swift @@ -119,6 +119,10 @@ struct WorkroomStatusResolver: Sendable { } } catch RepositoryRoutingError.registrationRequired { status = WorkroomStatus(dirty: nil, failure: .registrationRequired) + } catch RepositoryRoutingError.asleep { + status = WorkroomStatus(dirty: nil, failure: .asleep) + } catch RepositoryRoutingError.idle { + status = WorkroomStatus(dirty: nil, failure: .idle) } catch is RepositoryRoutingError, is HostConnectionError { status = WorkroomStatus(dirty: nil, failure: .unavailable) } catch is VCSTimeoutError, is VCSCancellationError { diff --git a/macapp/WorkroomApp/Views/ChangesPanel.swift b/macapp/WorkroomApp/Views/ChangesPanel.swift index 16237cb1d..b5bba9dc4 100644 --- a/macapp/WorkroomApp/Views/ChangesPanel.swift +++ b/macapp/WorkroomApp/Views/ChangesPanel.swift @@ -406,7 +406,9 @@ struct WakefulnessBadge: View { var body: some View { Group { if let status = model.status, status.running { - let glyph = Self.glyph(for: status.display(hostSleeps: model.hostSleeps), theme: theme) + let glyph = Self.glyph( + for: status.display(hostSleeps: model.hostSleeps, idleWindow: model.idleWindow), + theme: theme) let image = Image(systemName: glyph.symbol) .font(.caption) .foregroundStyle(glyph.tint) @@ -431,7 +433,8 @@ struct WakefulnessBadge: View { } .help( Self.help( - for: status, settings: isLocal ? .current : nil, hostSleeps: model.hostSleeps) + for: status, settings: isLocal ? .current : nil, hostSleeps: model.hostSleeps, + idleWindow: model.idleWindow) ) .accessibilityLabel(glyph.label) .accessibilityIdentifier(identifier) @@ -457,24 +460,39 @@ struct WakefulnessBadge: View { // The one state that must not be softened: work is running and nothing is keeping the box awake. case .busyUnprotected: return ("bolt.slash.fill", theme.tokens.warning, "Busy but not kept awake") + // A stalled service's IDLE may be stale: not "idle", which reads as nothing to worry about. + case .unknown: + return ("questionmark.circle", theme.tokens.warning, "Status unknown") } } /// `settings` is this Mac's, to name a running agent's that differ; nil for a remote host's /// agent, which this Mac's settings never started. static func help( - for status: AgentWakefulness, settings: AgentWakefulnessSettings?, hostSleeps: Bool = true + for status: AgentWakefulness, settings: AgentWakefulnessSettings?, hostSleeps: Bool = true, + idleWindow: TimeInterval? = nil ) -> String { let awake = wakefulnessDuration(status.awakeSeconds).formatted( .units(allowed: [.hours, .minutes], width: .narrow)) + let shortWindow = hostSleeps && AgentWakefulness.isTooShort(idleWindow) + let window = + "Its provider sleeps it after \(Int(idleWindow ?? 0)) s idle on the network, sooner than the " + + "keep-awake heartbeat's once a minute can hold it. Raise the machine's auto-suspend timeout." var text: String - switch status.display(hostSleeps: hostSleeps) { + switch status.display(hostSleeps: hostSleeps, idleWindow: idleWindow) { case .idle: text = "This machine is idle." case .busy: text = "This machine is busy (awake \(awake))." case .busyPastCeiling: text = "This machine has been busy for \(awake), past its awake ceiling. Nothing has been " + "slept — this is a report. Click to keep it awake." + case .unknown: + text = + "This machine's agent has stopped checking on it, so whether it is busy is unknown, and " + + "nothing is keeping it awake: it may sleep once it has been idle on the network long " + + "enough." + case .busyUnprotected where shortWindow && !status.unprotected: + text = "This machine is busy (awake \(awake)), but it may sleep under the job." case .busyUnprotected: text = status.suppressed @@ -496,6 +514,7 @@ struct WakefulnessBadge: View { text += " It has been busy for \(awake), past its awake ceiling. Click to keep it awake." } } + if shortWindow { text += " " + window } if let settings, let mismatch = status.settingsMismatch(against: settings) { text += " " + mismatch } @@ -1024,6 +1043,8 @@ struct ChangesPanel: View { case .timeout: return "Status unavailable (timed out)." case .busy: return "Repository is busy — another VCS command is running." case .staleWorkingCopy: return "The working tree changed while it was read. Try again." + case .asleep: return RepositoryRoutingError.asleepMessage + case .idle: return RepositoryRoutingError.idleMessage } } } diff --git a/macapp/WorkroomApp/Views/NewWorkroomDialog.swift b/macapp/WorkroomApp/Views/NewWorkroomDialog.swift index 1baf59a83..2e942c192 100644 --- a/macapp/WorkroomApp/Views/NewWorkroomDialog.swift +++ b/macapp/WorkroomApp/Views/NewWorkroomDialog.swift @@ -70,10 +70,10 @@ struct NewWorkroomDialog: View { onClose() let anchor = placeAnchor ?? (split ? store.selectedTargetID : nil) Task { - switch place { - case .thisMac: await store.createWorkroom(in: project, splitAnchor: anchor) - case .container(let runtime): - await store.createRemoteWorkroom(in: project, runtime: runtime, splitAnchor: anchor) + if let remote = place.remote { + await store.createRemoteWorkroom(in: project, place: remote, splitAnchor: anchor) + } else { + await store.createWorkroom(in: project, splitAnchor: anchor) } } } @@ -83,8 +83,8 @@ struct NewWorkroomDialog: View { private func isUsable(_ place: WorkroomPlace, in project: Project) -> Bool { switch place { case .thisMac: !store.isBusyProject(project.path) - case .container(let runtime): - RemoteWorkrooms.unavailability(of: runtime) == nil + case .remote(let remote): + RemoteWorkrooms.unavailability(of: remote) == nil && store.canCreateRemoteWorkroom(in: project) } } @@ -121,12 +121,7 @@ struct NewWorkroomDialog: View { -> some View { let usable = isUsable(place, in: project) - let reason: String? = - if case .container(let runtime) = place { - RemoteWorkrooms.unavailability(of: runtime) - } else { - nil - } + let reason = place.remote.flatMap(RemoteWorkrooms.unavailability(of:)) return Button { pick(place, in: project, split: PickerSplitIntent.requestedFromCurrentModifiers()) } label: { @@ -312,40 +307,45 @@ struct NewWorkroomPresenter: ViewModifier { } } -/// Where a new workroom can go (#309): this Mac, or a local container on one of the runtimes. +/// Where a new workroom can go (#309, #356): this Mac, a local container on one of the runtimes, +/// or a boxd machine. enum WorkroomPlace: Hashable { case thisMac - case container(RemoteWorkrooms.Runtime) + case remote(RemoteWorkrooms.Place) static var all: [WorkroomPlace] { - [.thisMac] + RemoteWorkrooms.Runtime.allCases.map { .container($0) } + [.thisMac] + RemoteWorkrooms.Runtime.allCases.map { .remote(.container($0)) } + [.remote(.boxd)] } - var name: String { - switch self { - case .thisMac: "This Mac" - case .container(let runtime): runtime.displayName - } + /// The remote place this is, or nil for this Mac. + var remote: RemoteWorkrooms.Place? { + if case .remote(let place) = self { return place } + return nil } + var name: String { remote?.displayName ?? "This Mac" } + var id: String { switch self { case .thisMac: "thisMac" - case .container(let runtime): runtime.rawValue + case .remote(.container(let runtime)): runtime.rawValue + case .remote(.boxd): RemoteWorkrooms.boxdDriver } } var icon: String { switch self { case .thisMac: "laptopcomputer" - case .container: "network" + case .remote(.container): "network" + case .remote(.boxd): "cloud" } } var detail: String { switch self { case .thisMac: "A workroom on this Mac" - case .container(let runtime): "A workroom in \(runtime.containerPhrase) on this Mac" + case .remote(.container(let runtime)): "A workroom in \(runtime.containerPhrase) on this Mac" + case .remote(.boxd): "A workroom on a boxd machine" } } } diff --git a/macapp/WorkroomApp/Views/ProjectSidebar.swift b/macapp/WorkroomApp/Views/ProjectSidebar.swift index 1678dae52..3e271d14c 100644 --- a/macapp/WorkroomApp/Views/ProjectSidebar.swift +++ b/macapp/WorkroomApp/Views/ProjectSidebar.swift @@ -200,9 +200,12 @@ struct ProjectSidebar: View { newLocalWorkroomButton(Text("This Mac"), in: project) Section("Local containers") { ForEach(RemoteWorkrooms.Runtime.allCases, id: \.self) { runtime in - newContainerWorkroomButton(on: runtime, in: project) + newRemoteWorkroomButton(at: .container(runtime), in: project) } } + Section("Remote") { + newRemoteWorkroomButton(at: .boxd, in: project) + } } /// New Workroom on this Mac. Disabled while a create is in flight, matching the row's own "+" @@ -218,22 +221,22 @@ struct ProjectSidebar: View { .disabled(store.isBusyProject(project.path)) } - /// New Workroom in a local container on `runtime` (#309), cloned from the project's GitHub - /// origin. One that can't be used now says why in its title. - private func newContainerWorkroomButton( - on runtime: RemoteWorkrooms.Runtime, in project: Project + /// New Workroom at `place` (#309, #356): a local container, or a boxd machine, cloned from the + /// project's GitHub origin. One that can't be used now says why in its title. + private func newRemoteWorkroomButton( + at place: RemoteWorkrooms.Place, in project: Project ) -> some View { - // The runtime's own reason only: one that holds the whole project (a create in flight) is said + // The place's own reason only: one that holds the whole project (a create in flight) is said // once, above the entries. - let reason = RemoteWorkrooms.unavailability(of: runtime) + let reason = RemoteWorkrooms.unavailability(of: place) return Button { - Task { await store.createRemoteWorkroom(in: project, runtime: runtime) } + Task { await store.createRemoteWorkroom(in: project, place: place) } } label: { - Text(reason.map { "\(runtime.displayName) — \($0)" } ?? runtime.displayName) + Text(reason.map { "\(place.displayName) — \($0)" } ?? place.displayName) } .help( - reason.map { "\(runtime.displayName) can't be used: \($0)" } - ?? "A workroom of \(project.displayName) in \(runtime.containerPhrase) on this Mac" + reason.map { "\(place.displayName) can't be used: \($0)" } + ?? "A workroom of \(project.displayName) in \(place.hostPhrase)" ) .disabled(reason != nil || !store.canCreateRemoteWorkroom(in: project)) } @@ -308,6 +311,13 @@ struct ProjectSidebar: View { .progressViewStyle(.circular) .controlSize(.small) .help("Downloading the workroom image: \(Int(pulled * 100))%") + } else if let step = store.createSteps[project.path] { + // A boxd create's steps take a while each (#356): which one it is on. + ProgressView(value: step.fraction) + .progressViewStyle(.circular) + .controlSize(.small) + .help(step.label) + .accessibilityLabel(step.label) } else { ProgressView().controlSize(.small) } diff --git a/macapp/WorkroomApp/Views/ToastStack.swift b/macapp/WorkroomApp/Views/ToastStack.swift index 4254db0ed..89073b842 100644 --- a/macapp/WorkroomApp/Views/ToastStack.swift +++ b/macapp/WorkroomApp/Views/ToastStack.swift @@ -38,6 +38,32 @@ struct ToastStack: View { ?? "a remote machine" } + /// How many hosts' prompt cards show before they scroll, and the height they then take. + static let visibleHostPrompts = 2 + static let hostPromptsHeight: CGFloat = 320 + + /// The hosts whose card is up, soonest deadline first; one with none known goes last, in a stable + /// order. + static func ordered(_ ids: Set, expiries: [UUID: Date]) -> [UUID] { + ids.sorted { a, b in + switch (expiries[a], expiries[b]) { + case (let x?, let y?) where x != y: return x < y + case (.some, nil): return true + case (nil, .some): return false + default: return a.uuidString < b.uuidString + } + } + } + + @ViewBuilder + private func hostPrompts(_ ids: [UUID]) -> some View { + ForEach(ids, id: \.self) { id in + if let model = hosts.models[id] { + HostCeilingPrompt(model: model, machine: Self.machineName(forHost: id, in: store.projects)) + } + } + } + var body: some View { // ONE container (issue #67): live run toasts ride above the transient notification toasts, so the // two never overlap in the bottom-right corner (they'd collide as separate overlays). @@ -48,12 +74,21 @@ struct ToastStack: View { AwakeCeilingToastView(model: wakefulness) .transition(.move(edge: .trailing).combined(with: .opacity)) } - // Each remote host's: one whose box sleeps is handed this Mac's ask setting (#257). - ForEach(hosts.showing.sorted { $0.uuidString < $1.uuidString }, id: \.self) { id in - if let model = hosts.models[id] { - HostCeilingPrompt( - model: model, machine: Self.machineName(forHost: id, in: store.projects)) + // Each remote host's: one whose box sleeps is handed this Mac's ask setting (#257). Soonest + // deadline first, and past a few they scroll (#356): stacked unbounded, a card could be pushed + // off the window while its countdown runs, and nobody answering lets that box sleep. + let asking = Self.ordered( + hosts.showing, expiries: hosts.models.compactMapValues(\.prompt.expiresAt)) + if asking.count > Self.visibleHostPrompts { + ScrollView { + VStack(alignment: .trailing, spacing: 8) { hostPrompts(asking) } } + .scrollIndicators(.visible) + .frame(maxHeight: Self.hostPromptsHeight) + .fixedSize(horizontal: true, vertical: false) + .accessibilityIdentifier("toasts.hostPrompts") + } else { + hostPrompts(asking) } // Tool-version warnings ride at the top: unlike the two below, these describe a STANDING // condition (an old `git` doesn't fix itself), so they never auto-dismiss and stay until the diff --git a/macapp/WorkroomAppTests/AgentWakefulnessTests.swift b/macapp/WorkroomAppTests/AgentWakefulnessTests.swift index d81a88531..3240fb271 100644 --- a/macapp/WorkroomAppTests/AgentWakefulnessTests.swift +++ b/macapp/WorkroomAppTests/AgentWakefulnessTests.swift @@ -191,6 +191,52 @@ final class AgentWakefulnessTests: XCTestCase { XCTAssertLessThan(WakefulnessBadge.help(for: flood, settings: nil).count, 600) } + /// A provider that sleeps a box after less idle time than the once-a-minute heartbeat can beat + /// sleeps it under a running job however healthy the heartbeat is (#356): the badge says so, and + /// says how to fix it. A long enough window, an unknown one, an idle box and a host that never + /// sleeps show nothing new. + func testAnIdleWindowTheHeartbeatCannotBeatLeavesBusyWorkUnprotected() throws { + let healthy = try status( + Self.notPending + [(#""awake_ceiling_exceeded":true"#, #""awake_ceiling_exceeded":false"#)]) + XCTAssertEqual(healthy.display(hostSleeps: true, idleWindow: nil), .busy) + XCTAssertEqual(healthy.display(hostSleeps: true, idleWindow: 120), .busy) + XCTAssertEqual(healthy.display(hostSleeps: true, idleWindow: 60), .busyUnprotected) + // The boundary: a window of exactly the minimum is long enough. + let minimum = AgentWakefulness.minimumIdleWindow + XCTAssertEqual(healthy.display(hostSleeps: true, idleWindow: minimum - 1), .busyUnprotected) + XCTAssertEqual(healthy.display(hostSleeps: true, idleWindow: minimum), .busy) + XCTAssertEqual(healthy.display(hostSleeps: false, idleWindow: 60), .busy) + let idle = try status( + Self.notPending + [ + (#""awake_ceiling_exceeded":true"#, #""awake_ceiling_exceeded":false"#), + (#""busy":true"#, #""busy":false"#), + ]) + XCTAssertEqual(idle.display(hostSleeps: true, idleWindow: 60), .idle) + + let help = WakefulnessBadge.help(for: healthy, settings: nil, idleWindow: 60) + XCTAssertTrue(help.contains("after 60 s idle"), help) + XCTAssertTrue(help.contains("auto-suspend"), help) + XCTAssertFalse(help.contains("heartbeat is failing"), help) + // An idle box is warned too, before a job is started on it. + XCTAssertTrue( + WakefulnessBadge.help(for: idle, settings: nil, idleWindow: 60).contains("auto-suspend")) + XCTAssertFalse( + WakefulnessBadge.help(for: healthy, settings: nil, idleWindow: 120).contains("auto-suspend")) + // Beside a failing heartbeat both are said, the window once; a host that never sleeps hears + // nothing about a window. + let failing = try status( + Self.notPending + [ + (#""awake_ceiling_exceeded":true"#, #""awake_ceiling_exceeded":false"#), + (#""error":null"#, #""error":"no IPv4 default route""#), + ]) + let both = WakefulnessBadge.help(for: failing, settings: nil, idleWindow: 60) + XCTAssertTrue(both.contains("heartbeat is failing"), both) + XCTAssertEqual(both.components(separatedBy: "auto-suspend").count - 1, 1, both) + XCTAssertFalse( + WakefulnessBadge.help(for: failing, settings: nil, hostSleeps: false, idleWindow: 60) + .contains("auto-suspend")) + } + /// "Keep awake" restarts the ceiling, so the badge offers it only past the ceiling, an unanswered /// prompt included, and never for a heartbeat it cannot fix. func testKeepAwakeIsOfferedOnlyWhereItHelps() throws { @@ -285,6 +331,349 @@ final class AgentWakefulnessTests: XCTestCase { WakefulnessBadge.help(for: status, settings: nil).contains("when it next starts")) } + // Value: protects=a boxd host whose agent reports IDLE is let go of, so the app stops holding it awake; + // fails_when=WakefulnessModel stops reporting its verdicts to RemoteHosts.observed, or reports busy as idle; + // why_new=RemoteHostsTests call observed() directly and the live test re-implements the poll; seam=none + /// The model is what feeds `RemoteHosts.observed` (#356): each applied reading of a remote host + /// reports whether it is busy, so an idle boxd box is let go of and a busy one is kept. + @MainActor + func testAnIdleReadingOfABoxdHostIsReportedSoItIsLetGo() async throws { + let (idle, busy) = (UUID(), UUID()) + RemoteHosts.shared.adopt( + [Project(path: "/proj", vcs: "git", workrooms: [boxdWorkroom(idle), boxdWorkroom(busy)])], + sweep: false) + let quiet = try status([(#""busy":true"#, #""busy":false"#)]) + let working = try status() + XCTAssertTrue(working.busy) + let idleScript = Script(quiet) + let busyScript = Script(working) + let idleModel = WakefulnessModel( + transport: .init(status: idleScript.status, keep: {}, prompts: { throw Unavailable() }), + host: idle) + let busyModel = WakefulnessModel( + transport: .init(status: busyScript.status, keep: {}, prompts: { throw Unavailable() }), + host: busy) + + await busyModel.refresh() + await idleModel.refresh() + + await eventually("an idle boxd host was never let go of") { + RemoteHosts.shared.isLetGo(.remote(idle)) + } + XCTAssertFalse(RemoteHosts.shared.isLetGo(.remote(busy)), "a busy box was let go of") + } + + /// A stalled service's IDLE may be stale while work runs, so it does not let a box go within its + /// ceiling (#356): the connection may be all that still holds the box awake. + @MainActor + func testAStalledIdleReadingDoesNotLetABoxGo() async throws { + let (id, control) = (UUID(), UUID()) + RemoteHosts.shared.adopt( + [Project(path: "/proj", vcs: "git", workrooms: [boxdWorkroom(id), boxdWorkroom(control)])], + sweep: false) + let idle = (#""busy":true"#, #""busy":false"#) + let stalledIdle = try status([idle, (#""stalled":false"#, #""stalled":true"#)]) + let model = WakefulnessModel( + transport: .init( + status: Script(stalledIdle).status, keep: {}, prompts: { throw Unavailable() }), + host: id) + // A plain IDLE on another host, reported after the stalled one, proves the reports landed. + let controlModel = WakefulnessModel( + transport: .init( + status: Script(try status([idle])).status, keep: {}, prompts: { throw Unavailable() }), + host: control) + await model.refresh() + await controlModel.refresh() + await eventually("the control host's IDLE was never reported") { + RemoteHosts.shared.isLetGo(.remote(control)) + } + try await Task.sleep(for: .milliseconds(100)) + XCTAssertFalse(RemoteHosts.shared.isLetGo(.remote(id)), "a stalled IDLE let the box go") + } + + // Value: protects=a box whose status service stopped while idle is let go of, and one that + // stopped under a job keeps its connection; fails_when=observe() reports only readings from a + // running service, or lets a stopped service's BUSY go; why_new=every other test reads a running + // service; seam=none + /// A service that is not running holds nothing awake, its heartbeat included (#356), so its box + /// is let go of, unless its last reading was BUSY: then the connection is all that keeps the + /// job's box awake. + @MainActor + func testAStoppedServiceLetsAnIdleBoxGoButKeepsABusyOne() async throws { + let (idleID, busyID) = (UUID(), UUID()) + RemoteHosts.shared.adopt( + [ + Project(path: "/proj", vcs: "git", workrooms: [boxdWorkroom(busyID), boxdWorkroom(idleID)]) + ], + sweep: false) + let stopped = (#""running":true"#, #""running":false"#) + let stoppedBusy = try status([stopped]) + XCTAssertTrue(stoppedBusy.busy, "the fixture's last verdict was BUSY") + XCTAssertEqual(stoppedBusy.ceilingSeconds, 14400) + let stoppedIdle = try status([stopped, (#""busy":true"#, #""busy":false"#)]) + let model = { (reply: AgentWakefulness, id: UUID) in + WakefulnessModel( + transport: .init(status: Script(reply).status, keep: {}, prompts: { throw Unavailable() }), + host: id) + } + let busyModel = model(stoppedBusy, busyID) + let idleModel = model(stoppedIdle, idleID) + await busyModel.refresh() + await idleModel.refresh() + await eventually("a box whose service stopped while idle was never let go of") { + RemoteHosts.shared.isLetGo(.remote(idleID)) + } + // The busy host reported first; a moment more lets a report that raced the control's land. + try await Task.sleep(for: .milliseconds(100)) + XCTAssertFalse( + RemoteHosts.shared.isLetGo(.remote(busyID)), "a box whose service died under a job was let go" + ) + + // Nothing will ever ask about that job, so its BUSY holds the box for one ceiling at most. + let expiredID = UUID() + RemoteHosts.shared.adopt( + [Project(path: "/proj", vcs: "git", workrooms: [boxdWorkroom(expiredID)])], sweep: false) + let pastCeiling = try status([ + stopped, (#""ceiling_seconds":14400.0"#, #""ceiling_seconds":0.0"#), + ]) + await model(pastCeiling, expiredID).refresh() + await eventually("a stopped service's BUSY held its box past its ceiling") { + RemoteHosts.shared.isLetGo(.remote(expiredID)) + } + } + + // Value: protects=a reading the app can't trust (failed, stalled, stopped under a job) holds its + // box across polls for one ceiling, then lets it go; fails_when=the hold restarts on every poll, + // is unbounded, or ends at once; why_new=the other hold tests read one refresh each; seam=none + /// A failed read, a stalled service and a stopped one under a job say nothing true about the job, + /// and nothing on the box will ask about it, so each holds its box for one ceiling, counted from + /// the first such reading, not from each poll (#356). + @MainActor + func testAnUntrustedReadingHoldsItsBoxForOneCeilingAcrossPolls() async throws { + let ceiling = (#""ceiling_seconds":14400.0"#, #""ceiling_seconds":1.0"#) + let stalled = (#""stalled":false"#, #""stalled":true"#) + let stopped = (#""running":true"#, #""running":false"#) + let idle = (#""busy":true"#, #""busy":false"#) + let (failingID, stalledID, stoppedID, control) = (UUID(), UUID(), UUID(), UUID()) + RemoteHosts.shared.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [failingID, stalledID, stoppedID, control].map(boxdWorkroom)) + ], sweep: false) + let failing = Script(try status([ceiling])) + let scripts = [ + (failingID, failing), (stalledID, Script(try status([ceiling, stalled, idle]))), + (stoppedID, Script(try status([ceiling, stopped]))), + ] + let models = scripts.map { id, script in + WakefulnessModel( + transport: .init(status: script.status, keep: {}, prompts: { throw Unavailable() }), + host: id) + } + // The failing host's first read is a BUSY with its ceiling; every read after that fails. + await models[0].refresh() + failing.reply = .failure(Unavailable()) + for model in models { await model.refresh() } + await WakefulnessModel( + transport: .init( + status: Script(try status([idle])).status, keep: {}, prompts: { throw Unavailable() }), + host: control + ).refresh() + await eventually("the control host's IDLE was never reported") { + RemoteHosts.shared.isLetGo(.remote(control)) + } + try await Task.sleep(for: .milliseconds(100)) + for (id, _) in scripts { + XCTAssertFalse(RemoteHosts.shared.isLetGo(.remote(id)), "\(id) was let go within its ceiling") + } + + // Polled again within the ceiling, then past it: the hold counts from the first reading. + try await Task.sleep(for: .milliseconds(600)) + for model in models { await model.refresh() } + try await Task.sleep(for: .milliseconds(600)) + for model in models { await model.refresh() } + for (id, _) in scripts { + await eventually("an untrusted reading held \(id) past its ceiling") { + RemoteHosts.shared.isLetGo(.remote(id)) + } + } + } + + // Value: protects=the untrusted hold's clock starts at a reading from the agent and restarts after + // a trusted one or a connect, so a stalled box that comes back after a let-go, a recovery or a + // reconnect is still held, while a read that reached the agent and failed starts it; + // fails_when=a refused read (any of the six) starts the clock, one that reached the agent does + // not, or a trusted reading or a connect leaves it running; why_new=the across-polls test never + // interleaves those; seam=none + /// A read refused before it left (the host let go of, the request refused or its connection + /// replaced) says nothing about the box, and a trusted reading or a connect ends an untrusted + /// spell, so none of them leaves an old clock to cut a later hold short. A read that reached the + /// agent and failed does start it. + @MainActor + func testAnUntrustedHoldCountsOnlyReadingsFromTheAgent() async throws { + let ceiling = (#""ceiling_seconds":14400.0"#, #""ceiling_seconds":1.0"#) + let stalled = (#""stalled":false"#, #""stalled":true"#) + let idle = (#""busy":true"#, #""busy":false"#) + let (refusedID, recoveredID, reconnectedID, control) = (UUID(), UUID(), UUID(), UUID()) + let reachedIDs = [UUID(), UUID(), UUID()] + let resumedID = UUID() + RemoteHosts.shared.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: ([refusedID, recoveredID, reconnectedID, control, resumedID] + reachedIDs) + .map(boxdWorkroom)) + ], sweep: false) + let trusted = try status([ceiling]) + let (refused, recovered, reconnected) = (Script(trusted), Script(trusted), Script(trusted)) + let model = { (script: Script, id: UUID) in + WakefulnessModel( + transport: .init(status: script.status, keep: {}, prompts: { throw Unavailable() }), + host: id) + } + let (refusedModel, recoveredModel, reconnectedModel) = ( + model(refused, refusedID), model(recovered, recoveredID), model(reconnected, reconnectedID) + ) + defer { reconnectedModel.stopPollingWhileConnected() } + for model in [refusedModel, recoveredModel, reconnectedModel] { await model.refresh() } + // One host's reads are refused before they leave, every way they can be; another's reach an + // agent that fails, then answers again; the third's fail, and then it connects afresh; and + // three more reach the agent and fail, which starts their clocks. + let host = HostID.remote(refusedID) + let refusals: [Error] = [ + RepositoryRoutingError.idle(host), RepositoryRoutingError.asleep(host), + RepositoryRoutingError.unavailable(host), HostConnectionError.notDispatched, + HostConnectionError.staleGeneration, HostConnectionError.mismatchedContext, + ] + for refusal in refusals { + refused.reply = .failure(refusal) + await refusedModel.refresh() + } + recovered.reply = .failure(Unavailable()) + reconnected.reply = .failure(Unavailable()) + await recoveredModel.refresh() + await reconnectedModel.refresh() + recovered.reply = .success(trusted) + await recoveredModel.refresh() + // Each of these reached the agent, so it starts the clock, and the stalled reading after the + // ceiling holds nothing. + let reached: [Error] = [ + HostConnectionError.connectionLost, HostConnectionError.requestTimedOut, + HostConnectionError.serviceUnavailable("wakefulness"), + ] + // A poll resumed after a let-go was taken back is no connect: its clock keeps running. + let resumed = Script(trusted) + let resumedModel = model(resumed, resumedID) + defer { resumedModel.stopPollingWhileConnected() } + await resumedModel.refresh() + resumed.reply = .failure(Unavailable()) + await resumedModel.refresh() + var reachedModels: [(WakefulnessModel, Script)] = [] + for (id, failure) in zip(reachedIDs, reached) { + let script = Script(trusted) + let reachedModel = model(script, id) + await reachedModel.refresh() + script.reply = .failure(failure) + await reachedModel.refresh() + reachedModels.append((reachedModel, script)) + } + // Past the ceiling those reads would have started, all come back stalled: a full hold for the + // first three, whose clocks never started or restarted, and none for the reached ones. + try await Task.sleep(for: .milliseconds(1100)) + let stalledReply = try status([ceiling, stalled, idle]) + for script in [refused, recovered, reconnected] + reachedModels.map(\.1) { + script.reply = .success(stalledReply) + } + for (reachedModel, _) in reachedModels { await reachedModel.refresh() } + resumed.reply = .success(stalledReply) + resumedModel.pollWhileConnected(connected: false) + await resumedModel.refresh() + await refusedModel.refresh() + await recoveredModel.refresh() + reconnectedModel.pollWhileConnected() + await reconnectedModel.refresh() + await model(Script(try status([idle])), control).refresh() + await eventually("the control host's IDLE was never reported") { + RemoteHosts.shared.isLetGo(.remote(control)) + } + try await Task.sleep(for: .milliseconds(100)) + XCTAssertFalse( + RemoteHosts.shared.isLetGo(.remote(refusedID)), "a refused read started the hold") + XCTAssertFalse( + RemoteHosts.shared.isLetGo(.remote(recoveredID)), "a trusted reading left the hold's clock") + XCTAssertFalse( + RemoteHosts.shared.isLetGo(.remote(reconnectedID)), "a connect left the hold's old clock") + for id in reachedIDs + [resumedID] { + await eventually("a failed read that reached the agent did not start the hold's clock") { + RemoteHosts.shared.isLetGo(.remote(id)) + } + } + } + + // Value: protects=an absurd ceiling on the wire (1e300 decodes as a JSON number) is clamped, not + // handed to Duration.seconds, which traps; fails_when=the hold uses the raw wire value; + // why_new=every other hold test sends a sane ceiling; seam=none + /// A stalled reading naming a ceiling past `Duration`'s range holds the box rather than crash the + /// app: wire durations are sanitised here as everywhere else (#356). + @MainActor + func testAnAbsurdCeilingOnTheWireHoldsRatherThanTraps() async throws { + let (huge, negative, control) = (UUID(), UUID(), UUID()) + RemoteHosts.shared.adopt( + [Project(path: "/proj", vcs: "git", workrooms: [huge, negative, control].map(boxdWorkroom))], + sweep: false) + let stalled = (#""stalled":false"#, #""stalled":true"#) + // A huge ceiling is capped, not trapped on; a negative one is dropped for the setting's, so + // neither ends the hold at once. + for (id, ceiling) in [(huge, "1e300"), (negative, "-5.0")] { + let reply = try status([ + (#""ceiling_seconds":14400.0"#, #""ceiling_seconds":\#(ceiling)"#), stalled, + ]) + await WakefulnessModel( + transport: .init(status: Script(reply).status, keep: {}, prompts: { throw Unavailable() }), + host: id + ).refresh() + } + // A plain IDLE on another host, reported after them, proves the reports landed. + await WakefulnessModel( + transport: .init( + status: Script(try status([(#""busy":true"#, #""busy":false"#)])).status, keep: {}, + prompts: { throw Unavailable() }), + host: control + ).refresh() + await eventually("the control host's IDLE was never reported") { + RemoteHosts.shared.isLetGo(.remote(control)) + } + try await Task.sleep(for: .milliseconds(100)) + XCTAssertFalse(RemoteHosts.shared.isLetGo(.remote(huge)), "a huge ceiling held nothing") + XCTAssertFalse(RemoteHosts.shared.isLetGo(.remote(negative)), "a negative ceiling held nothing") + } + + /// A connected boxd host is polled with nothing on screen (#356), so an IDLE reading lets it go + /// when its row is scrolled away; stopping ends the polls. + @MainActor + func testAConnectedHostIsPolledWithNothingOnScreen() async throws { + let script = Script(try status()) + let model = WakefulnessModel( + transport: .init(status: script.status, keep: {}, prompts: { throw Unavailable() }), + host: UUID()) + model.pollWhileConnected() + model.pollWhileConnected() + await eventually("a connected host was never polled") { script.calls == 1 } + XCTAssertTrue(model.isPollingWhileConnected) + model.stopPollingWhileConnected() + XCTAssertFalse(model.isPollingWhileConnected) + try await Task.sleep(for: .milliseconds(100)) + XCTAssertEqual(script.calls, 1, "a second poll loop was started") + // Value: protects=stopping the connection poll ends its loop, not just the flag, so a let-go box gets no more polls; + // fails_when=stopPollingWhileConnected drops the task without cancelling it, leaving poll() running; + // why_new=the 10s poll interval makes the call count above unchanged whether or not the loop ended; seam=none + // A loop still running would answer a new poll on its 10 s timer; an ended one reads at once. + model.pollWhileConnected() + await eventually("the stopped poll's loop was still running") { script.calls == 2 } + model.stopPollingWhileConnected() + } + /// One model per remote host, kept for the launch, and never this Mac's (#254). @MainActor func testEachRemoteHostHasItsOwnWakefulnessModel() { @@ -324,6 +713,43 @@ final class AgentWakefulnessTests: XCTestCase { XCTAssertEqual(ToastStack.machineName(forHost: UUID(), in: projects), "a remote machine") } + /// Several hosts asking at once (#356): the most urgent card comes first, so the one about to + /// let its box sleep is never the one scrolled away; a card with no known deadline goes last. + func testHostPromptsAreOrderedBySoonestDeadline() { + let (soon, later, unknown) = (UUID(), UUID(), UUID()) + let now = Date() + let order = ToastStack.ordered( + [unknown, later, soon], + expiries: [soon: now.addingTimeInterval(30), later: now.addingTimeInterval(500)]) + XCTAssertEqual(order, [soon, later, unknown]) + // Hosts with no deadline, or the same one, keep one order however the set is built, so the + // cards don't reshuffle between renders. + let ids = (0..<4).map { _ in UUID() } + let stable = ids.sorted { $0.uuidString < $1.uuidString } + let tie = now.addingTimeInterval(60) + for expiries in [[:], Dictionary(uniqueKeysWithValues: ids.map { ($0, tie) })] { + XCTAssertEqual(ToastStack.ordered(Set(ids), expiries: expiries), stable) + XCTAssertEqual(ToastStack.ordered(Set(ids.reversed()), expiries: expiries), stable) + } + } + + /// A stalled service on a host that sleeps: its last reading said IDLE, which may be stale while + /// work runs, and nothing is keeping the box awake either way, so it reads unknown, not idle + /// (#356). A host that never sleeps keeps reading idle: there is nothing to keep it awake from. + func testAStalledIdleReadingIsUnknownWhereTheBoxSleeps() throws { + let stalledIdle = try status( + Self.notPending + [ + (#""stalled":false"#, #""stalled":true"#), (#""busy":true"#, #""busy":false"#), + (#""awake_ceiling_exceeded":true"#, #""awake_ceiling_exceeded":false"#), + ]) + XCTAssertFalse(stalledIdle.unprotected) + XCTAssertEqual(stalledIdle.display(hostSleeps: true), .unknown) + XCTAssertEqual(stalledIdle.display(hostSleeps: false), .idle) + XCTAssertTrue( + WakefulnessBadge.help(for: stalledIdle, settings: nil).contains("unknown"), + WakefulnessBadge.help(for: stalledIdle, settings: nil)) + } + /// A remote host asks too, since the app hands its agent this Mac's ask-at-ceiling setting /// (#257): its model is published for the toast stack and watching for prompts from the moment /// it is made, and a deleted host's goes, watch and all. @@ -1009,6 +1435,14 @@ final class AgentWakefulnessTests: XCTestCase { private struct Unavailable: Error {} + /// A remote workroom on a boxd host `id`, for `RemoteHosts.shared.adopt`. + private func boxdWorkroom(_ id: UUID) -> Workroom { + Workroom( + name: id.uuidString, path: "/home/boxd/r", vcsName: "workroom/\(id)", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: id)) + } + /// Waits for `condition` on the main actor, failing the test rather than hanging it. @MainActor private func eventually( diff --git a/macapp/WorkroomAppTests/BoxdHostDriverTests.swift b/macapp/WorkroomAppTests/BoxdHostDriverTests.swift index 9c2454fad..bd55c10a0 100644 --- a/macapp/WorkroomAppTests/BoxdHostDriverTests.swift +++ b/macapp/WorkroomAppTests/BoxdHostDriverTests.swift @@ -103,6 +103,9 @@ final class BoxdHostDriverTests: XCTestCase { self.orgs = orgs } + /// Every call's full arguments, org checks included. + var arguments: [[String]] { lock.withLock { calls } } + /// Every command but the org checks. var commands: [String] { lock.withLock { calls.map { $0.prefix(2).joined(separator: " ") } } @@ -115,6 +118,10 @@ final class BoxdHostDriverTests: XCTestCase { lock.withLock { calls.append(args) } let command = args.prefix(2).joined(separator: " ") if command == cancelling { withUnsafeCurrentTask { $0?.cancel() } } + // A delete also removes the derive snapshot named after the machine, usually absent. + if command == "snapshots remove", answers[command] == nil { + return BoxdHostDriverTests.notFound("error: snapshot not found") + } // The account's own org is active unless a test says otherwise. if command == "auth --json", answers[command] == nil { let org = lock.withLock { orgs.isEmpty ? nil : orgs.removeFirst() } @@ -129,6 +136,7 @@ final class BoxdHostDriverTests: XCTestCase { fileprivate static func ok(_ json: String) -> CommandResult { CommandResult(stdout: json, stderr: "", exitCode: 0, timedOut: false) } + fileprivate static func notFound(_ said: String) -> CommandResult { failed(said) } private static func failed(_ said: String) -> CommandResult { CommandResult(stdout: "", stderr: "\n A new version…\n\(said)\n", exitCode: 1, timedOut: false) } @@ -157,6 +165,28 @@ final class BoxdHostDriverTests: XCTestCase { XCTAssertEqual(cli.commands, ["machine new", "machine remove"], "the machine was kept") } + /// A create says which step it is on, for its project row (#356): the machine first, then its + /// setup, which this one never reaches (the CLI wrote no ssh details). + func testACreateReportsItsSteps() async throws { + let cli = StubCLI([ + "machine new": Self.ok(#"{"name":"x"}"#), + "machine remove": Self.ok(#"{"status":"destroyed"}"#), + ]) + let steps = Steps() + let report: @Sendable (RemoteProvisioning.Step) -> Void = { steps.add($0) } + await RemoteProvisioning.$reportStep.withValue(report) { + _ = try? await driver(cli).create() + } + XCTAssertEqual(steps.all, [.machine, .setup]) + } + + private final class Steps: @unchecked Sendable { + private let lock = NSLock() + private var seen: [RemoteProvisioning.Step] = [] + var all: [RemoteProvisioning.Step] { lock.withLock { seen } } + func add(_ step: RemoteProvisioning.Step) { lock.withLock { seen.append(step) } } + } + func testACreateWhoseMachineIsRefusedSaysTheCLIsReasonAndHasNothingToRemove() async throws { let cli = StubCLI([ "machine new": Self.failed("error: quota exceeded"), @@ -178,9 +208,15 @@ final class BoxdHostDriverTests: XCTestCase { do { _ = try await driver(cli).create() XCTFail("a refused machine became a base") - } catch HostDriverError.leftBehind(let cause, let leftover) { + } catch HostDriverError.leftBehind(let cause, let leftover, let host) { XCTAssertEqual(cause, "Couldn't provision the host: boxd machine new: error: quota exceeded") + // Named, so the caller can record the machine for a delete to take down. XCTAssertEqual(leftover.count, 1) + // Named exactly, so a later delete reaches the machine the leftover names. + guard case .remote(let id) = host else { + return XCTFail("a machine left behind was not named") + } + XCTAssertTrue(leftover[0].contains(id.uuidString.lowercased()), leftover[0]) XCTAssertTrue(leftover[0].hasPrefix("machine workroom-"), leftover[0]) } } @@ -229,7 +265,126 @@ final class BoxdHostDriverTests: XCTestCase { configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd"), org: "acme"), directory: FileManager.default.temporaryDirectory, runner: cli) try await named.destroy(.remote(UUID())) - XCTAssertEqual(cli.commands, ["machine remove"]) + XCTAssertEqual(cli.commands, ["machine remove", "snapshots remove"]) + } + + // Value: protects=every new boxd host records the account it was made in, so a delete checks it; + // fails_when=a boxd auth reply with no user_id is accepted and the host records no account; + // why_new=the account check is tested only with an account recorded; seam=none + /// A new host records its account (#356): an auth reply that names none is refused, so a later + /// delete never skips the check that the same account is signed in. + func testANewHostNeedsTheSignedInAccountsID() throws { + let decode = { (json: String) in + try JSONDecoder().decode(BoxdHostDriver.Account.self, from: Data(json.utf8)) + } + XCTAssertEqual(try decode(#"{"active_org":null,"user_id":"usr_1"}"#).requiredUserID(), "usr_1") + XCTAssertThrowsError(try decode(#"{"active_org":"acme"}"#).requiredUserID()) { error in + XCTAssertTrue(error.localizedDescription.contains("boxd auth login"), "\(error)") + } + } + + /// Every personal account's org is nil, so after a switch to another account the org check alone + /// passes and the first account's machines read as not found, which `destroy` would take for gone + /// (#356). The account is checked too, and only the one that made the machines goes ahead. + func testNothingIsDoneWhileAnotherAccountIsSignedIn() async throws { + let cli = StubCLI([ + "auth --json": Self.ok(#"{"active_org":null,"user_id":"usr_other"}"#), + "machine remove": Self.failed("error: VM 'x' not found"), + ]) + let mine = BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd"), account: "usr_mine"), + directory: FileManager.default.temporaryDirectory, runner: cli) + do { + try await mine.destroy(.remote(UUID())) + XCTFail("a destroy went ahead on another account") + } catch HostDriverError.invalidConfiguration(let detail) { + XCTAssertTrue(detail.contains("boxd auth login"), detail) + } + XCTAssertEqual(cli.commands, [], "a command ran on another account") + + let other = BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd"), account: "usr_other"), + directory: FileManager.default.temporaryDirectory, runner: cli) + try await other.destroy(.remote(UUID())) + XCTAssertEqual(cli.commands, ["machine remove", "snapshots remove"]) + } + + /// boxd's idle timers, whose encoding isn't documented: the shorter of the two that are set, in + /// seconds, from a number or a string with a unit; a timer that is off, absent or unreadable + /// never counts, so it never warns (#356). + func testTheIdleWindowIsTheShorterTimerThatIsSet() async throws { + let cases: [(String, TimeInterval?)] = [ + (#"{"source":"standalone","auto_suspend":120,"auto_hibernate":600}"#, 120), + (#"{"source":"standalone","auto_suspend":"2m","auto_hibernate":"45s"}"#, 45), + (#"{"source":"standalone","auto_suspend":0,"auto_hibernate":"1h"}"#, 3600), + (#"{"source":"standalone","auto_suspend":null}"#, nil), + (#"{"source":"standalone","auto_suspend":"never","auto_hibernate":false}"#, nil), + (#"{"source":"standalone"}"#, nil), + (#"{"source":"standalone","auto_suspend":"120"}"#, 120), + // A timer no Duration can hold would trap the revisit arithmetic: it reads as unknown. + (#"{"source":"standalone","auto_suspend":1e30,"auto_hibernate":"inf"}"#, nil), + // One unreadable timer leaves the other; a week is the most read. + (#"{"source":"standalone","auto_suspend":1e30,"auto_hibernate":60}"#, 60), + (#"{"source":"standalone","auto_suspend":604800}"#, 604800), + (#"{"source":"standalone","auto_suspend":604801}"#, nil), + ] + for (json, expected) in cases { + let window = try await driver(StubCLI(["machine get": Self.ok(json)])).idleWindow( + .remote(UUID())) + XCTAssertEqual(window, expected, json) + } + // A failed call is not "no timer": it throws, so a caller keeps what it knew. + do { + _ = try await driver(StubCLI([:])).idleWindow(.remote(UUID())) + XCTFail("a failed call read as no timer") + } catch {} + } + + /// Asleep is boxd's own status: suspended (`standby`) or hibernated, never a guess. + func testAsleepIsBoxdsStandbyOrHibernated() async throws { + for (status, presence) in [ + ("standby", BoxdHostDriver.Presence.asleep), ("hibernated", .asleep), ("running", .awake), + ("stopped", .awake), + ] { + let cli = StubCLI(["machine get": Self.ok(#"{"source":"standalone","status":"\#(status)"}"#)]) + let answer = await driver(cli).presence(.remote(UUID())) + XCTAssertEqual(answer, presence, status) + } + // A reply without `source`, which only a derive reads, still says whether the box is asleep. + let sourceless = await driver(StubCLI(["machine get": Self.ok(#"{"status":"standby"}"#)])) + .presence(.remote(UUID())) + XCTAssertEqual(sourceless, .asleep, "a reply without source failed open") + // A reply without a status can't say the box is awake, so it doesn't. + let statusless = await driver(StubCLI(["machine get": Self.ok(#"{"source":"standalone"}"#)])) + .presence(.remote(UUID())) + XCTAssertNil(statusless, "a reply without status failed open") + // boxd answered with an error (signed out, another account): leave it alone. + let refused = await driver(StubCLI(["machine get": Self.failed("error: not logged in")])) + .presence(.remote(UUID())) + XCTAssertEqual(refused, .asleep, "a CLI error woke the box") + // boxd says the machine is not found: gone, which a read reports rather than "asleep". + let gone = await driver(StubCLI(["machine get": Self.failed("error: VM 'x' not found")])) + .presence(.remote(UUID())) + XCTAssertEqual(gone, .gone) + // boxd couldn't answer at all (timed out): nil, which `RemoteHosts` takes as asleep. + let timedOut = await driver( + StubCLI([ + "machine get": CommandResult(stdout: "", stderr: "", exitCode: 15, timedOut: true) + ]) + ).presence(.remote(UUID())) + XCTAssertNil(timedOut, "a CLI that never answered was taken for an answer") + } + + /// Signed out of boxd, a create stops at its first CLI call and tells the user how to sign in. + func testSignedOutSaysToSignInAndRunsNothingElse() async throws { + let cli = StubCLI(["auth --json": Self.failed("error: not logged in")]) + do { + _ = try await driver(cli).create() + XCTFail("a create went ahead signed out") + } catch HostDriverError.provisioning(let detail) { + XCTAssertTrue(detail.contains("boxd auth login"), detail) + } + XCTAssertEqual(cli.commands, [], "a command ran signed out") } /// The org can change between the check and the removal: "not found" then means nothing. @@ -244,14 +399,21 @@ final class BoxdHostDriverTests: XCTestCase { } } - /// A cancelled call is a cancellation, not a provisioning failure. + /// A cancelled call is a cancellation, not a provisioning failure: also while a delete removes + /// the derive snapshot, which the best-effort step must not swallow, or a cancelled delete would + /// read as done and its record go while the snapshot stays (#356). func testACancelledCLICallThrowsCancellation() async throws { - let cli = StubCLI(["machine remove": Self.ok("{}")]) - cli.cancelling = "machine remove" - do { - try await driver(cli).destroy(.remote(UUID())) - XCTFail("a cancelled destroy reported success") - } catch is CancellationError {} + for step in ["machine remove", "snapshots remove"] { + let cli = StubCLI([ + "machine remove": Self.ok("{}"), "snapshots remove": Self.failed("error: internal"), + ]) + cli.cancelling = step + let destroy = Task { [driver = driver(cli)] in try await driver.destroy(.remote(UUID())) } + do { + try await destroy.value + XCTFail("a destroy cancelled at \(step) reported success") + } catch is CancellationError {} + } } func testDestroyingAMachineAlreadyGoneSucceedsAndAFailedRemovalThrows() async throws { @@ -265,5 +427,27 @@ final class BoxdHostDriverTests: XCTestCase { } catch HostDriverError.provisioning(let detail) { XCTAssertTrue(detail.hasSuffix("boxd machine remove: error: internal"), detail) } + + // Value: protects=a delete also removes the derive snapshot a failed derive left, named as its + // machine is, even when the machine is already gone, and a snapshot that will not go never + // makes a removed machine read as running; fails_when=destroy removes only the machine, names + // another snapshot, or throws on the snapshot step; why_new=destroy tests never named a + // snapshot; seam=none + for machine in [Self.ok(#"{"status":"destroyed"}"#), Self.failed("error: VM 'x' not found")] { + let snapshot = StubCLI(["machine remove": machine, "snapshots remove": Self.ok("{}")]) + try await driver(snapshot).destroy(.remote(UUID())) + XCTAssertEqual(snapshot.commands, ["machine remove", "snapshots remove"]) + let removals = snapshot.arguments.filter { $0.count > 2 && $0[1] == "remove" } + XCTAssertEqual(removals.count, 2) + XCTAssertEqual( + removals.first?.dropFirst(2).first, removals.last?.dropFirst(2).first, + "the snapshot removed was not the one named after the machine") + } + let stuck = StubCLI([ + "machine remove": Self.ok(#"{"status":"destroyed"}"#), + "snapshots remove": Self.failed("error: internal"), + ]) + try await driver(stuck).destroy(.remote(UUID())) + XCTAssertEqual(stuck.commands, ["machine remove", "snapshots remove"]) } } diff --git a/macapp/WorkroomAppTests/BoxdIntegrationTests.swift b/macapp/WorkroomAppTests/BoxdIntegrationTests.swift index e455e5ecb..b0e2f5350 100644 --- a/macapp/WorkroomAppTests/BoxdIntegrationTests.swift +++ b/macapp/WorkroomAppTests/BoxdIntegrationTests.swift @@ -40,6 +40,14 @@ final class BoxdIntegrationTests: XCTestCase { override func tearDown() async throws { for connection in connections { await connection.close() } connections.removeAll() + for host in attached { + // `connect` starts the shared model's poll, which would otherwise outlive the test. + if case .remote(let id) = host { await MainActor.run { WakefulnessModel.forgetHost(id) } } + if let lease = await HostConnectionManager.shared.snapshot(for: host).lease { + await HostConnectionManager.shared.disconnect(lease) + } + } + attached.removeAll() if cli != nil, prefix != nil { for name in (try? leftovers("machine")) ?? [] { _ = try? boxd(["machine", "remove", name, "-y"]) @@ -532,4 +540,236 @@ final class BoxdIntegrationTests: XCTestCase { } try await RemoteProvisioning.destroyBase(base.host, in: environment(driver), forget: {}) } + + // MARK: In the app (#356) + + /// A remote workroom made and taken down the way the app does it (`RemoteWorkrooms.create` and + /// `delete` on a boxd driver key): the base is built in the boxd user's home, the workroom's + /// record names boxd with its org and account, and deleting the workroom, then the base, leaves + /// no machine and no live grant. + @MainActor + func testARemoteWorkroomIsMadeAndTakenDownOnBoxdAsTheAppDoesIt() async throws { + let driver = WithFakeGitHub(driver: driver()) + let environment = environment(driver) + let account = + try JSONSerialization.jsonObject(with: Data(try boxd(["auth"]).utf8)) + as? [String: Any] + let key = RemoteHosts.DriverKey.boxd( + org: account?["active_org"] as? String, account: account?["user_id"] as? String) + let records = Records() + let recorder = RemoteWorkrooms.Recorder( + reserve: { _, descriptor in + records.set(nil, descriptor, as: "w") + return "w" + }, record: { name, descriptor in records.set(name, descriptor) }, + forget: { name in records.forget(name) }) + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + + BrokerStub.reset([Self.cloneToken, Self.grant]) + let created = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/origin.git", base: nil, key: key, + driver: driver, environment: environment, recorder: recorder) + await created.instance.connection.close() + let base = try XCTUnwrap(records.project) + let workroom = try XCTUnwrap(records.workrooms["w"]) + XCTAssertEqual(base.path, "/home/boxd/r") + for descriptor in [base, workroom] { + XCTAssertEqual(descriptor.driver, RemoteWorkrooms.boxdDriver) + XCTAssertEqual(descriptor.org, key.org) + XCTAssertEqual(descriptor.account, key.account) + XCTAssertNil(descriptor.container) + } + XCTAssertEqual(RemoteHosts.DriverKey(workroom), key) + + BrokerStub.reset([Self.cancelled]) + try await RemoteWorkrooms.delete( + "w", host: workroom, environment: environment, recorder: recorder) + XCTAssertNil(records.workrooms["w"], "the workroom's entry was kept") + XCTAssertEqual(grantsCancelled, 1, "deleting the workroom left its grant live") + try await RemoteWorkrooms.deleteBase(base, environment: environment, clear: {}) + XCTAssertEqual(try leftovers("machine"), []) + } + + private final class Records: @unchecked Sendable { + private let lock = NSLock() + private var base: HostDescriptor? + private var named: [String: HostDescriptor] = [:] + var project: HostDescriptor? { lock.withLock { base } } + var workrooms: [String: HostDescriptor] { lock.withLock { named } } + func set(_ name: String?, _ descriptor: HostDescriptor, as reserved: String? = nil) { + lock.withLock { + if let name = name ?? reserved { named[name] = descriptor } else { base = descriptor } + } + } + func forget(_ name: String) { _ = lock.withLock { named.removeValue(forKey: name) } } + } + + // MARK: Sleep (#257, #356) + + /// boxd's own record of whether a machine is asleep, from its control plane, which never + /// reaches the machine and so never wakes it. + private func status(_ host: HostID) throws -> String { + let got = try JSONSerialization.jsonObject( + with: Data(try boxd(["machine", "get", name(host)]).utf8)) + return (got as? [String: Any])?["status"] as? String ?? "?" + } + + private func asleep(_ host: HostID) -> Bool { + BoxdHostDriver.asleepStatuses.contains((try? status(host)) ?? "") + } + + /// Suspend and hibernate after `seconds` idle on the network. + private func idleTimers(_ host: HostID, _ seconds: Int) throws { + for setting in ["auto-suspend.timeout", "auto-hibernate.timeout"] { + try boxd(["machine", "config", "set", name(host), setting, String(seconds)]) + } + } + + /// A job of `seconds` that burns a core and uses no network, writing one line a second to + /// `~/ticks`, detached so it outlives the exec. A box that sleeps stops its clock, and the ticks. + private func startJob(_ driver: any HostDriver, _ host: HostID, seconds: Int) async throws { + try await onHost( + driver, host, + "rm -f ~/ticks; nohup sh -c 'i=0; while [ $i -lt \(seconds) ]; do i=$((i+1)); echo $i >> ~/ticks;" + + " t=$(date +%s); while [ $(date +%s) -eq $t ]; do :; done; done' > /dev/null 2>&1 &") + } + + /// The ticks the job logged. Reading them wakes the box. + private func ticks(_ driver: any HostDriver, _ host: HostID) async throws -> Int { + Int(try await onHost(driver, host, "wc -l < ~/ticks")) ?? 0 + } + + /// Polls boxd every 20 s until `host` is asleep or `limit` passes; when it fell asleep, measured + /// from `since`, or nil. + private func sleeps(_ host: HostID, since: ContinuousClock.Instant, within limit: Duration) + async throws -> Duration? + { + while ContinuousClock.now - since < limit { + if asleep(host) { return ContinuousClock.now - since } + try await Task.sleep(for: .seconds(20)) + } + return nil + } + + /// The app attached to `host` as it is to a workroom that is not selected: its `RemoteHosts` + /// holds the host's service connection (ssh, with keepalives) through `HostConnectionManager`. + /// The connect hands the agent no settings, so a test can set its own. + private func attach(_ driver: BoxdHostDriver, _ host: HostID) async throws -> RemoteHosts { + guard case .remote(let id) = host else { throw HostDriverError.unknownHost(host) } + let socket = driver.configuration.agentSocket + let remote = RemoteHosts( + makeDriver: { _ in driver }, + connectAgent: { host, driver in + try await AgentBootstrap.connect( + host: host, driver: driver, socket: socket, handOff: false, watched: false) + }) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "w", path: Self.path, vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id)) + ]) + ], sweep: false) + try await remote.connect(host, driver: driver) + attached.append(host) + return remote + } + + private var attached: [HostID] = [] + + /// The sidebar's badge on that workroom: its agent's status every 10 s, each verdict reported to + /// `RemoteHosts.observed`, as `WakefulnessModel` does. A host let go of answers nothing, and is + /// sent nothing. + private func pollLikeTheBadge(_ remote: RemoteHosts, _ host: HostID) -> Task { + Task { + while !Task.isCancelled { + if let status = try? await HostConnectionManager.shared.wakefulness(host: host).status() { + await remote.observed(host, busy: status.busy) + } + try? await Task.sleep(for: .seconds(10)) + } + } + } + + /// #257's acceptance, rerunnable (#356; it was a TODO). Two machines with 120 s timers run the same 6-minute job with no + /// network use and no client attached. The control, with no agent, sleeps mid-job, which is + /// what lets the other result mean anything; the agent's machine logs every tick, then sleeps + /// once the job has ended. About 10 minutes. + func testTheHeartbeatKeepsABusyBoxAwakeAndABoxWithoutOneSleeps() async throws { + let driver = driver() + let control = HostID.remote(UUID()) + try boxd([ + "machine", "new", name(control), "--auto-suspend-timeout=120", "--auto-hibernate-timeout=120", + ]) + let kept = try await driver.create() + try idleTimers(kept, 120) + // Connected once, to push the agent, then let go: no client is attached for the run. + let pushed = try await connect(driver, kept) + await pushed.close() + let start = ContinuousClock.now + try await startJob(driver, control, seconds: 360) + try await startJob(driver, kept, seconds: 360) + + let controlSlept = try await sleeps(control, since: start, within: .seconds(330)) + XCTAssertNotNil(controlSlept, "the control never slept mid-job, so the run proves nothing") + let keptSlept = try await sleeps(kept, since: start, within: .seconds(600)) + XCTAssertNotNil(keptSlept, "the agent's machine never slept after its job") + if let keptSlept { XCTAssertGreaterThan(keptSlept, .seconds(360), "it slept under its job") } + let controlTicks = try await ticks(driver, control) + let keptTicks = try await ticks(driver, kept) + XCTAssertLessThan(controlTicks, 360) + XCTAssertEqual(keptTicks, 360, "the agent's machine missed ticks") + } + + /// D8, first half: with the app attached (a watched connection, ssh keepalives, the badge + /// polling every 10 s), an idle box must still sleep, or every open workroom would keep its box + /// awake and billing. A fresh machine's start-up traffic holds its timer for about 290 s (#257's + /// acceptance), so it has 7 minutes. About 6 minutes. + func testAnIdleBoxStillSleepsWithTheAppAttached() async throws { + let driver = driver() + let host = try await driver.create() + try idleTimers(host, 120) + let remote = try await attach(driver, host) + let badge = pollLikeTheBadge(remote, host) + defer { badge.cancel() } + let slept = try await sleeps(host, since: .now, within: .seconds(420)) + XCTAssertTrue(remote.isLetGo(host), "the app never let go of the idle box") + XCTAssertNotNil(slept, "an idle box with the app attached never slept") + } + + /// D8, second half: with the app attached and a prompt nobody answers, a busy box past its + /// ceiling must sleep, or the prompt's "let it sleep" would mean nothing while the app is open. + /// Ceiling 60 s, prompt 30 s, timers 120 s, a 15-minute job; 8 minutes to sleep. About 7 minutes. + func testAnUnansweredPromptLetsABusyBoxSleepWithTheAppAttached() async throws { + let driver = driver() + let host = try await driver.create() + try idleTimers(host, 120) + let remote = try await attach(driver, host) + try await HostConnectionManager.shared.wakefulness(host: host).apply( + AgentWakefulnessSettings(ceiling: 60, promptTimeout: 30, ask: true)) + let start = ContinuousClock.now + try await startJob(driver, host, seconds: 900) + // The badge starts once the job is running, so the box is let go of for its unanswered prompt, + // not for being idle before the job began. + let deadline = ContinuousClock.now + .seconds(60) + while ContinuousClock.now < deadline, + (try? await HostConnectionManager.shared.wakefulness(host: host).status().busy) != true + { + try await Task.sleep(for: .seconds(2)) + } + let badge = pollLikeTheBadge(remote, host) + defer { badge.cancel() } + let slept = try await sleeps(host, since: start, within: .seconds(480)) + XCTAssertTrue(remote.isLetGo(host), "the app never let go of the box") + XCTAssertNotNil( + slept, "a busy box whose prompt went unanswered never slept with the app attached") + badge.cancel() + let logged = try await ticks(driver, host) + XCTAssertLessThan(logged, 900) + } } diff --git a/macapp/WorkroomAppTests/ContainerProvisioningIntegrationTests.swift b/macapp/WorkroomAppTests/ContainerProvisioningIntegrationTests.swift index cd4d38b33..ba6accdc7 100644 --- a/macapp/WorkroomAppTests/ContainerProvisioningIntegrationTests.swift +++ b/macapp/WorkroomAppTests/ContainerProvisioningIntegrationTests.swift @@ -417,7 +417,7 @@ final class ContainerProvisioningIntegrationTests: XCTestCase { do { _ = try await driver.deriveFromBase(base) XCTFail("a derive that failed succeeded") - } catch HostDriverError.leftBehind(let cause, let leftover) { + } catch HostDriverError.leftBehind(let cause, let leftover, _) { XCTAssertTrue(cause.contains("never minted"), cause) XCTAssertTrue(leftover.contains { $0.hasPrefix("container workroom-") }, "\(leftover)") } diff --git a/macapp/WorkroomAppTests/GhosttySurfaceViewTests.swift b/macapp/WorkroomAppTests/GhosttySurfaceViewTests.swift index bbe14c22b..4c78bfcda 100644 --- a/macapp/WorkroomAppTests/GhosttySurfaceViewTests.swift +++ b/macapp/WorkroomAppTests/GhosttySurfaceViewTests.swift @@ -349,6 +349,36 @@ final class RemotePaneReconnectTests: XCTestCase { XCTAssertTrue(view.persistentSessionIsRestored) } + // Value: protects=a pane holds its remote host only while its ssh lives, so a dead link, a closed + // pane or a pane pointed at another session lets an idle box go; fails_when=the surface's exit, + // teardown or session swap stops clearing the attach; why_new=only endSession and forget were + // covered; seam=none + /// A pane's ssh holds its box awake, and the app keeps the host while one is attached (#356). An + /// ssh that exits, a pane pointed at another session, or a closed pane holds nothing. Not + /// reached here: a window freeing a live surface without a teardown (`deinit`), which needs one. + func testAPaneHoldsItsHostOnlyWhileItsSshLives() throws { + let view = try remotePane(log: "") + let sessions = PersistentSessionService.shared + let session = try XCTUnwrap(view.persistentSessionID) + let host = try XCTUnwrap(sessions.remoteHost(of: session)) + addTeardownBlock { PersistentSessionService.shared.forgetRemoteSession(session) } + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: view.attachToken)) + XCTAssertTrue(sessions.hasAttachedPane(on: host)) + + view.handleChildExited(exitCode: 1) + XCTAssertFalse(sessions.hasAttachedPane(on: host), "a pane whose ssh exited holds its host") + + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: view.attachToken)) + view.persistentSessionID = UUID() + XCTAssertFalse( + sessions.hasAttachedPane(on: host), "a pane pointed at another session holds the old host") + view.persistentSessionID = session + + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: view.attachToken)) + view.tearDown() + XCTAssertFalse(sessions.hasAttachedPane(on: host), "a closed pane holds its host") + } + func testClosingThePaneCancelsAReconnectThatIsWaiting() throws { let view = try remotePane(log: "") view.handleChildExited(exitCode: 255) diff --git a/macapp/WorkroomAppTests/HostConnectionManagerTests.swift b/macapp/WorkroomAppTests/HostConnectionManagerTests.swift index 7c60920df..7a73fca41 100644 --- a/macapp/WorkroomAppTests/HostConnectionManagerTests.swift +++ b/macapp/WorkroomAppTests/HostConnectionManagerTests.swift @@ -23,6 +23,22 @@ final class HostConnectionManagerTests: XCTestCase { await connection.statusRelease.open() } + /// A let-go's disconnect asks, on the manager, whether the host was taken back meanwhile: one a + /// click or a selection took back keeps the connection it found up (#356). + func testADisconnectUnlessTakenBackKeepsATakenBackConnection() async throws { + let manager = HostConnectionManager() + let host = HostID.remote(UUID()) + let lease = try await manager.connect(host: host) { ConnectionFixture() } + let disconnectedTakenBack = await manager.disconnect(lease, unless: { true }) + XCTAssertFalse(disconnectedTakenBack) + var state = await manager.snapshot(for: host) + XCTAssertEqual(state.status, .connected, "a taken-back host was disconnected") + let disconnected = await manager.disconnect(lease, unless: { false }) + XCTAssertTrue(disconnected) + state = await manager.snapshot(for: host) + XCTAssertNotEqual(state.status, .connected) + } + func testAlreadyCancelledCallerCannotReplaceConnectionOrStartOperation() async throws { let manager = HostConnectionManager() let host = HostID.remote(UUID()) diff --git a/macapp/WorkroomAppTests/RemoteHostsTests.swift b/macapp/WorkroomAppTests/RemoteHostsTests.swift index ebdc2fe2d..0f740c0ca 100644 --- a/macapp/WorkroomAppTests/RemoteHostsTests.swift +++ b/macapp/WorkroomAppTests/RemoteHostsTests.swift @@ -87,6 +87,384 @@ final class RemoteHostsTests: XCTestCase { XCTAssertEqual(connects.calls, 2) } + /// A background read never connects to a boxd box boxd says is asleep: the ssh login would wake + /// it, and every status sweep would keep every box awake and billing (#356). boxd is asked once a + /// `retryAfter`; opening the workroom connects and wakes it; when boxd can't say, the read leaves + /// the box be too, so a broken CLI cannot wake every box, and asks again only after a + /// `retryAfter`; and when it says it's awake, the read goes ahead as before. + func testABackgroundReadLeavesAnAsleepBoxdBoxAsleep() async throws { + let connects = Connects() + connects.hold(false) + let asked = Asleep() + let remote = RemoteHosts( + connectHost: { try await connects.connect($0) }, isConnected: { _ in false }, + startHost: { _ in }, now: { connects.now }, + presence: { _ in asked.answer() }) + let id = UUID() + let host = HostID.remote(id) + remote.adopt([ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: id, + account: "usr_1")) + ]) + ]) + + asked.set(true) + for _ in 0..<2 { + do { + try await remote.ensureConnected(host) + XCTFail("a background read connected to an asleep box") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .asleep(host)) + } + } + XCTAssertEqual(connects.calls, 0) + XCTAssertEqual(asked.calls, 1, "boxd was asked again inside the window") + + // Opening the workroom wakes it. + remote.activate(host) + try await remote.ensureConnected(host) + XCTAssertEqual(connects.calls, 1) + + // boxd can't say: the read leaves the box be, and boxd isn't asked again inside the window. + asked.set(nil) + connects.advance(RemoteHosts.retryAfter) + let unsure = asked.calls + for _ in 0..<2 { + do { + try await remote.ensureConnected(host) + XCTFail("a background read connected to a box boxd could not say about") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .asleep(host)) + } + } + XCTAssertEqual(connects.calls, 1) + XCTAssertEqual( + asked.calls, unsure + 1, "boxd was asked again inside the window after no answer") + // boxd says it's running: the read connects as it always did. + asked.set(false) + connects.advance(RemoteHosts.retryAfter) + try await remote.ensureConnected(host) + XCTAssertEqual(connects.calls, 2) + + // A machine boxd says is gone is reported as gone, not asleep, and then left for the window. + asked.setGone() + connects.advance(RemoteHosts.retryAfter) + let before = asked.calls + for expected in [RepositoryRoutingError.gone(host), .unavailable(host)] { + do { + try await remote.ensureConnected(host) + XCTFail("a gone machine was connected to") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, expected) + } + } + XCTAssertEqual(asked.calls, before + 1, "boxd was asked again inside the window") + XCTAssertEqual(connects.calls, 2) + } + + /// boxd counts inbound traffic as activity, so a connection's keepalives and the badge's polls + /// held an idle box awake for good (measured live, #356). A boxd host that reports IDLE while + /// not selected is let go of, and a background read then never reconnects it; selecting or + /// opening its workroom does. A busy, selected or container host is kept. + @MainActor + func testAnIdleBoxdHostThatIsNotSelectedIsLetGoOf() async throws { + let connects = Connects() + connects.hold(false) + let asked = Asleep() + let remote = RemoteHosts( + connectHost: { try await connects.connect($0) }, isConnected: { _ in false }, + startHost: { _ in }, now: { connects.now }, presence: { _ in asked.answer() }) + let (boxd, container) = (UUID(), UUID()) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "b", path: "/home/boxd/r", vcsName: "workroom/b", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: boxd, account: "usr_1")), + Workroom( + name: "c", path: "/home/workroom/r", vcsName: "workroom/c", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.containerDriver, provisioner: RemoteWorkrooms.provisioner, + id: container)), + ]) + ], sweep: false) + let host = HostID.remote(boxd) + + let busyKept = await remote.observed(host, busy: true) + XCTAssertFalse(busyKept, "a busy box was let go of") + let containerKept = await remote.observed(.remote(container), busy: false) + XCTAssertFalse(containerKept, "a container was let go of") + remote.select(host) + let selectedKept = await remote.observed(host, busy: false) + XCTAssertFalse(selectedKept, "the selected workroom's box was let go of") + + remote.select(nil) + // Value: protects=a workroom whose pane attached keeps its status, as that pane's ssh holds the + // box awake, while a restored pane that never attached does not; fails_when=observed() stops + // checking panes, or counts a registration as attached; why_new=no test opens a pane; seam=none + let session = UUID() + let sessions = PersistentSessionService.shared + sessions.registerRemoteSession( + session, on: host, via: DerivingDriver(derived: boxd), workingDirectory: "/home/boxd/r") + // A restored pane registers its session but spawns nothing until it enters a window. + XCTAssertFalse(sessions.hasAttachedPane(on: host), "a pane that never attached holds the box") + // An attach no pane runs (a probe, a test's own call) holds nothing. + XCTAssertNotNil(sessions.attachCommand(forSession: session)) + XCTAssertFalse(sessions.hasAttachedPane(on: host), "an attach no pane runs holds the box") + // Value: protects=each pane clears only its own attach, so a pane freed late cannot clear a + // live pane's of the same session; fails_when=attaches are kept by session alone; why_new=the + // detach tests use one pane per session; seam=none + let (first, second) = (UUID(), UUID()) + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: first)) + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: second)) + sessions.paneDetached(session, by: first) + XCTAssertTrue(sessions.hasAttachedPane(on: host), "one pane's detach cleared another's attach") + let paneKept = await remote.observed(host, busy: false) + sessions.forgetRemoteSession(session) + XCTAssertFalse(paneKept, "a host with a pane attached was let go of") + // A forgotten pane holds nothing, even if its id is registered again; nor does one whose host + // the driver can't reach, whose pane only says why. + sessions.registerRemoteSession( + session, on: host, via: DerivingDriver(derived: boxd), workingDirectory: "/home/boxd/r") + XCTAssertFalse(sessions.hasAttachedPane(on: host), "a forgotten pane still holds the box") + sessions.forgetRemoteSession(session) + sessions.registerRemoteSession( + session, on: host, via: ContainerHostDriver(hosts: [:], directory: RemoteHosts.directory), + workingDirectory: "/home/boxd/r") + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: UUID())) + XCTAssertFalse(sessions.hasAttachedPane(on: host), "a pane that can't reach its host holds it") + sessions.forgetRemoteSession(session) + + let letGo = await remote.observed(host, busy: false) + XCTAssertTrue(letGo) + XCTAssertTrue(remote.isLetGo(host)) + do { + try await remote.ensureConnected(host) + XCTFail("a background read reconnected a box let go of as idle") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .idle(host)) + } + XCTAssertEqual(connects.calls, 0) + XCTAssertEqual(asked.calls, 0, "boxd was asked about a box already let go of") + + // Selecting its workroom takes it back. + remote.select(host) + XCTAssertFalse(remote.isLetGo(host)) + try await remote.ensureConnected(host) + XCTAssertEqual(connects.calls, 1) + } + + /// A click reaches a boxd host a background read would leave alone (#356): closing a pane (to + /// end its session there) and "Keep awake" wake a box let go of as idle, or asleep, where a + /// status read is refused. + @MainActor + func testAClickWakesABoxdHostABackgroundReadLeavesAlone() async throws { + let connects = Connects() + connects.hold(false) + let asked = Asleep() + asked.set(true) + let remote = RemoteHosts( + connectHost: { try await connects.connect($0) }, isConnected: { _ in false }, + startHost: { _ in }, now: { connects.now }, presence: { _ in asked.answer() }) + let id = UUID() + let host = HostID.remote(id) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "b", path: "/home/boxd/r", vcsName: "workroom/b", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id, account: "usr_1")) + ]) + ], sweep: false) + + // Asleep: a read is refused, a click connects. + do { + try await remote.ensureConnected(host) + XCTFail("a background read woke an asleep box") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .asleep(host)) + } + try await remote.ensureConnected(host, wake: true) + XCTAssertEqual(connects.calls, 1) + + // Let go of as idle: the same, and the click takes it back. + asked.set(false) + // Not within the grace after its connect, so the read that connected it is answered first. + let tooSoon = await remote.observed(host, busy: false) + XCTAssertFalse(tooSoon, "a host was let go of straight after its connect") + // The grace is one poll: a reading just inside it is refused, the next poll's may let it go. + connects.advance(RemoteHosts.letGoGrace - .seconds(1)) + let stillTooSoon = await remote.observed(host, busy: false) + XCTAssertFalse(stillTooSoon, "a host was let go of within its grace") + connects.advance(.seconds(1)) + let letGo = await remote.observed(host, busy: false) + XCTAssertTrue(letGo) + try await remote.ensureConnected(host, wake: true) + XCTAssertEqual(connects.calls, 2) + XCTAssertFalse(remote.isLetGo(host), "a click left the host let go of") + + // A click right after one that failed is still tried: waking a box can fail once while it + // resumes, and the retry must not wait out the failure window. + connects.fail(true) + do { + try await remote.ensureConnected(host, wake: true) + XCTFail("the failing connect succeeded") + } catch {} + connects.fail(false) + try await remote.ensureConnected(host, wake: true) + XCTAssertEqual(connects.calls, 4, "a second click inside the window was refused") + } + + /// A box let go of as idle is asked about again once its idle window has passed (#356): boxd + /// sleeps a box nothing holds by then, so one still awake has work holding it, and is connected + /// again so its ceiling prompts are heard. Inside the window boxd is not asked. + @MainActor + func testABoxLetGoOfIsAskedAboutAgainOnceItsIdleWindowHasPassed() async throws { + let connects = Connects() + connects.hold(false) + let asked = Asleep() + asked.set(false) + let remote = RemoteHosts( + connectHost: { try await connects.connect($0) }, isConnected: { _ in false }, + startHost: { _ in }, now: { connects.now }, presence: { _ in asked.answer() }) + let id = UUID() + let host = HostID.remote(id) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "b", path: "/home/boxd/r", vcsName: "workroom/b", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id, account: "usr_1")) + ]) + ], sweep: false) + let letGo = await remote.observed(host, busy: false) + XCTAssertTrue(letGo) + + connects.advance(remote.revisitAfter(host) - .seconds(1)) + do { + try await remote.ensureConnected(host) + XCTFail("a box let go of was reconnected inside its idle window") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .idle(host)) + } + XCTAssertEqual(asked.calls, 0) + + // Value: protects=a box let go of that boxd says is asleep past its idle window is not woken by a read; + // fails_when=the revisit connects without asking boxd, or keeps asking it inside retryAfter; + // why_new=the revisit was only tested with boxd answering awake; seam=none + asked.set(true) + connects.advance(.seconds(1)) + for _ in 0..<2 { + do { + try await remote.ensureConnected(host) + XCTFail("a read past the idle window woke a box boxd says is asleep") + } catch { + XCTAssertEqual(error as? RepositoryRoutingError, .asleep(host)) + } + } + XCTAssertEqual(asked.calls, 1, "boxd was asked again inside retryAfter") + XCTAssertEqual(connects.calls, 0) + + asked.set(false) + connects.advance(RemoteHosts.retryAfter) + try await remote.ensureConnected(host) + XCTAssertEqual(asked.calls, 2) + XCTAssertEqual(connects.calls, 1, "a box still awake past its idle window was left alone") + XCTAssertFalse(remote.isLetGo(host)) + } + + /// A selected workroom's box is reached by a background read, asleep or let go of (#356): after + /// the Mac sleeps, the open workroom must not read as asleep until it is selected again. + @MainActor + func testASelectedWorkroomsBoxIsReachedEvenAsleep() async throws { + let connects = Connects() + connects.hold(false) + let asked = Asleep() + asked.set(true) + let remote = RemoteHosts( + connectHost: { try await connects.connect($0) }, isConnected: { _ in false }, + startHost: { _ in }, now: { connects.now }, presence: { _ in asked.answer() }) + let id = UUID() + let host = HostID.remote(id) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "b", path: "/home/boxd/r", vcsName: "workroom/b", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id, account: "usr_1")) + ]) + ], sweep: false) + remote.select(host, in: "window") + try await remote.ensureConnected(host) + XCTAssertEqual(connects.calls, 1) + XCTAssertEqual(asked.calls, 0, "boxd was asked about the selected workroom's box") + } + + /// Each window has its own selection (one AppStore per window): a host selected in any window + /// is kept, whatever another window selects, and a closed window lets it go (#356). + @MainActor + func testAHostSelectedInAnyWindowIsKept() async throws { + let remote = RemoteHosts() + let id = UUID() + let host = HostID.remote(id) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "b", path: "/home/boxd/r", vcsName: "workroom/b", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id, account: "usr_1")) + ]) + ], sweep: false) + remote.select(host, in: "window A") + remote.select(nil, in: "window B") + let keptAcrossWindows = await remote.observed(host, busy: false) + XCTAssertFalse(keptAcrossWindows, "another window's selection let go of window A's host") + remote.select(nil, in: "window A") + let letGoOnceUnselected = await remote.observed(host, busy: false) + XCTAssertTrue(letGoOnceUnselected) + } + + private final class Asleep: @unchecked Sendable { + private let lock = NSLock() + private var value: BoxdHostDriver.Presence? + private var count = 0 + var calls: Int { lock.withLock { count } } + func set(_ asleep: Bool?) { lock.withLock { value = asleep.map { $0 ? .asleep : .awake } } } + func setGone() { lock.withLock { value = .gone } } + func answer() -> BoxdHostDriver.Presence? { + lock.withLock { + count += 1 + return value + } + } + } + /// Only a host whose workroom the user opened has its container started before connecting: the /// status sweep connects to every remote workroom, and must not start them all (#309). func testOnlyAnOpenedHostIsStartedBeforeItsConnect() async throws { @@ -625,6 +1003,210 @@ final class RemoteHostsTests: XCTestCase { image: nil, context: context) } + // Value: protects=a boxd box with a provider idle timer shorter than the heartbeat is flagged on its badge; + // fails_when=connect stops reading the machine's idle timers into its wakefulness model, or reads them wrongly; + // why_new=idleWindow parsing and the badge copy are unit-tested but nothing sets the model on connect; seam=none + /// Connecting a boxd host reads its idle timers into its wakefulness model (#356), so the badge + /// can say a 60 s timer sleeps the box under a job the once-a-minute heartbeat cannot hold. + @MainActor + func testConnectingABoxdHostGivesItsBadgeTheMachinesIdleWindow() async throws { + let fake = try FakeAgent(version: 4, status: true) + defer { fake.stop() } + let id = UUID() + let host = HostID.remote(id) + defer { WakefulnessModel.forgetHost(id) } + let remote = RemoteHosts(connectAgent: { host, _ in + try await AgentVCSConnection.connect(host: host, socketPath: fake.socketPath) + }) + let driver = BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd")), + directory: FileManager.default.temporaryDirectory, + runner: MachineGet(#"{"source":"standalone","auto_suspend":60}"#)) + + try await remote.connect(host, driver: driver) + let snapshot = await HostConnectionManager.shared.snapshot(for: host) + let lease = try XCTUnwrap(snapshot.lease) + addTeardownBlock { await HostConnectionManager.shared.disconnect(lease) } + + let model = try XCTUnwrap(WakefulnessModel.Hosts.shared.models[id]) + XCTAssertTrue(model.hostSleeps) + let deadline = ContinuousClock.now + .seconds(5) + while model.idleWindow == nil, ContinuousClock.now < deadline { + try await Task.sleep(for: .milliseconds(20)) + } + XCTAssertEqual(model.idleWindow, 60) + // Value: protects=a connected boxd box is polled off screen, revisited by its own idle window, and a deleted one stops polling; + // fails_when=connect stops starting the poll or recording the window, or forgetHost leaves the poll running; + // why_new=the poll test starts it by hand and the revisit test only reaches the unknown-window fallback; seam=none + XCTAssertTrue(model.isPollingWhileConnected, "a connected boxd host isn't polled") + XCTAssertEqual(remote.revisitAfter(host), .seconds(60) + RemoteHosts.retryAfter) + + // Value: protects=a reconnect whose idle-window read fails keeps the window last read, while + // one that finds no timer set clears it; fails_when=connect treats a failed read as "no + // timer", or "no timer" as a failure; why_new=only a successful read is connected here; + // seam=none + let reconnect = { (runner: MachineGet) in + if let current = await HostConnectionManager.shared.snapshot(for: host).lease { + await HostConnectionManager.shared.disconnect(current) + } + try await remote.connect( + host, + driver: BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd")), + directory: FileManager.default.temporaryDirectory, runner: runner)) + } + let failing = MachineGet("", fails: true) + try await reconnect(failing) + let asked = ContinuousClock.now + .seconds(5) + while failing.answered.calls == 0, ContinuousClock.now < asked { + try await Task.sleep(for: .milliseconds(10)) + } + XCTAssertEqual(failing.answered.calls, 1, "the reconnect never read the idle window") + try await Task.sleep(for: .milliseconds(50)) + XCTAssertEqual(remote.revisitAfter(host), .seconds(60) + RemoteHosts.retryAfter) + XCTAssertEqual(model.idleWindow, 60, "a failed read forgot the badge's idle window") + + try await reconnect(MachineGet(#"{"source":"standalone"}"#)) + let cleared = ContinuousClock.now + .seconds(5) + while model.idleWindow != nil, ContinuousClock.now < cleared { + try await Task.sleep(for: .milliseconds(10)) + } + XCTAssertNil(model.idleWindow, "turning the timers off left the badge's idle window") + XCTAssertEqual( + remote.revisitAfter(host), .seconds(RemoteHosts.unknownIdleWindow) + RemoteHosts.retryAfter) + if let last = await HostConnectionManager.shared.snapshot(for: host).lease { + addTeardownBlock { await HostConnectionManager.shared.disconnect(last) } + } + WakefulnessModel.forgetHost(id) + XCTAssertFalse(model.isPollingWhileConnected, "a forgotten host is still polled") + } + + // Value: protects=a boxd workroom deleted while its connect reads the idle timers stays deleted; + // fails_when=the late read looks the host's model up with the accessor that makes one; + // why_new=the idle-window test never deletes across the read; seam=none + /// A delete that lands while the connect is still reading the machine's idle timers forgets the + /// host's model, and the late answer does not make it anew, watching for prompts for good (#356). + @MainActor + func testADeleteAcrossTheIdleWindowReadDoesNotBringTheHostsModelBack() async throws { + let fake = try FakeAgent(version: 4, status: true) + defer { fake.stop() } + let id = UUID() + let host = HostID.remote(id) + defer { WakefulnessModel.forgetHost(id) } + let remote = RemoteHosts(connectAgent: { host, _ in + try await AgentVCSConnection.connect(host: host, socketPath: fake.socketPath) + }) + let (opened, open) = AsyncStream.makeStream() + let read = MachineGet(#"{"source":"standalone","auto_suspend":60}"#, until: opened) + try await remote.connect( + host, + driver: BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd")), + directory: FileManager.default.temporaryDirectory, runner: read)) + if let lease = await HostConnectionManager.shared.snapshot(for: host).lease { + addTeardownBlock { await HostConnectionManager.shared.disconnect(lease) } + } + XCTAssertNotNil(WakefulnessModel.Hosts.shared.models[id]) + + WakefulnessModel.forgetHost(id) + open.yield() + open.finish() + // The window is recorded just before the hop that would give it to a model. + let deadline = ContinuousClock.now + .seconds(5) + while remote.revisitAfter(host) != .seconds(60) + RemoteHosts.retryAfter, + ContinuousClock.now < deadline + { + try await Task.sleep(for: .milliseconds(10)) + } + XCTAssertEqual(read.answered.calls, 1, "the connect never read the idle window") + try await Task.sleep(for: .milliseconds(200)) + XCTAssertNil( + WakefulnessModel.Hosts.shared.models[id], "a late read brought a deleted host back") + } + + // Value: protects=a boxd host carrying a port forward stays connected when idle, so the user's dev server still answers; + // fails_when=observed() lets go of an idle boxd host without asking PortForwardingModel.hasForwards; + // why_new=pass-1 covers the idle release, not its one exception; no test pairs a live forward with observed; seam=none + /// An idle boxd host is let go of (#356) unless it is carrying a forward: closing its connection + /// would close the listener the user's browser is pointed at. + @MainActor + func testAnIdleBoxdHostWithAPortForwardIsKept() async throws { + let fake = try FakeAgent(version: 5, forward: true) + defer { fake.stop() } + let id = UUID() + let host = HostID.remote(id) + let clock = Connects() + let remote = RemoteHosts( + now: { clock.now }, + connectAgent: { host, _ in + try await AgentVCSConnection.connect(host: host, socketPath: fake.socketPath) + }) + remote.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id, org: "acme", account: "usr_1")) + ]) + ], sweep: false) + let driver = BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd")), + directory: FileManager.default.temporaryDirectory, + runner: MachineGet(#"{"source":"standalone","auto_suspend":60}"#)) + try await remote.connect(host, driver: driver) + let snapshot = await HostConnectionManager.shared.snapshot(for: host) + let lease = try XCTUnwrap(snapshot.lease) + addTeardownBlock { await HostConnectionManager.shared.disconnect(lease) } + defer { WakefulnessModel.forgetHost(id) } + let forwards = PortForwardingModel.model(for: host) + forwards.draft = "8080" + await forwards.add() + let forward = try XCTUnwrap(forwards.forwards.first, forwards.message ?? "no forward was made") + clock.advance(RemoteHosts.letGoGrace) + + let whileForwarding = await remote.observed(host, busy: false) + + XCTAssertFalse(whileForwarding, "an idle host carrying a forward was let go of") + XCTAssertFalse(remote.isLetGo(host)) + forwards.remove(forward.id) + // Value: protects=a boxd host let go of as idle stops being polled, so the poll can't hold it awake; + // fails_when=observed() closes the connection but leaves the connection poll running; + // why_new=no test lets go of a host whose connect started the poll; seam=none + let model = try XCTUnwrap(WakefulnessModel.Hosts.shared.models[id]) + XCTAssertTrue(model.isPollingWhileConnected) + let afterwards = await remote.observed(host, busy: false) + XCTAssertTrue(afterwards, "the host was never eligible, so the forward proved nothing") + XCTAssertFalse(model.isPollingWhileConnected, "a host let go of is still polled") + } + + /// A boxd CLI that answers every `machine get` with `json`. + private struct MachineGet: StatusCommandRunning { + let json: String + var fails = false + /// How many times it has answered, so a test can wait for a read it can't otherwise see. + let answered = Asleep() + /// Holds each answer back until it yields, for a test that acts across a read. + let until: AsyncStream? + init(_ json: String, fails: Bool = false, until: AsyncStream? = nil) { + self.json = json + self.fails = fails + self.until = until + } + func run(_ executable: String, _ args: [String], in directory: String, timeout: TimeInterval) + async -> CommandResult + { + if let until { for await _ in until { break } } + _ = answered.answer() + return fails + ? CommandResult(stdout: "", stderr: "error: unreachable", exitCode: 1, timedOut: false) + : CommandResult(stdout: json, stderr: "", exitCode: 0, timedOut: false) + } + } + /// A host's ceiling prompts are watched from its connect (#257): the connect hands its agent this /// Mac's ask-at-ceiling setting, so a prompt that reached no card would let the box sleep under a /// running job. The model is made, watching, and a prompt the agent raises shows on it. @@ -785,13 +1367,355 @@ final class RemoteHostsTests: XCTestCase { remote.adopt([project]) - XCTAssertNil(try XCTUnwrap(remote.existingDriver(holding: base)).provisioning?.context) - XCTAssertEqual(remote.existingDriver(holding: pinned)?.provisioning?.context, "orbstack") + XCTAssertNil( + try XCTUnwrap(remote.existingDriver(holding: base) as? ContainerHostDriver).provisioning? + .context) + XCTAssertEqual( + (remote.existingDriver(holding: pinned) as? ContainerHostDriver)?.provisioning?.context, + "orbstack") for _ in 0..<500 where swept.calls.count < 2 { try await Task.sleep(for: .milliseconds(2)) } XCTAssertEqual(Set(swept.calls.map(\.context)), [nil, "orbstack"]) for call in swept.calls { XCTAssertEqual(call.known, [base, pinned], call.context ?? "nil") } } + /// A boxd host (#356) is its own driver key, never Docker's: before boxd had a key, a delete of + /// one fell back to Docker (`DriverKey(host) ?? DriverKey()`), and a descriptor naming a driver + /// this build doesn't know is refused rather than taken down with another. + func testABoxdHostIsItsOwnDriverKeyNeverDockers() throws { + let boxd = HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: UUID(), + org: "acme", account: "usr_1") + let key = RemoteHosts.DriverKey(boxd) + XCTAssertEqual(key, .boxd(org: "acme", account: "usr_1")) + XCTAssertNil(key?.runtime) + XCTAssertEqual(key?.place, .boxd) + // A boxd machine logs in as `boxd`, so the clone goes in its home, not the container user's. + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + XCTAssertEqual( + RemoteWorkrooms.clonePath(for: repository, on: try XCTUnwrap(key)), "/home/boxd/r") + XCTAssertEqual(RemoteWorkrooms.clonePath(for: repository), "/home/workroom/r") + XCTAssertEqual(try RemoteHosts.deletionKey(boxd), .boxd(org: "acme", account: "usr_1")) + // A descriptor from before the `driver` field is Docker's, as it always was. + XCTAssertEqual(try RemoteHosts.deletionKey(HostDescriptor(id: UUID())), RemoteHosts.DriverKey()) + XCTAssertThrowsError(try RemoteHosts.deletionKey(HostDescriptor(driver: "exe.dev", id: UUID()))) + // A boxd base serves only its own org and account; there is no unpinned form to fall back to. + XCTAssertEqual( + RemoteWorkrooms.base(in: boxd, for: .boxd(org: "acme", account: "usr_1"))?.id, boxd.id) + XCTAssertNil(RemoteWorkrooms.base(in: boxd, for: .boxd(org: nil, account: "usr_1"))) + XCTAssertNil(RemoteWorkrooms.base(in: boxd, for: RemoteHosts.DriverKey())) + // And the descriptor round-trips through config with its org and account. + XCTAssertEqual( + try JSONDecoder().decode(HostDescriptor.self, from: try JSONEncoder().encode(boxd)), boxd) + } + + /// A delete of a boxd host asks for boxd's driver and environment, never Docker's. + @MainActor + func testDeletingABoxdHostUsesTheBoxdDriver() throws { + let asked = Asked() + let remote = RemoteHosts(makeDriver: { key in + asked.add(key) + return BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/nonexistent/boxd")), + directory: FileManager.default.temporaryDirectory) + }) + let host = HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: UUID(), + org: "acme", account: "usr_1") + let deletion = try XCTUnwrap(try remote.environment(toDelete: [host])) + XCTAssertEqual(asked.keys, [.boxd(org: "acme", account: "usr_1")]) + let environment = try XCTUnwrap(deletion.environment(for: host)) + XCTAssertTrue(environment.driver is BoxdHostDriver) + XCTAssertEqual( + environment.agentSocket, + BoxdHostDriver.Configuration(cli: URL(fileURLWithPath: "/boxd")).agentSocket) + XCTAssertNil(environment.gitHubToken, "a remote host's git never borrows the Mac's gh") + } + + /// After a relaunch a boxd workroom is reached without Docker: its driver is made by name, with + /// the org and account its record holds, and a missing container runtime never reads as its host + /// being gone, which would report a closed pane's session ended while it runs on (#356). + func testABoxdWorkroomIsReachedAfterARelaunchWithoutDocker() throws { + let remote = RemoteHosts() + let id = UUID() + let project = Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: id, + org: "acme", account: "usr_1")) + ]) + + remote.adopt([project]) + + let driver = try XCTUnwrap(remote.existingDriver(holding: id) as? BoxdHostDriver) + XCTAssertEqual(driver.configuration.org, "acme") + XCTAssertEqual(driver.configuration.account, "usr_1") + // Discriminates only where no container runtime is installed (CI's runners): on a Mac with + // Docker, `runtimeIsMissing` is false for any host, and no seam-free way picks the runtime + // lookup (`RemoteHosts.executable`, fixed paths). Kept for CI; eng review D6. + XCTAssertFalse(remote.runtimeIsMissing(for: .remote(id))) + } + + /// A workroom created on boxd records `driver: "boxd"` with its org and account, and no + /// container record, from the moment its machine exists; a base of another place is refused. + func testABoxdWorkroomRecordsItsOrgAndAccount() async throws { + let made = UUID() + let driver = DerivingDriver(derived: made) + let key = RemoteHosts.DriverKey.boxd(org: "acme", account: "usr_1") + let environment = RemoteProvisioning.Environment( + driver: driver, + agentSocket: BoxdHostDriver.Configuration(cli: URL(fileURLWithPath: "/boxd")).agentSocket, + client: BrokerClient( + baseURL: BrokerEndpoint.development, key: .software(P256.Signing.PrivateKey())), + connect: { _ in throw HostDriverError.provisioning("stop after the checkpoint") }) + let base = { (driver: String, org: String?, account: String?) in + HostDescriptor( + driver: driver, provisioner: RemoteWorkrooms.provisioner, id: UUID(), repository: "o/r", + cloneURL: "https://github.com/o/r.git", path: "/home/boxd/r", org: org, account: account) + } + let recorded = Recorded() + let recorder = RemoteWorkrooms.Recorder( + reserve: { _, descriptor in + recorded.add(descriptor) + return "x" + }, record: { _, descriptor in recorded.add(descriptor) }, forget: { _ in }) + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", + base: base(RemoteWorkrooms.boxdDriver, "acme", "usr_1"), key: key, driver: driver, + environment: environment, recorder: recorder) + XCTFail("the connect was meant to fail") + } catch {} + let live = try XCTUnwrap(recorded.all.last { $0.id == made }) + XCTAssertEqual(live.driver, RemoteWorkrooms.boxdDriver) + XCTAssertEqual(live.org, "acme") + XCTAssertEqual(live.account, "usr_1") + XCTAssertNil(live.container) + XCTAssertTrue( + recorded.all.allSatisfy { $0.driver == RemoteWorkrooms.boxdDriver && $0.account == "usr_1" }) + + for (place, other, said) in [ + (key, base(RemoteWorkrooms.containerDriver, nil, nil), "Docker"), + (RemoteHosts.DriverKey(), base(RemoteWorkrooms.boxdDriver, "acme", "usr_1"), "boxd"), + ] { + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", base: other, key: place, + driver: driver, environment: environment, + recorder: RemoteWorkrooms.Recorder( + reserve: { _, _ in "x" }, record: { _, _ in XCTFail("recorded") }, forget: { _ in })) + XCTFail("a workroom went on another place than its base") + } catch RemoteWorkrooms.Failure.baseOnOtherRuntime(let name) { + XCTAssertEqual(name, said) + } + } + } + + // Value: protects=a derive that fails and cannot remove its machine keeps the workroom's entry, + // failed, naming the machine, so a delete can take the paid machine down; fails_when=the + // catch-all forgets the entry on leftBehind; why_new=no test fails a derive's rollback; seam=none + /// A derive whose rollback leaves its machine running keeps the workroom's entry, `failed`, + /// naming the machine, rather than forgetting it: otherwise a paid machine runs on with nothing + /// in the app to find it (#356). + func testADeriveThatLeavesItsMachineBehindKeepsItsEntry() async throws { + let made = UUID() + let driver = DerivingDriver(derived: made, leavesItBehind: true) + let key = RemoteHosts.DriverKey.boxd(org: "acme", account: "usr_1") + let environment = RemoteProvisioning.Environment( + driver: driver, + agentSocket: BoxdHostDriver.Configuration(cli: URL(fileURLWithPath: "/boxd")).agentSocket, + client: BrokerClient( + baseURL: BrokerEndpoint.development, key: .software(P256.Signing.PrivateKey()))) + let recorded = Recorded() + let forgot = Recorded() + let recorder = RemoteWorkrooms.Recorder( + reserve: { _, descriptor in + recorded.add(descriptor) + return "x" + }, record: { name, descriptor in recorded.add(descriptor, as: name) }, + forget: { _ in forgot.add(HostDescriptor()) }) + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", + base: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: UUID(), + repository: "o/r", cloneURL: "https://github.com/o/r.git", path: "/home/boxd/r", + org: "acme", account: "usr_1"), + key: key, driver: driver, environment: environment, recorder: recorder) + XCTFail("a derive that left its machine behind succeeded") + } catch RemoteProvisioning.Failure.rollbackIncomplete(_, let host, _, let cleanup) { + XCTAssertEqual(host, .remote(made)) + XCTAssertEqual(cleanup, ["machine x"]) + } + XCTAssertTrue(forgot.all.isEmpty, "the entry naming a live machine was forgotten") + let kept = try XCTUnwrap(recorded.all.last) + XCTAssertEqual(kept.id, made) + XCTAssertEqual(kept.state, "failed") + XCTAssertNil(kept.grantID) + // Over the entry reserved for it, not beside it: the `creating` placeholder would dangle. + XCTAssertEqual(recorded.names.last, "x") + } + + // Value: protects=a base create whose rollback fails records the base machine failed on the + // project, so the project knows it and a delete takes it down; fails_when=the leftBehind catch is + // dropped or its record is not written; why_new=the derive test passes a base and only covers the + // derive catch; seam=none + /// A project's first boxd create whose rollback leaves its machine running records that base on + /// the project, `failed`, rather than forgetting a paid machine (#356, #370). + func testABaseCreateThatLeavesItsMachineBehindIsRecorded() async throws { + let made = UUID() + let driver = DerivingDriver(derived: made, createLeavesItBehind: true) + let key = RemoteHosts.DriverKey.boxd(org: "acme", account: "usr_1") + let environment = RemoteProvisioning.Environment( + driver: driver, + agentSocket: BoxdHostDriver.Configuration(cli: URL(fileURLWithPath: "/boxd")).agentSocket, + client: BrokerClient( + baseURL: BrokerEndpoint.development, key: .software(P256.Signing.PrivateKey()))) + let recorded = Recorded() + let forgot = Recorded() + let recorder = RemoteWorkrooms.Recorder( + reserve: { _, descriptor in + recorded.add(descriptor) + return "x" + }, record: { _, descriptor in recorded.add(descriptor) }, + forget: { _ in forgot.add(HostDescriptor()) }) + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", base: nil, key: key, + driver: driver, environment: environment, recorder: recorder) + XCTFail("a base create that left its machine behind succeeded") + } catch HostDriverError.leftBehind(_, let left, let host) { + XCTAssertEqual(host, .remote(made)) + XCTAssertEqual(left, ["machine x"]) + } + XCTAssertTrue(forgot.all.isEmpty) + let failed = try XCTUnwrap( + recorded.all.flatMap { [$0] + ($0.bases ?? []) }.last { $0.id == made }, + "the base machine left behind was not recorded") + XCTAssertEqual(failed.state, "failed") + XCTAssertEqual(failed.driver, RemoteWorkrooms.boxdDriver) + XCTAssertEqual(failed.account, "usr_1") + } + + // Value: protects=a base whose build fails after its machine exists, and whose rollback cannot + // destroy it, is recorded failed on the project; fails_when=only the driver's own leftBehind is + // recorded; why_new=the other base test fails inside create(); seam=none + /// A base build that fails after its machine exists (here its first connect), with a destroy that + /// fails too, records the machine on the project, `failed`, as a create's own undo does (#356). + func testABaseBuildWhoseRollbackFailsIsRecorded() async throws { + let made = UUID() + let driver = DerivingDriver(derived: made, createMakes: true, destroyFails: true) + let environment = RemoteProvisioning.Environment( + driver: driver, + agentSocket: BoxdHostDriver.Configuration(cli: URL(fileURLWithPath: "/boxd")).agentSocket, + client: BrokerClient( + baseURL: BrokerEndpoint.development, key: .software(P256.Signing.PrivateKey())), + connect: { _ in throw HostDriverError.provisioning("no agent") }) + let recorded = Recorded() + let recorder = RemoteWorkrooms.Recorder( + reserve: { _, _ in "x" }, record: { _, descriptor in recorded.add(descriptor) }, + forget: { _ in }) + let repository = try XCTUnwrap(GitHubRepository(host: "github.com", owner: "o", name: "r")) + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", base: nil, + key: .boxd(org: "acme", account: "usr_1"), driver: driver, environment: environment, + recorder: recorder) + XCTFail("a base build whose rollback failed succeeded") + } catch RemoteProvisioning.Failure.rollbackIncomplete(_, let host, _, _) { + XCTAssertEqual(host, .remote(made)) + } + let failed = try XCTUnwrap( + recorded.all.flatMap { [$0] + ($0.bases ?? []) }.last { $0.id == made }, + "the base machine left running was not recorded") + XCTAssertEqual(failed.state, "failed") + + // A record that fails too leaves the caller the original failure, not the record's. + do { + _ = try await RemoteWorkrooms.create( + repository: repository, cloneURL: "https://github.com/o/r.git", base: nil, + key: .boxd(org: "acme", account: "usr_1"), driver: driver, environment: environment, + recorder: RemoteWorkrooms.Recorder( + reserve: { _, _ in "x" }, + record: { _, _ in throw HostDriverError.provisioning("no record") }, + forget: { _ in })) + XCTFail("a base build whose rollback failed succeeded") + } catch RemoteProvisioning.Failure.rollbackIncomplete(_, let host, _, _) { + XCTAssertEqual(host, .remote(made)) + } + } + + /// A driver whose derive makes `derived` and whose destroy succeeds unless told otherwise; + /// `create` fails, leaving `derived` behind or not, unless `createMakes`. + private struct DerivingDriver: HostTerminalDriver { + let derived: UUID + /// The derive fails and cannot remove the machine it made. + var leavesItBehind = false + /// The create fails and cannot remove the machine it made. + var createLeavesItBehind = false + /// The create makes `derived`, and a destroy of it fails. + var createMakes = false + var destroyFails = false + var traits: HostDriverTraits { + HostDriverTraits( + transport: .sshStdio, deriveSpeed: nil, deriveCarriesLiveProcesses: false, + durableDisk: true, maxLifetime: nil, keepAwakeHoldsCredential: false, + sleepsWhenIdle: true) + } + func create() async throws -> HostID { + if createMakes { return .remote(derived) } + guard createLeavesItBehind else { throw HostDriverError.provisioning("no create") } + throw HostDriverError.leftBehind( + cause: "setup timed out", leftover: ["machine x"], host: .remote(derived)) + } + func deriveFromBase(_ base: HostID) async throws -> HostID { + guard leavesItBehind else { return .remote(derived) } + throw HostDriverError.leftBehind( + cause: "reboot timed out", leftover: ["machine x"], host: .remote(derived)) + } + func destroy(_ host: HostID) async throws { + if destroyFails { throw HostDriverError.provisioning("remove failed") } + } + func openStream(to host: HostID) async throws -> HostStream { + throw HostDriverError.provisioning("no stream") + } + func exec(_ command: String, on host: HostID) async throws -> HostStream { + throw HostDriverError.provisioning("no exec") + } + func attachCommand( + to host: HostID, session: UUID, workingDirectory: String, restored: Bool, + metadata: [(key: String, value: String)] + ) throws -> String { "" } + func hostRefusedLastAttach(of session: UUID, on host: HostID) -> Bool { false } + } + + private final class Recorded: @unchecked Sendable { + private let lock = NSLock() + private var made: [HostDescriptor] = [] + private var under: [String?] = [] + var all: [HostDescriptor] { lock.withLock { made } } + /// The entry name each descriptor was recorded under, in step with `all`. + var names: [String?] { lock.withLock { under } } + func add(_ descriptor: HostDescriptor, as name: String? = nil) { + lock.withLock { + made.append(descriptor) + under.append(name) + } + } + } + + private final class Asked: @unchecked Sendable { + private let lock = NSLock() + private var made: [RemoteHosts.DriverKey] = [] + var keys: [RemoteHosts.DriverKey] { lock.withLock { made } } + func add(_ key: RemoteHosts.DriverKey) { lock.withLock { made.append(key) } } + } + private final class Swept: @unchecked Sendable { private let lock = NSLock() private var made: [(context: String?, known: Set)] = [] @@ -1113,8 +2037,12 @@ final class RemoteHostsTests: XCTestCase { host: host(apple, .apple)) ], host: host(docker, .docker)) remote.adopt([project]) - XCTAssertEqual(remote.existingDriver(holding: docker)?.provisioning?.dialect, .docker) - XCTAssertEqual(remote.existingDriver(holding: apple)?.provisioning?.dialect, .apple) + XCTAssertEqual( + (remote.existingDriver(holding: docker) as? ContainerHostDriver)?.provisioning?.dialect, + .docker) + XCTAssertEqual( + (remote.existingDriver(holding: apple) as? ContainerHostDriver)?.provisioning?.dialect, + .apple) } /// A project's workrooms are derived from its base, so they go on the base's runtime: asking @@ -1134,7 +2062,7 @@ final class RemoteHostsTests: XCTestCase { driver: RemoteWorkrooms.Runtime.docker.rawValue, provisioner: RemoteWorkrooms.provisioner, id: UUID(), repository: "o/r", cloneURL: "https://github.com/o/r.git", path: "/home/workroom/r"), - runtime: .apple, driver: driver, environment: environment, + key: RemoteHosts.DriverKey(runtime: .apple), driver: driver, environment: environment, recorder: RemoteWorkrooms.Recorder( reserve: { _, _ in "x" }, record: { _, _ in XCTFail("recorded") }, forget: { _ in })) @@ -1175,6 +2103,33 @@ final class RemoteHostsTests: XCTestCase { "sign in to Codaset or run gh auth login") } + /// A boxd create's row shows the step it is on as a fraction of the steps it takes: all eight + /// when it builds the project's base first, the five of a derive when the base exists (#356). + func testABoxdCreateShowsItsStepOfTheStepsItTakes() { + XCTAssertEqual( + AppStore.createStep(.machine, buildsBase: true), + .init(fraction: 0, label: "Creating the base machine (step 1 of 8)")) + XCTAssertEqual(AppStore.createStep(.snapshot, buildsBase: true)?.fraction, 3.0 / 8) + XCTAssertEqual( + AppStore.createStep(.snapshot, buildsBase: false), + .init(fraction: 0, label: "Copying the base machine (step 1 of 5)")) + XCTAssertEqual(AppStore.createStep(.checkout, buildsBase: false)?.fraction, 4.0 / 5) + XCTAssertNil(AppStore.createStep(.clone, buildsBase: false), "a derive clones nothing") + } + + /// boxd's entry is off without its CLI, and without Codaset: a boxd workroom takes the broker's + /// tokens only, never the Mac's gh, so `gh auth login` is no way in (OQ20, #356). Whether boxd + /// itself is signed in is a CLI call, which a create makes rather than the menu. + func testTheBoxdEntrySaysWhyItIsOff() { + XCTAssertNil(RemoteWorkrooms.unavailability(ofBoxdInstalled: true, codasetSignedIn: true)) + XCTAssertEqual( + RemoteWorkrooms.unavailability(ofBoxdInstalled: false, codasetSignedIn: false), + "not installed") + XCTAssertEqual( + RemoteWorkrooms.unavailability(ofBoxdInstalled: true, codasetSignedIn: false), + "sign in to Codaset") + } + /// A driver takes on only a host of its own context: its commands would not reach another's. func testADriverRefusesAHostOfAnotherContext() throws { let (runtime, _) = try stubRuntime() @@ -1206,7 +2161,7 @@ final class RemoteHostsTests: XCTestCase { environments: Dictionary(uniqueKeysWithValues: keys.map { ($0, environment($0)) })) for key in keys { let host = HostDescriptor( - driver: key.runtime.rawValue, provisioner: RemoteWorkrooms.provisioner, id: UUID(), + driver: key.runtime?.rawValue, provisioner: RemoteWorkrooms.provisioner, id: UUID(), container: Self.record(context: key.context)) let driver = try XCTUnwrap(deletion.environment(for: host)?.driver as? ContainerHostDriver) XCTAssertEqual(driver.provisioning?.context, key.context) diff --git a/macapp/WorkroomAppTests/RemoteProvisioningTests.swift b/macapp/WorkroomAppTests/RemoteProvisioningTests.swift index 2fbb69074..d2279ff5e 100644 --- a/macapp/WorkroomAppTests/RemoteProvisioningTests.swift +++ b/macapp/WorkroomAppTests/RemoteProvisioningTests.swift @@ -189,6 +189,56 @@ final class RemoteProvisioningCredentialsTests: XCTestCase { } catch RemoteWorkrooms.Failure.signedOut {} XCTAssertEqual(driver.derives, 0, "the base was copied first") } + + // Value: protects=a boxd create's row says it is cloning, then enrolling, then checking out, per path taken; + // fails_when=buildBase or derive stops reporting its step, or a relayed workroom reports an enrol it skips; + // why_new=BoxdHostDriverTests stop at the driver's machine/setup steps; nothing reaches these three; seam=none + /// The steps a create's project row shows come from the sequence itself (#356): a base is + /// cloned, a broker workroom enrols and then checks out, and a relayed one only checks out. + func testTheSequenceReportsItsCloneEnrolAndCheckoutSteps() async throws { + BrokerStub.reset([]) + // A connection the sequence reaches and then loses, so the git it runs fails at once. + let connect: @Sendable (HostID) async throws -> AgentVCSConnection = { host in + let fake = try FakeAgent(version: 4, status: true) + defer { fake.stop() } + return try await AgentVCSConnection.connect(host: host, socketPath: fake.socketPath) + } + let steps = StepLog() + let report: @Sendable (RemoteProvisioning.Step) -> Void = { steps.add($0) } + let repository = repository + func environment(client: Bool) -> RemoteProvisioning.Environment { + RemoteProvisioning.Environment( + driver: MakingDriver(), agentSocket: RemoteWorkrooms.agentSocket, + client: client + ? BrokerClient( + baseURL: URL(string: "https://codaset.localhost")!, + key: .software(P256.Signing.PrivateKey()), session: BrokerStub.session) + : nil, + gitHubToken: { "gho_mac" }, connect: connect, revoke: { _ in }) + } + func base(relayed: Bool) -> RemoteProvisioning.Base { + .init(host: UUID(), repository: repository, cloneURL: "u", path: "/p", relayed: relayed) + } + + await RemoteProvisioning.$reportStep.withValue(report) { + _ = try? await RemoteProvisioning.buildBase( + repository: repository, cloneURL: "u", path: "/p", in: environment(client: false), + record: { _ in }) + } + XCTAssertEqual(steps.take(), [.clone]) + + await RemoteProvisioning.$reportStep.withValue(report) { + _ = try? await RemoteProvisioning.derive( + from: base(relayed: false), workroom: UUID(), branch: "b", in: environment(client: true)) + } + XCTAssertEqual(steps.take(), [.enrol]) + + await RemoteProvisioning.$reportStep.withValue(report) { + _ = try? await RemoteProvisioning.derive( + from: base(relayed: true), workroom: UUID(), branch: "b", in: environment(client: true)) + } + XCTAssertEqual(steps.take(), [.checkout]) + } } extension RemoteProvisioningCredentialsTests { @@ -241,3 +291,33 @@ private final class RefusingDriver: HostDriver, @unchecked Sendable { throw HostDriverError.notImplemented("exec") } } + +private final class StepLog: @unchecked Sendable { + private let lock = NSLock() + private var seen: [RemoteProvisioning.Step] = [] + func add(_ step: RemoteProvisioning.Step) { lock.withLock { seen.append(step) } } + func take() -> [RemoteProvisioning.Step] { + lock.withLock { + defer { seen = [] } + return seen + } + } +} + +/// Makes a host for every create and derive, and takes it down again. +private struct MakingDriver: HostDriver { + let traits = HostDriverTraits( + transport: .sshStdio, deriveSpeed: nil, deriveCarriesLiveProcesses: false, + durableDisk: false, maxLifetime: nil, keepAwakeHoldsCredential: false, + sleepsWhenIdle: false) + + func create() async throws -> HostID { .remote(UUID()) } + func deriveFromBase(_ base: HostID) async throws -> HostID { .remote(UUID()) } + func destroy(_ host: HostID) async throws {} + func openStream(to host: HostID) async throws -> HostStream { + throw HostDriverError.notImplemented("openStream") + } + func exec(_ command: String, on host: HostID) async throws -> HostStream { + throw HostDriverError.notImplemented("exec") + } +} diff --git a/macapp/WorkroomAppTests/RemoteWorkroomRoutingTests.swift b/macapp/WorkroomAppTests/RemoteWorkroomRoutingTests.swift index 44e6ba0d5..364785b87 100644 --- a/macapp/WorkroomAppTests/RemoteWorkroomRoutingTests.swift +++ b/macapp/WorkroomAppTests/RemoteWorkroomRoutingTests.swift @@ -148,6 +148,228 @@ final class RemoteWorkroomRoutingTests: XCTestCase { XCTAssertEqual(hosts, [remote.host, remote.host, remote.host]) } + // Value: protects=an asleep boxd box reads as asleep (moon, no alarm), not as a broken repository; + // fails_when=the resolver's .asleep catch goes or .asleep joins the weighted failures; + // why_new=RemoteHostsTests stops at the thrown error and nothing maps it to the badge; seam=none + /// A boxd box left asleep (#356) is not a failure: the sidebar says asleep and neither the dot nor + /// the project row's aggregate raises an alarm, where "unavailable" is a question mark that does. + func testAnAsleepBoxReadsAsAsleepNotUnavailable() async throws { + let router = RepositoryRouter(connectRemote: { throw RepositoryRoutingError.asleep($0) }) + let remote = try RepositoryLocation.remote(host: UUID(), path: path) + router.replaceRemote([try .init(location: remote, sharedLocation: remote)]) + + let status = await WorkroomStatusResolver().resolve(location: remote, router: router) + + XCTAssertEqual(status.failure, .asleep) + XCTAssertNil(status.dirty) + let dot = try XCTUnwrap(VCSStatusPresentation.dot(status)) + XCTAssertEqual(dot.accessibility, "asleep") + XCTAssertEqual(dot.semantic, .neutral) + XCTAssertEqual(status.aggregateWeight, 0, "an asleep box must not mark its project") + } + + // Value: protects=a box the app let go of reads as idle, not asleep, as boxd hasn't slept it yet; + // fails_when=the resolver folds .idle into .asleep or into the weighted failures; + // why_new=the idle state is new and only RemoteHostsTests see it thrown; seam=none + /// A box the app let go of as idle (#356) may still be awake: the dot says so, without an alarm. + func testABoxLetGoOfReadsAsIdleNotAsleep() async throws { + let router = RepositoryRouter(connectRemote: { throw RepositoryRoutingError.idle($0) }) + let remote = try RepositoryLocation.remote(host: UUID(), path: path) + router.replaceRemote([try .init(location: remote, sharedLocation: remote)]) + + let status = await WorkroomStatusResolver().resolve(location: remote, router: router) + + XCTAssertEqual(status.failure, .idle) + let dot = try XCTUnwrap(VCSStatusPresentation.dot(status)) + XCTAssertEqual(dot.accessibility, "idle, not connected") + XCTAssertEqual(dot.semantic, .neutral) + XCTAssertEqual(status.aggregateWeight, 0, "an idle box must not mark its project") + } + + // Value: protects=a deleted or other-account boxd machine raises an alarm on its row, never the quiet asleep moon; + // fails_when=the resolver folds .gone into its .asleep catch, or drops it from the failures that mark a project; + // why_new=RemoteHostsTests stop at the thrown .gone and nothing maps it to the sidebar; seam=none + /// A machine boxd says is gone (#356) is a failure the user must act on, not a box asleep. + func testAGoneBoxdMachineReadsAsAFailureNotAsleep() async throws { + let router = RepositoryRouter(connectRemote: { throw RepositoryRoutingError.gone($0) }) + let remote = try RepositoryLocation.remote(host: UUID(), path: path) + router.replaceRemote([try .init(location: remote, sharedLocation: remote)]) + + let status = await WorkroomStatusResolver().resolve(location: remote, router: router) + + XCTAssertNotEqual(status.failure, .asleep) + XCTAssertNotNil(status.failure) + XCTAssertNotEqual(VCSStatusPresentation.dot(status)?.accessibility, "asleep") + XCTAssertEqual(status.aggregateWeight, 1, "a gone machine must mark its project") + } + + // Value: protects=the workroom the user has selected never has its boxd box's connection dropped as idle; + // fails_when=AppStore stops telling RemoteHosts which host is selected, or never releases it on deselect; + // why_new=RemoteHostsTests call select() directly; nothing drives it from the app's selection; seam=none + /// Selecting a boxd workroom (#356) takes its box back from being let go of and keeps it, idle or + /// not, for as long as it is selected; leaving it makes the box eligible again. + @MainActor + func testSelectingABoxdWorkroomKeepsItsBoxFromBeingLetGoOf() async throws { + let id = UUID() + let box = Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: id)) + let projects = [Project(path: "/proj", vcs: "git", workrooms: [box])] + RemoteHosts.shared.adopt(projects, sweep: false) + let store = AppStore() + store.projects = projects + let host = HostID.remote(id) + // The store selects under its own window key, which its deinit clears. + + let letGo = await RemoteHosts.shared.observed(host, busy: false) + XCTAssertTrue(letGo, "the box was never let go of, so there is nothing to take back") + + store.selectedTargetID = .workroom(project: "/proj", name: "w") + XCTAssertFalse( + RemoteHosts.shared.isLetGo(host), "selecting its workroom left the box let go of") + let whileSelected = await RemoteHosts.shared.observed(host, busy: false) + XCTAssertFalse(whileSelected, "the selected workroom's box was let go of") + + store.selectedTargetID = .root(project: "/proj") + let afterLeaving = await RemoteHosts.shared.observed(host, busy: false) + XCTAssertTrue(afterLeaving, "leaving the workroom left its box held for good") + } + + /// Each window is its own AppStore: a second window selecting elsewhere leaves the first + /// window's box kept, and closing the first window lets it go (#356). + @MainActor + func testAnotherWindowsSelectionKeepsThisWindowsBoxAndClosingItLetsGo() async throws { + let id = UUID() + let box = Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, id: id)) + let projects = [Project(path: "/proj", vcs: "git", workrooms: [box])] + RemoteHosts.shared.adopt(projects, sweep: false) + let host = HostID.remote(id) + var first: AppStore? = AppStore() + first?.projects = projects + first?.selectedTargetID = .workroom(project: "/proj", name: "w") + let second = AppStore() + second.projects = projects + second.selectedTargetID = .root(project: "/proj") + + let kept = await RemoteHosts.shared.observed(host, busy: false) + XCTAssertFalse(kept, "another window's selection let go of this window's box") + + weak var closed = first + first = nil + XCTAssertNil(closed, "the first window's store outlived its window") + let letGo = await RemoteHosts.shared.observed(host, busy: false) + XCTAssertTrue(letGo, "a closed window kept its box held") + RemoteHosts.shared.activate(host) + _ = second + } + + /// A boxd host the app has adopted and let go of as idle, on the shared `RemoteHosts` the two + /// click call sites below use. + @MainActor + private func boxdHostLetGoOf() async -> HostID { + let id = UUID() + RemoteHosts.shared.adopt( + [ + Project( + path: "/proj", vcs: "git", + workrooms: [ + Workroom( + name: "w", path: "/home/boxd/r", vcsName: "workroom/w", warnings: [], + host: HostDescriptor( + driver: RemoteWorkrooms.boxdDriver, provisioner: RemoteWorkrooms.provisioner, + id: id)) + ]) + ], sweep: false) + let letGo = await RemoteHosts.shared.observed(.remote(id), busy: false) + XCTAssertTrue(letGo, "the box was never let go of, so there is nothing to wake") + return .remote(id) + } + + // Value: protects=Keep awake on a boxd box the app let go of as idle is a click that wakes it, not a + // refused background read; fails_when=the host model's transport drops wake: true from ensureConnected; + // why_new=RemoteHostsTests pass wake: true by hand and nothing drives this call site; seam=none + /// The connect it makes fails here (boxd wrote no ssh details for the host), but a click clears + /// the let-go mark before connecting, and a refused background read never reaches that. + @MainActor + func testKeepAwakeOnABoxLetGoOfIsAClickNotABackgroundRead() async throws { + let host = await boxdHostLetGoOf() + guard case .remote(let id) = host else { return XCTFail("not a remote host") } + defer { WakefulnessModel.forgetHost(id) } + + let model = WakefulnessModel.model(forHost: id) + model.keep() + await model.keepInFlight?.value + + XCTAssertFalse(RemoteHosts.shared.isLetGo(host), "Keep awake was refused as a background read") + } + + // Value: protects=closing a pane on a boxd box the app let go of wakes it to end the session, rather + // than leaving the session running; fails_when=endSession's remote kill drops wake: true; why_new= + // RemoteHostsTests pass wake: true by hand and nothing drives this call site; seam=none + /// As above: the kill fails here, so the session stays registered, which is the right outcome + /// for a host that can't be reached; the click still reached the box. + @MainActor + func testClosingAPaneOnABoxLetGoOfIsAClickNotABackgroundRead() async throws { + let host = await boxdHostLetGoOf() + let session = UUID() + let sessions = PersistentSessionService.shared + sessions.registerRemoteSession( + session, on: host, + via: BoxdHostDriver( + configuration: .init(cli: URL(fileURLWithPath: "/usr/bin/false")), + directory: FileManager.default.temporaryDirectory), + workingDirectory: "/home/boxd/r") + defer { sessions.forgetRemoteSession(session) } + + let ended = await sessions.endSession(sessionID: session) + + XCTAssertFalse(ended, "a session on an unreachable host was reported ended") + XCTAssertTrue(sessions.isRemote(session), "the failed kill dropped the session's registration") + XCTAssertFalse(RemoteHosts.shared.isLetGo(host), "closing the pane was refused as a read") + } + + // Value: protects=a pane attaching to a box the app let go of takes the box back, so its prompts + // are heard while the pane's ssh holds it; fails_when=attachCommand stops waking a let-go host; + // why_new=no test attaches a pane to a let-go host; seam=none + /// A pane attaching (a reconnect after a dropped link, a window reopened) is the user at the box: + /// its ssh holds the box awake, so the app takes it back rather than leave it unwatched. + @MainActor + func testAPaneAttachingToABoxLetGoOfTakesItBack() async throws { + let host = await boxdHostLetGoOf() + guard case .remote(let id) = host else { return XCTFail("not a remote host") } + let session = UUID() + let sessions = PersistentSessionService.shared + sessions.registerRemoteSession( + session, on: host, + via: ContainerHostDriver( + hosts: [ + id: .init( + address: "127.0.0.1", port: 1, user: "boxd", identityFile: "/keys/id", + hostKey: "ssh-ed25519 AAAA", agentSocket: "/s") + ], directory: FileManager.default.temporaryDirectory), + workingDirectory: "/home/boxd/r") + defer { sessions.forgetRemoteSession(session) } + + // An attach whose ssh is gone before the take-back runs holds nothing, so takes nothing back. + let pane = UUID() + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: pane)) + sessions.paneDetached(session, by: pane) + try await Task.sleep(for: .milliseconds(200)) + XCTAssertTrue(RemoteHosts.shared.isLetGo(host), "a pane gone before the take-back took it back") + + XCTAssertNotNil(sessions.attachCommand(forSession: session, by: pane)) + // The take-back runs in its own task. + for _ in 0..<100 where RemoteHosts.shared.isLetGo(host) { + try await Task.sleep(for: .milliseconds(20)) + } + XCTAssertFalse( + RemoteHosts.shared.isLetGo(host), "a pane attached to a box the app let go of left it so") + } + /// A target names its repository by host: a reachable remote one by its host's location, and one /// it can't reach by none, so a viewer never reads its path on this Mac. func testATargetResolvesItsRemoteLocationFromItsHost() throws { diff --git a/macapp/WorkroomAppTests/Session/AgentBootstrapTests.swift b/macapp/WorkroomAppTests/Session/AgentBootstrapTests.swift index e0ddfd14a..513ac4c09 100644 --- a/macapp/WorkroomAppTests/Session/AgentBootstrapTests.swift +++ b/macapp/WorkroomAppTests/Session/AgentBootstrapTests.swift @@ -446,6 +446,37 @@ final class AgentBootstrapTests: XCTestCase { XCTAssertTrue(request.contains(#""ask_at_ceiling":true"#), request) } + /// A connect nothing watches for prompts (provisioning's) hands the agent no settings at all + /// (#356): settings are the whole agent's, so its `ask: false` could land after a watched + /// connection's `ask: true` and switch prompting off under it. + func testAnUnwatchedConnectSendsNoSettings() async throws { + let fake = try FakeAgent(version: 4, status: true) + defer { fake.stop() } + let driver = StubDriver([ + probe(installed: digest, handOff: "0 current"), + probe(installed: digest, handOff: "0 current"), + ]) + driver.streamSocket = fake.socketPath + let unwatched = try await AgentBootstrap.connect( + host: host, driver: driver, socket: socket, agent: bundled(_:), handOff: true, + resources: nil, watched: false) + // A watched connect to the same agent, after it, shows the fake records the request a connect + // sends. The two connections are not ordered, so the count is read only after a settle of a + // second past that request, ~100x a send's measured latency, to catch an unwatched one too. + let watched = try await AgentBootstrap.connect( + host: host, driver: driver, socket: socket, agent: bundled(_:), handOff: true, + resources: nil) + let deadline = ContinuousClock.now + .seconds(5) + while fake.receivedStatusRequests.isEmpty && ContinuousClock.now < deadline { + try await Task.sleep(for: .milliseconds(20)) + } + try await Task.sleep(for: .seconds(1)) + await unwatched.close() + await watched.close() + XCTAssertEqual( + fake.receivedStatusRequests.count, 1, "an unwatched connect sent settings, or none landed") + } + /// A host its provider never sleeps (a container) is not handed ask-at-ceiling (#257): its prompt /// would say the box may sleep, and nothing ever will. The rest of the settings still go. func testAHostThatNeverSleepsIsNotAskedAboutSleeping() async throws { diff --git a/macapp/WorkroomAppTests/Session/PersistentSessionAttachGateTests.swift b/macapp/WorkroomAppTests/Session/PersistentSessionAttachGateTests.swift index c518070e4..ab13b69f3 100644 --- a/macapp/WorkroomAppTests/Session/PersistentSessionAttachGateTests.swift +++ b/macapp/WorkroomAppTests/Session/PersistentSessionAttachGateTests.swift @@ -103,6 +103,9 @@ final class PersistentSessionAttachGateTests: XCTestCase { XCTAssertTrue( body.contains("forSession: persistentSessionID"), "applyPersistentSession no longer routes per session") + // Without its token the pane's attach holds nothing, and its box can be let go under it (#356). + XCTAssertTrue( + body.contains("by: attachToken"), "applyPersistentSession no longer marks the pane attached") } /// The other call site this commit added, and the one with the same blind spot. diff --git a/macapp/WorkroomAppUITests/NewWorkroomPlacesUITests.swift b/macapp/WorkroomAppUITests/NewWorkroomPlacesUITests.swift index 35d706d6c..29e4b4c57 100644 --- a/macapp/WorkroomAppUITests/NewWorkroomPlacesUITests.swift +++ b/macapp/WorkroomAppUITests/NewWorkroomPlacesUITests.swift @@ -1,8 +1,11 @@ import XCTest /// New Workroom asks where once the remote preview is on (#309): This Mac, then Docker and Apple -/// Container under "Local containers", each disabled with its reason when it can't be used here. -/// Without the preview it stays one plain item. +/// Container under "Local containers" and boxd under "Remote" (#356), each disabled with its reason +/// when it can't be used here. Without the preview it stays one plain item. +// Value: protects=boxd is offered in New Workroom, disabled exactly when its title gives a reason; +// fails_when=WorkroomPlace.all or the sidebar's Remote section drops boxd; why_new=the places +// loops listed only Docker and Apple Container; seam=none final class NewWorkroomPlacesUITests: XCTestCase { override func setUp() { super.setUp() @@ -38,7 +41,7 @@ final class NewWorkroomPlacesUITests: XCTestCase { thisMac.hover() XCTAssertTrue(thisMac.isEnabled) // Each runtime is listed, usable or saying why not; which depends on this Mac. - for runtime in ["Docker", "Apple Container"] { + for runtime in ["Docker", "Apple Container", "boxd"] { let entry = app.menuItems.matching( NSPredicate(format: "title == %@ OR title BEGINSWITH %@", runtime, "\(runtime) — ") ).firstMatch @@ -68,7 +71,7 @@ final class NewWorkroomPlacesUITests: XCTestCase { add(idle) plus.click() XCTAssertTrue(app.menuItems["This Mac"].waitForExistence(timeout: 5), "+ didn't ask where") - for runtime in ["Docker", "Apple Container"] { + for runtime in ["Docker", "Apple Container", "boxd"] { XCTAssertTrue( app.menuItems.matching( NSPredicate(format: "title == %@ OR title BEGINSWITH %@", runtime, "\(runtime) — ") @@ -86,7 +89,7 @@ final class NewWorkroomPlacesUITests: XCTestCase { XCTAssertTrue(pickerThisMac.waitForExistence(timeout: 5), "the picker didn't ask where") XCTAssertFalse(app.textFields["newWorkroom.filter"].exists, "still listing projects") XCTAssertTrue(pickerThisMac.isEnabled) - for runtime in ["container", "apple-container"] { + for runtime in ["container", "apple-container", "boxd"] { XCTAssertTrue(app.buttons["newWorkroom.place.\(runtime)"].exists, "\(runtime) isn't offered") } XCTAssertFalse(app.staticTexts["newWorkroom.placesBlocked"].exists) @@ -126,7 +129,7 @@ final class NewWorkroomPlacesUITests: XCTestCase { let thisMac = app.menuItems["This Mac"] XCTAssertTrue(thisMac.waitForExistence(timeout: 5)) XCTAssertFalse(thisMac.isEnabled) - for runtime in ["Docker", "Apple Container"] { + for runtime in ["Docker", "Apple Container", "boxd"] { let entry = app.menuItems.matching( NSPredicate(format: "title == %@ OR title BEGINSWITH %@", runtime, "\(runtime) — ") ).firstMatch diff --git a/vcs/crates/wr-agent/src/wakefulness.rs b/vcs/crates/wr-agent/src/wakefulness.rs index eaa99bf00..6d73060e6 100644 --- a/vcs/crates/wr-agent/src/wakefulness.rs +++ b/vcs/crates/wr-agent/src/wakefulness.rs @@ -415,6 +415,8 @@ pub struct Classifier { wake_masked_last: bool, /// The last tick's keystroke age, so the ceiling can tell a user acting from a job running. since_input: f64, + /// The last tick's CPU and network signals were masked (see [`WAKE_GAP_S`]). + masked: bool, } impl Classifier { @@ -434,6 +436,7 @@ impl Classifier { resumed: false, wake_masked_last: false, since_input: f64::INFINITY, + masked: false, } } @@ -442,6 +445,12 @@ impl Classifier { self.resumed } + /// Whether the last tick's CPU and network signals were masked by a resume: its IDLE verdict + /// then says nothing about a job that uses only those, and the ceiling must not act on it. + pub fn wake_masked(&self) -> bool { + self.masked + } + /// Whether the user acted within the policy's grace as of the last tick: the keystroke signal, /// which `session.rs` feeds only for input its classifier calls the user's own. pub fn user_acted(&self) -> bool { @@ -503,6 +512,7 @@ impl Classifier { // mask still measures the last masked second; it is masked too. let masked = wake_masked || self.wake_masked_last; self.wake_masked_last = wake_masked; + self.masked = masked; // A masked tick moves the window's base to itself instead of feeding it: bytes that arrive // under a mask must not sit in the window and vote BUSY the tick the mask ends (the resume // blip is ~1.5 KB/s for three seconds; unmasking on the fourth with those bytes still inside @@ -679,6 +689,8 @@ pub struct Ceiling { pub settings: Settings, busy_since: Option, state: CeilingState, + /// The time of the last `step`, so a resume can carry the awake time from before the sleep. + last_t: Option, } impl Ceiling { @@ -687,12 +699,14 @@ impl Ceiling { settings, busy_since: None, state: CeilingState::Below, + last_t: None, } } /// Feeds the classifier's verdict. Returns true when a prompt should be raised NOW (once per /// crossing, not once per tick). pub fn step(&mut self, t: f64, verdict: Verdict) -> bool { + self.last_t = Some(t); if verdict == Verdict::Idle { // The box went idle on its own: the ceiling has nothing left to cap. self.busy_since = None; @@ -763,14 +777,35 @@ impl Ceiling { } } - /// The box resumed from a sleep. Whatever continuous awake period the ceiling was capping has - /// ended, so it starts over: a job that resumes with the box gets the full ceiling again, and - /// `awake_for` does not count the hours the box spent asleep. - pub fn resumed(&mut self) { - self.busy_since = None; + /// The box resumed from a sleep at `t`. `awake_for` never counts the hours the box spent + /// asleep. + /// + /// After an unanswered prompt (`Suppressed`), the awake time from before the sleep carries + /// over, so the box is still past its ceiling and the next BUSY tick asks again at once. A user + /// who woke the box gets a fresh prompt card; with nobody there, the box sleeps again once that + /// prompt times out, not a whole ceiling later. A wake is never an answer: a connect wakes the + /// box too (#356). + /// + /// Any other state starts over: a job that resumes with the box gets the full ceiling again. + pub fn resumed(&mut self, t: f64) { + let carried = match (self.state, self.busy_since, self.last_t) { + (CeilingState::Suppressed, Some(since), Some(last)) => Some(last - since), + _ => None, + }; + self.busy_since = carried.map(|awake| t - awake); self.state = CeilingState::Below; } + /// `step`, except that an IDLE verdict from a tick whose CPU and network were masked by a + /// resume (`Classifier::wake_masked`) is ignored: a job that uses only those reads IDLE for the + /// few seconds after every wake, and acting on it would end the awake time `resumed` carried. + pub fn step_unless_masked(&mut self, t: f64, verdict: Verdict, masked: bool) -> bool { + if masked && verdict == Verdict::Idle { + return false; + } + self.step(t, verdict) + } + pub fn state(&self) -> CeilingState { self.state } @@ -1181,6 +1216,26 @@ pub fn save_settings(path: &Path, settings: Settings) -> std::io::Result<()> { std::fs::rename(&tmp, path) } +/// One tick's ceiling steps after the classifier's, as the service loop runs them: a resume +/// carries the awake time across the sleep, a "keep" is applied, then the step ignores the IDLE +/// ticks the resume mask produces (#356). Returns whether a prompt was raised. +#[cfg(any(target_os = "linux", test))] +fn ceiling_step( + classifier: &Classifier, + ceiling: &mut Ceiling, + t: f64, + raw: Verdict, + keep: bool, +) -> bool { + if classifier.resumed() { + ceiling.resumed(t); + } + if keep { + ceiling.keep(t); + } + ceiling.step_unless_masked(t, raw, classifier.wake_masked()) +} + #[cfg(target_os = "linux")] pub use service::spawn; @@ -1278,13 +1333,9 @@ mod service { ceiling.set_settings(s.t, settings); } if classifier.resumed() { - ceiling.resumed(); keep_awake.resumed(); } - if keep { - ceiling.keep(s.t); - } - let prompted = ceiling.step(s.t, raw); + let prompted = super::ceiling_step(&classifier, &mut ceiling, s.t, raw, keep); // After the step, so a keystroke inside the grace on the very tick the prompt is // raised or expires answers it at once rather than a tick later. if classifier.user_acted() { diff --git a/vcs/crates/wr-agent/src/wakefulness/tests.rs b/vcs/crates/wr-agent/src/wakefulness/tests.rs index a161d3e7d..ee9ca2d6f 100644 --- a/vcs/crates/wr-agent/src/wakefulness/tests.rs +++ b/vcs/crates/wr-agent/src/wakefulness/tests.rs @@ -246,12 +246,14 @@ fn keep_resets_the_ceiling_and_going_idle_clears_it() { assert_eq!(c.state(), CeilingState::Below, "idle clears the ceiling"); } -/// The two exits from `Suppressed` a real user gets without the prompt card: typing, and the box -/// coming back from the sleep the suppression allowed. Neither existed at first, so a user who -/// woke the box and typed kept the classifier BUSY while the service kept publishing IDLE, and the -/// provider hibernated the box under them, again after every resume. +/// The exits from `Suppressed` a real user gets without the prompt card: typing answers it, and +/// the box coming back from the sleep the suppression allowed asks again. Neither existed at +/// first, so a user who woke the box and typed kept the classifier BUSY while the service kept +/// publishing IDLE, and the provider hibernated the box under them, again after every resume. +/// A resume once cleared the ceiling outright; but a connect wakes the box too, so every status +/// probe then bought a running job another full ceiling (#356). #[test] -fn typing_or_resuming_clears_a_suppressed_ceiling() { +fn typing_clears_a_suppressed_ceiling_and_resuming_asks_again() { let settings = Settings { ask: true, ..Settings::default() @@ -286,18 +288,36 @@ fn typing_or_resuming_clears_a_suppressed_ceiling() { "well inside the new ceiling" ); - // So is the box resuming: the continuous awake period the ceiling capped has ended. + // The box resuming is not an answer: it keeps the awake time from before the sleep, so the + // next BUSY tick asks again, with a fresh deadline. let (mut c, deadline) = suppressed(); - c.resumed(); + let before = c.awake_for(deadline); + let woke = deadline + 3600.0; + c.resumed(woke); assert_eq!(c.state(), CeilingState::Below); + assert_eq!(c.awake_for(woke), before, "sleep is not awake time"); + assert!( + c.step(woke + 1.0, Verdict::Busy), + "past the ceiling: ask again at once" + ); assert_eq!( - c.awake_for(deadline + 3600.0), - 0.0, - "sleep is not awake time" + c.state(), + CeilingState::Prompted { + deadline: woke + 1.0 + 600.0 + } + ); + assert!( + !c.suppressing(), + "a fresh prompt keeps the box awake until its deadline" ); - c.step(deadline + 3600.0, Verdict::Busy); - assert_eq!(c.awake_for(deadline + 3600.0), 0.0); - assert!(!c.exceeded()); + + // A resume from any other state starts the ceiling over: a job that resumes with the box gets + // the full ceiling again. + let mut c = Ceiling::new(settings); + busy_until(&mut c, 3.0 * 3600.0); + c.resumed(5.0 * 3600.0); + assert_eq!(c.awake_for(5.0 * 3600.0), 0.0); + assert!(!c.step(5.0 * 3600.0 + 60.0, Verdict::Busy)); // Typing while the prompt is still pending answers it too: the user is there and working. let mut c = Ceiling::new(settings); @@ -476,6 +496,128 @@ fn the_resume_masks_cpu_for_the_tick_after_it_too() { assert_eq!(c.verdict(), Verdict::Busy); } +/// One service tick: the classifier's step, then the service loop's own ceiling steps +/// (`ceiling_step`). Without `honour_mask`, a control that steps the ceiling on every tick, masked +/// or not. Returns whether a prompt was raised, and the verdict the service publishes. +fn ceiling_tick( + c: &mut Classifier, + ceiling: &mut Ceiling, + s: &Sample, + honour_mask: bool, +) -> (bool, Verdict) { + c.step(s, false); + let raw = c.verdict(); + let prompted = if honour_mask { + ceiling_step(c, ceiling, s.t, raw, false) + } else { + if c.resumed() { + ceiling.resumed(s.t); + } + ceiling.step(s.t, raw) + }; + (prompted, ceiling.published(raw)) +} + +/// A CPU-only job outlasts its ceiling, nobody answers, the box sleeps, and a connect wakes it with +/// the job still running. The wake must ask again, not buy the job another ceiling. The resume mask +/// reads such a job IDLE for its first few ticks, and an IDLE step ends the ceiling, so those ticks +/// have to be ignored or the carried awake time is lost before the job is seen again. +fn wake_with_a_cpu_job_running(honour_mask: bool) -> Option { + let settings = Settings { + ceiling: 30.0, + prompt_timeout: 10.0, + ask: true, + }; + let mut c = Classifier::new(Policy::production(), Boundary::agent(999)).with_wake_mask(); + let mut ceiling = Ceiling::new(settings); + let mut s = quiet(1000.0, 0); + // One core of CPU a second. + let tick = |s: &mut Sample, t: f64| { + s.t = t; + s.procs[1].ticks += 100; + }; + let mut t = 1000.0; + for _ in 0..60 { + tick(&mut s, t); + ceiling_tick(&mut c, &mut ceiling, &s, honour_mask); + t += 1.0; + } + assert!( + ceiling.suppressing(), + "unanswered: the service let the box sleep" + ); + // Asleep for 300 s; the job's CPU stops with the box. + t += 300.0; + for _ in 0..20 { + tick(&mut s, t); + let (prompted, published) = ceiling_tick(&mut c, &mut ceiling, &s, honour_mask); + if prompted { + assert!(matches!(ceiling.state(), CeilingState::Prompted { .. })); + assert_eq!( + published, + Verdict::Busy, + "a pending prompt keeps the box awake" + ); + return Some(t); + } + t += 1.0; + } + None +} + +#[test] +fn a_wake_with_the_job_still_running_asks_again() { + let asked = wake_with_a_cpu_job_running(true).expect("the wake must ask again"); + assert!( + asked - 1360.0 < 10.0, + "asked within the resume mask's few seconds, at {asked}" + ); + // The other half of the claim: acting on the masked IDLE ticks loses the carried awake time, + // so the box is not asked and the job gets a whole new ceiling. + assert_eq!(wake_with_a_cpu_job_running(false), None); +} + +/// A Keep awake that lands on the tick the box resumes is the user's answer to the carried ceiling, +/// so it restarts the awake time and nothing is asked; without it, that tick asks at once. +#[test] +fn a_keep_on_the_resume_tick_restarts_the_ceiling() { + let settings = Settings { + ask: true, + ..Settings::default() + }; + let resumed_and_suppressed = || { + let mut ceiling = Ceiling::new(settings); + busy_until(&mut ceiling, 4.0 * 3600.0); + let CeilingState::Prompted { deadline } = ceiling.state() else { + panic!("expected a prompt") + }; + ceiling.step(deadline, Verdict::Busy); + assert!(ceiling.suppressing()); + let mut c = Classifier::new(Policy::production(), Boundary::agent(999)).with_wake_mask(); + let mut t = deadline + 1.0; + c.step(&quiet(t, 10_000), false); + t += 300.0; + c.step(&quiet(t, 10_000), false); + assert!(c.resumed()); + (c, ceiling, t) + }; + + let (c, mut ceiling, t) = resumed_and_suppressed(); + assert!(!ceiling_step(&c, &mut ceiling, t, Verdict::Busy, true)); + assert_eq!(ceiling.state(), CeilingState::Below); + assert_eq!( + ceiling.awake_for(t), + 0.0, + "the ceiling restarts from the keep" + ); + + let (c, mut ceiling, t) = resumed_and_suppressed(); + assert!( + ceiling_step(&c, &mut ceiling, t, Verdict::Busy, false), + "without the keep, the carried time asks at once" + ); +} + /// What the service reads off the classifier to drive the ceiling: the resume flag is up for the /// first tick after the gap only, and the keystroke signal reflects the last tick's grace. #[test]