Repository navigation
[CUT15-01][OPS15-01][VM15-03] S15 installer, atomic upgrade fence and a separate QEMU identity - #379
Conversation
…OS-released lock Fence acquisition's check-and-link and clearUpgradeFence(ownerId)'s read-compare-unlink now run inside a SQLite BEGIN IMMEDIATE transaction with a write probe on the always-empty <db>.upgrade-fence.lock (Schway's credential-store lock pattern). The OS releases the lock when its holder dies, so a stale clearer can no longer remove a newer fence and a killed holder never blocks the next acquisition. Reader semantics are unchanged. Closes #375
…entity QemuHostOptions gains a vmm seam (VM15-03). With it, every hypervisor-side command (the transient QEMU unit, unit show/stop/kill and qemu-img on the VM disk) runs as the QEMU identity through the fixed prefix sudo -n -u <vmm> -- <launcher>. The root-owned launcher, rendered from qemuArgv itself, validates every argument and rebuilds the restricted QEMU argv, so /proc re-verification keeps matching byte for byte. The overlay, seed and QMP socket live in the QEMU identity's machine directory; base images and the registry-proxy socket (0660) live in a controller-owned, group-readable control directory; the SSH client key, known_hosts and VM state stay controller-private. /dev/kvm readiness is probed as the QEMU identity, and a QEMU process owned by any other uid is never treated as the VM. Without the seam behavior is unchanged. Approved subscription routes now follow the service identity's local credential readiness, so the owner's Web device login makes the Codex route usable without editing configuration. A root-owned S15 runtime config readable only by the service group is accepted.
…ble install and upgrade install.sh --s15 (mutually exclusive with --s14/--legacy; S14 stays the default) installs jit-s15 beside any legacy or S14 installation without reading or changing them. It installs the nested working VM's packages, creates the jit-s15 controller and jit-s15-vmm QEMU identities and the shared jit-s15-vm group (jit-s15 leaves kvm), lingering user managers, the argv-fixed VM launcher with a visudo-checked sudoers rule, a frozen release, a generated credential-free s15-runtime/v1 config and fresh state through init --s15, then a hardened jit-s15.service. Upgrades stop the controller (the working VM keeps running), switch the release link atomically and restore the previous release and unit when the new one fails to start or answer health. A failed fresh install removes the new unit and the state it initialized. The installer then follows the automatic working computer's durable progress and prints the private first-owner link and the owner's Web steps. Refs #341
…ide the controller unit
A live check on an Ubuntu 26.04 KVM host with disposable identities
showed two gaps in the separate QEMU identity layout:
- AppArmor's nc.openbsd profile refuses the guest-forward connect to a
socket bound inside the controller unit's private mount namespace
("disconnected path"). The installer now ships
jit-s15-vm-proxy.socket, which systemd binds in the root namespace
and passes to serve; the registry proxy serves the inherited fd.
- RestrictSUIDSGID forbids setting setgid bits from the controller and
its sudo children, so control/images is installer-made (setgid 2750)
and the launcher's mkdir relies on the setgid parent.
Also documents the install, upgrade, rollback and uninstall procedure
(docs/runbooks/s15-install.md), the separate QEMU identity in ADR 0067,
the working-VM contract (revision 6) and runbook, and corrects the
vm_kvm_denied next step for the separate identity.
Refs #341
[High impact] The PR reaches S15 installation, its working-computer boundary, and shared upgrade and restore fencing. Fix the expired first-owner link and the blocked controller fallback before merging. Findings
Fix with agent prompt### Issue 1
ops/package/installer/s15.ts:551-552
`owner-link-prepare` issues a link valid for 15 minutes, but `working-computer` can then wait 90 minutes before `pendingOwner.deliver()`. If image download or guest setup takes longer than 15 minutes, the owner receives a link rejected as `owner_setup_link_invalid` and cannot set their password. Deliver the link after health and before waiting for the computer, or issue a fresh link immediately before delivery.
### Issue 2
ops/package/installer/s15.ts:371
If the host has no `kvm` group, this check aborts installation before the controller or Web enrollment page exists. The `dnf` path deliberately skips QEMU packages and promises owner-supplied VM enrollment instead, so it can reach this failure on a host without local VM support. Allow the controller installation to continue without KVM and report the VM fallback instead.
### Issue 3
ops/package/installer/s15.ts:330-331
The result of `call()` is discarded here. If `systemctl --user stop` fails or times out, uninstall still removes the controller units and reports success while a working VM may remain running. Check the result and confirm no matching VM units remain active before reporting that uninstall finished.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryAdds a separate S15 installation, moves QEMU commands to a dedicated user, and serializes upgrade-fence changes with a SQLite lock.
|
| step("owner-link", "re-run install.sh --s15 from the operator terminal; an existing owner is never reset"); | ||
| if (!dry) pendingOwner!.deliver(); |
There was a problem hiding this comment.
First-owner link expires before delivery
owner-link-prepare issues a link valid for 15 minutes, but working-computer can then wait 90 minutes before pendingOwner.deliver(). If image download or guest setup takes longer than 15 minutes, the owner receives a link rejected as owner_setup_link_invalid and cannot set their password. Deliver the link after health and before waiting for the computer, or issue a fresh link immediately before delivery.
Prompt To Fix With AI
This is a comment left during a code review.
Path: ops/package/installer/s15.ts
Line: 551-552
Comment:
**First-owner link expires before delivery**
`owner-link-prepare` issues a link valid for 15 minutes, but `working-computer` can then wait 90 minutes before `pendingOwner.deliver()`. If image download or guest setup takes longer than 15 minutes, the owner receives a link rejected as `owner_setup_link_invalid` and cannot set their password. Deliver the link after health and before waiting for the computer, or issue a fresh link immediately before delivery.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| const ctl = await passwd(p.user), vmm = await passwd(p.vmmUser), kvm = await group("kvm"); | ||
| if (!ctl || ctl.name !== p.user || !valid(ctl.uid) || !valid(ctl.gid) || ctl.home !== p.home || !/\/(nologin|false)$/.test(ctl.shell)) throw new InstallerFailure(currentStep, next); | ||
| if (!vmm || vmm.name !== p.vmmUser || !valid(vmm.uid) || vmm.gid !== shared.gid || vmm.home !== p.vmmHome || !/\/(nologin|false)$/.test(vmm.shell) || vmm.uid === ctl.uid) throw new InstallerFailure(currentStep, next); | ||
| if (!kvm || !valid(kvm.gid)) throw new InstallerFailure(currentStep, "install the QEMU/KVM packages so the kvm group exists (qemu-system-x86 or qemu-system-arm), then re-run install.sh --s15"); |
There was a problem hiding this comment.
Missing KVM group blocks installation
If the host has no kvm group, this check aborts installation before the controller or Web enrollment page exists. The dnf path deliberately skips QEMU packages and promises owner-supplied VM enrollment instead, so it can reach this failure on a host without local VM support. Allow the controller installation to continue without KVM and report the VM fallback instead.
Prompt To Fix With AI
This is a comment left during a code review.
Path: ops/package/installer/s15.ts
Line: 371
Comment:
**Missing KVM group blocks installation**
If the host has no `kvm` group, this check aborts installation before the controller or Web enrollment page exists. The `dnf` path deliberately skips QEMU packages and promises owner-supplied VM enrollment instead, so it can reach this failure on a host without local VM support. Allow the controller installation to continue without KVM and report the VM fallback instead.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| if (vmm.exitCode === 0 && Number.isSafeInteger(vmmId) && vmmId > 0) | ||
| await call(["runuser", "-u", p.vmmUser, "--", "env", `XDG_RUNTIME_DIR=/run/user/${vmmId}`, "systemctl", "--user", "stop", "jit-work-*.service"], 120_000); |
There was a problem hiding this comment.
Uninstall hides VM stop failures
The result of call() is discarded here. If systemctl --user stop fails or times out, uninstall still removes the controller units and reports success while a working VM may remain running. Check the result and confirm no matching VM units remain active before reporting that uninstall finished.
Prompt To Fix With AI
This is a comment left during a code review.
Path: ops/package/installer/s15.ts
Line: 330-331
Comment:
**Uninstall hides VM stop failures**
The result of `call()` is discarded here. If `systemctl --user stop` fails or times out, uninstall still removes the controller units and reports success while a working VM may remain running. Check the result and confirm no matching VM units remain active before reporting that uninstall finished.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.…without KVM and prove the VM stop on uninstall - The 15-minute first-owner link is now shown right after the health check, before the working computer's setup (which can take far longer), so a slow image download or first boot never hands the owner an expired link. - A host without a kvm group no longer aborts: the controller is installed alone (no QEMU identity, launcher, sudoers rule or proxy socket; the unit keeps NoNewPrivileges) with vm.local false, the installer reports the owner-supplied VM fallback, and a later re-run adds the Jit-managed VM in place once QEMU/KVM is installed. - Uninstall checks the working-VM stop and reads it back (user manager list-units and the kernel's process table) before removing any unit; a VM that is still running fails with the exact stop command. Refs #341
|
Fixed the three review findings in
The runbook ( |
Refs #341
Closes #375
Closes #378
Tested head:
55818f87a4390daa5f107bd34f33328719348f21(baseorigin/main2bc99757).What this delivers
CUT15-01 (#341):
install.sh --s15(installer code only)--s15picks S15 as its own installation,jit-s15. It cannot be combined with--s14,--legacyor--computer-slots. S14 is still the default. Legacy, S14 and omitted-runtime configs are never read or reinterpreted, and the other installers never touch S15. Steps (ops/package/installer/s15.ts):/etc/jit-s15/s15-runtime.jsonthat is not ans15-runtime/v1config for/var/lib/jit-s15/state(for example a legacy or S14 service config) is refused before any change.install.sh, Debian/Ubuntu):sudo openssh-client qemu-system-x86 | qemu-system-arm qemu-efi-aarch64, qemu-utils cloud-image-utils netcat-openbsd, plus the core packages. These match whatlocalVmToolsexpects. On other distributions the controller still installs, and the Web offers an owner-supplied VM.jit-s15(controller) andjit-s15-vmm(QEMU, inkvm, primary groupjit-s15-vm) and addsjit-s15tojit-s15-vm.jit-s15is removed fromkvm, andjit-s15-vmmis never injit-s15's group.jit-s15's Secret Service (needed for the Codex credential store). Installs the root-owned launcher/opt/jit-s15/libexec/jit-s15-vmmand/etc/sudoers.d/jit-s15-vmm(jit-s15 ALL=(jit-s15-vmm) NOPASSWD: <launcher>). The rule is checked withvisudo -cunder a dotted name that sudo ignores before it goes in place, and then sudo is proven end to end (launcher check)./opt/jit-s15/releases/<artifact id>.vm.vmm.jit-s15-vmm's user manager.init --s15runs only for a missing state directory. An interrupted init (noruntime/composition.json) is moved aside, never deleted.jit-s15.service, thejit-s15-vm-proxy.socketunit and thejit-s15journald namespace. Switches/opt/jit-s15/currentatomically, then checks/s14/loginon the local listener with the origin's Host header.main.ts s15-computer, which never composes the runtime) and prints each step in English and Thai from the existing i18n copy./dev/tty, like S14, then prints the owner steps: password; Codex device login at/settings/provider; Telegram token and binding at/resources/telegram; rules in the Web UI.The installer never sets the owner password.
setup --s15needs that password, so the installer usesinit --s15plus the existingserveboot pass, which creates the working computer automatically, and it follows that job's progress.Codex route without editing config.
availabilitywas a static field that blocked the route. Approved subscription routes now follow the service identity's local Schway credential readiness (checkAuthReadiness, no network, re-read every 15 s):configured-unverified→ available,quota-exceeded→ quota, anything else → unavailable. A route configuredunavailable/quotastays held, and metered routes are unchanged.runtimeDisclosure.modelRouteignores availability, which is live state rather than disclosure identity. Dispatch still lists the account catalog for every step.Browser:
browser: null. The protected browser needs owner-approved public origins and a disclosure route, and Playwright's Chromium download has no pinned hash Jit can verify, so this PR does not provision it. The runbook says so.Generated
/etc/jit-s15/s15-runtime.json(root:jit-s15 0640;readS15RuntimeConfignow also accepts a root-owned file that is only group-readable):{ "contract": "s15-runtime/v1", "installationId": "s15_<16 hex>", "directory": "/var/lib/jit-s15/state", "origin": "http://127.0.0.1:3002", "policy": null, "resourceBots": [], "browser": null, "github": null, "google": null, "catalog": ["bot jit", "resource computer", "route codex_route", "model codex_model"], "conversation": { "route": { "provider": "codex", "model": "gpt-6.1-sol" }, "...": "staging limits" }, "models": [{ "provider": "codex", "modelId": "gpt-6.1-sol", "charge": "subscription", "availability": "available", "efforts": ["high"], "...": "staging bounds" }], "disclosure": { "sources": [], "approvals": [], "githubRecipient": null, "googleRecipient": null, "browser": [], "acquisitions": [], "registries": [] }, "team": { "maxConcurrentOperations": 2, "memory": "off" }, "vm": { "id": "working", "limaInstance": null, "local": true, "vmm": { "user": "jit-s15-vmm", "uid": "<uid>", "directory": "/var/lib/jit-s15-vm", "launcher": "/opt/jit-s15/libexec/jit-s15-vmm" } } }(The real file is fully expanded;
catalogand"..."are abbreviated here.)VM15-03 (#378): QEMU under a separate unprivileged identity
QemuHostOptions.vmm(S15RuntimeConfig.vm.vmm, only valid withlocal: true). With it set, every hypervisor-side command runs through/usr/bin/sudo -n -u <vmm> -- <launcher> <op> …:show/stop/kill,start(systemd-run in the vmm user manager,UMask=0007),info/resize/create/snapshot,seedover stdin, andremove.ssh,ssh-keygen,ssh-keyscan,cloud-localdsand QMP stay with the controller. Without the seam, behavior and argv are unchanged, and the existing qemu/local/provisioning tests pass untouched apart from one fixture lookup.qemuArgvitself, so the controller's/procbyte-for-byte re-verification still holds. It validates every argument, refuses anything else with exit 64, and reads no environment input.vm-local/machines/<vm>(state, client key, known_hosts, pinned host key);jit-s15-vmm-ownedmachines/<vm>(disk, seed, QMP socket); controller-owned setgidcontrol/andcontrol/images/(base images 0440). A QEMU process counts as the VM only when its/procuid is the vmm uid./dev/kvmreadiness is probed as the vmm identity.nc.openbsdprofile refuses the guest-forward connect to a socket bound inside any controller mount namespace ("Failed name lookup - disconnected path"; evenPrivateTmp=truealone triggers it).jit-s15-vm-proxy.socket(ownerjit-s15, groupjit-s15-vm, 0660,PartOf=jit-s15.service) binds it in the root namespace, andservetakes it asjit-s15-vm-proxy(takeInheritedSocket→startVmRegistryProxy({ socketFd })). Note for [Frontier] [EVID15-01] Prove the protected boundaries and the GitHub scenario on the tested build #340: the existing evidence controller runs withPrivateTmp=trueand binds its proxy itself, so its guest-forward would hit the same denial if exercised.RestrictSUIDSGIDblocks setting setgid bits from the controller and from its sudo children. The installer therefore creates the setgid directories, and the launcher'smkdirrelies on the setgid parent.s15-working-vm-v1revision 6, the working-VM runbook. The residual is now: a QEMU escape lands asjit-s15-vmm, which can reach that identity's VMs, the base images, the proxy socket,/dev/kvmand the network as an ordinary user, and can forge vmm-side observations. It cannot read the controller's home, vault, state, keys or config, and it has no sudo rule.en/th):vm_kvm_deniedno longer tells the owner to add Jit's service user tokvm. It now says to re-runinstall.sh --s15, which repairs that identity's access.OPS15-01 (#375): atomic owner-scoped fence release
writeUpgradeFence's check-and-link andclearUpgradeFence(ownerId)'s read-compare-unlink now both run underwithUpgradeFenceLock. The lock is the Schway pattern: SQLiteBEGIN IMMEDIATEplus a write probe on the always-empty<db>.upgrade-fence.lock, one held connection, no non-SQLite descriptor (it is created via a temp file and hard link), an inode re-check, and a 5 s busy timeout ending inupgrade_in_progress; an unusable path givesupgrade_storage_unavailable. Readers are unchanged. The residual comment and the contract residual are replaced.Failure / rollback matrix
currentlink and new state removed--s15 --uninstallRunbook:
docs/runbooks/s15-install.md(install, layout, generated config, owner steps, upgrade, manual rollback, failure table, uninstall, security notes).Tests run locally (this Mac; full suite not run, per the owner instruction)
bun install --frozen-lockfile,bun run typecheck,git diff --check,shellcheck install.sh: pass.s15-installer(new: fresh-install shape, idempotent re-run, upgrade, health rollback, failure injection at every fresh and upgrade step followed by a clean re-run, working-computer outcomes, byte-identical legacy and S14 trees through S15 install/upgrade/rollback/uninstall and the reverse, foreign-config refusal, uninstall retention,install.sh --s15dry run, flag conflicts, re-exec, internal apply validation, generated config → realinit --s15);s15-vm-vmm(new: the real rendered launcher executed with recorder tools for every operation and refusal, start argv byte-equal toqemuArgvon both architectures, the sudo prefix on every hypervisor command and never on ssh/keygen/keyscan/cloud-localds, file modes, wrong-uid QEMU refused, snapshots/stop/restore/re-attach, layout refusals, inherited-socket parsing and a real inherited-fd proxy);ops-upgrade-fence(new: child-process barrier race, which fails on the old read-then-unlink; SIGKILLed holder; reader semantics; unusable lock path);s15-route-availability(new); plus the existings15-vm-*,s15-vm,s15-runtime,s15-runtime-bindings,s15-composition,s15-composition-pool,s15-web-setup,installer,s14-service-installer,runtime-selection,ops-restore,ops-upgradeandops-service-package.Live dry exercise on
jit-vm(disposable identities, removed afterwards)These used new disposable names
jdry15/jdry15-vmm/jdry15-vm. Nojit-s15identity was created, the production install was not done, andjit-s15-evid, its processes and its files were not touched. Units rendered by this code (names substituted) ran with the production unit's hardening. Results: sudo-rs 0.2.13 runs the launcher underProtectSystem=strict/ProtectHome=tmpfs/RestrictSUIDSGID; an arbitrarysudo -u vmm /bin/shis refused; the controller has no/dev/kvm; the host probe reads ready through the launcher; a provision run (blank 1 GiB base image) covered the group image dir, launchercreate, seed handover and start. QEMU ran as the vmm uid inuser@<vmm>.service/app.slicewithnetwork: proxy-only(exact argv), the QMP socket was 0770 vmm:shared and reachable by the controller, and live QMP snapshot, stop proof, offline snapshot, restore and retire all worked. The confinedncas the vmm identity got the proxy's407through the socket-activated fd. As vmm, reading the controller's home, client key and state, and writingcontrol//images/, were all denied; the controller could not writemachines/.visudo -c -fwith sudo-rs rejects an invalid rule (exit 1). Everything was removed afterwards: identities, groups, units, files, sudoers and linger.Not run / limits
install.sh --s15onjit-vm, the working computer booting a real Ubuntu guest underjit-s15-vmm, owner acceptance and the live vmm permission probe for [Frontier] [CUT15-01] Install S15 fresh on jit-vm with owner acceptance and a recoverable install #341/[Frontier] [VM15-03] Run the nested working VM's QEMU under a separate unprivileged identity #378 are orchestrator/owner steps after merge.jit-s15.servicekeeps privilege transitions available (noNoNewPrivileges), because the controller reaches the QEMU identity through sudo. The rule allows exactly one command.127.0.0.1:3001, the same as a default S14 installation on the same host.