coop selects its VM backend at compile time. macOS builds use Lima. Linux builds use Firecracker. The binary determines the backend; there is no runtime override.
Both backends expose the same CLI commands and produce the same guest environment: Ubuntu with Docker, GitHub CLI, Claude Code, and Codex pre-installed. The backends differ in how they create and manage the VM underneath.
The Lima backend runs VMs through Lima, which wraps Apple's Virtualization.framework. Lima manages disk images, networking, and SSH port forwarding, so coop delegates most operations to limactl.
Install Lima before running coop setup:
brew install lima
Coop resolves limactl through the shared host-tool validator and uses a fixed helper PATH: /opt/homebrew/bin:/usr/local/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin. Setup verifies that limactl --version is reachable there; otherwise it fails with an install hint. The macOS Nix package instead embeds its exact Lima and helper store paths at build time and trusts the immutable Nix runtime closure. Neither route selects Lima from the caller's PATH.
coop setup builds a golden disk image that all instances clone from:
- Generates an ed25519 SSH key pair, stored in the coop data directory.
- Creates a temporary builder VM from an Ubuntu 24.04 cloud image. The Lima YAML template includes a cloud-init provision script.
- The provision script installs all packages (Docker, GitHub CLI, Claude Code, Codex, Grok Build, and any profile packages), creates the
ubuntuuser with SSH access, and enables services. - After provisioning completes, cleans cloud-init state so it re-runs on cloned instances.
- Stops the builder VM and extracts its disk as the golden image.
- Generates a fast-start Lima template that references the golden image directly. No cloud-init provisioning runs on instance start.
The builder VM is deleted after extraction, whether the build succeeds or fails.
Each instance is a Lima VM created with limactl start using the fast-start template. Lima names are prefixed with coop- (e.g., coop-my-instance). The instance gets its own disk, a copy-on-write layer over the golden image. Lima handles SSH port allocation automatically; coop reads the assigned port from limactl list --json.
The Lima template configures:
vmType: "vz"(Virtualization.framework, not QEMU)- Rosetta enabled for x86_64 binary translation on Apple Silicon
mountType: "virtiofs"with no host mounts (emptymounts: []). Whencoop up --mountis used, Lima adds virtiofs mount entries for the specified host directories, providing live mounts where changes are visible immediately on both sides. Host and guest paths are serialized as YAML scalars so special characters remain part of the path.- Lima's built-in containerd disabled (Docker is installed in the guest instead)
Resizing a stopped instance's disk truncates the Lima disk to the new size and updates the instance lima.yaml disk: field to match, so the next start sees the grown size. Cloud-init's growpart module expands the partition and filesystem on next boot. Shrinking is not supported.
Memory and vCPU changes rewrite the cpus/memory fields in the instance's lima.yaml, which Lima re-reads on limactl start. The edit is written atomically, then coop starts the instance to validate and apply the new spec — if limactl rejects it (e.g. a spec larger than the host), the previous lima.yaml is restored. Without --start the instance is stopped again after the validating boot. The lima.yaml is authoritative: the global [vm] cpus/memory settings only seed new instances.
Lima runs as the current user. No sudo is required for any Lima operation: setup, start, stop, or destroy.
If a coop command fails with kex_exchange_identification: read: Connection reset by peer, or coop start times out waiting for SSH, inspect the Lima messages too. A repeated tcpproxy: error dialing "<guest-ip>:22": ... no route to host and/or an unavailable vsock guest agent can occur while limactl list still says Running. That state describes the VZ process, not a completed guest boot or a working SSH server. The address in the message is diagnostic; do not assume it stays the same.
Try the less invasive recovery steps first:
coop stop <instance>
coop start <instance>Check the host's VPN and network state. If several VZ guests or host networking are affected, reboot macOS and retry. Do not start recovery by running coop destroy: it deletes the instance disk. If the same SSH and guest-agent errors persist, they may be symptoms of a guest boot or filesystem failure rather than a host network problem. coop logs <instance> and Lima's logs under the instance directory may show where boot stopped.
To inspect the VZ console, stop the instance, enable Lima video output, then start it directly:
coop stop <instance>
limactl edit --tty=false --video coop-<instance>
limactl start coop-<instance>The display may show an initramfs prompt asking for a manual fsck. Follow the device named by the boot message only after confirming it is the root filesystem and is not mounted. If console repair is unavailable, use the offline procedure below. After diagnosis, stop the VM before touching its disk; limactl edit --tty=false --video=false coop-<instance> restores the usual headless display while it is stopped.
Never run host-side fsck or e2fsck on an instance disk while its VM is running. Filesystem repair changes guest data and can lose damaged files. Keep a separate backup so you can recover the pre-repair disk.
Host-side repair crosses coop's VM isolation boundary. The guest disk is untrusted. Attaching it causes macOS to parse guest-controlled partition metadata, and the commands below run debugfs and e2fsck as root on guest-controlled filesystem metadata. A maliciously crafted or corrupted disk could exploit a vulnerability in those host parsers and compromise the host. The safer option is to perform the diagnosis and repair inside a disposable VM that has access only to the stopped instance disk; configuring such a helper VM is outside the scope of this guide. Proceed with the host-side steps only if you accept this risk.
-
Run
coop stop <instance>and checklimactl listuntilcoop-<instance>isStopped. If a broken VZ process does not stop gracefully,limactl stop -f coop-<instance>is a last resort: it kills the VM without a clean guest shutdown and may add filesystem damage. ConfirmStoppedagain before continuing. -
Locate the instance directory at
${LIMA_HOME:-$HOME/.lima}/coop-<instance>. Identify the instance's disk there; Lima versions usediskordiffdisk, anddiskmay be a symlink todiffdisk. Resolve the path to the actual file, assign that absolute path todisk_image, and print it for confirmation. Verify that it is inside the selected instance directory and is not the golden image or another instance's disk. Copy it to a separate backup location with enough free space, for example:disk_image="/absolute/path/to/coop-INSTANCE/diffdisk" backup_image="/separate/backup/location/diffdisk.pre-fsck-YYYYMMDDTHHMMSS" printf 'Instance disk: %s\n' "$disk_image" if [ -e "$backup_image" ]; then printf 'Backup already exists: %s\n' "$backup_image" >&2 else cp -p "$disk_image" "$backup_image" fi
Replace both example paths, choose a new backup path, then verify that the copy completed.
-
Install
e2fsprogs(brew install e2fsprogs). Attach the stopped raw instance disk read-only withhdiutil attach -nomount -readonly -imagekey diskimage-class=CRawDiskImage "$disk_image". Note the whole-disk device reported byhdiutil, then rundiskutil list /dev/diskN(replacediskNwith that device). Identify the Linux ext4 root partition from the partition map; do not assume a partition number. If more than one ext4 partition exists, inspect each candidate withsudo "$(brew --prefix e2fsprogs)/sbin/debugfs" -R 'stat /etc' /dev/rdiskNsYand use the one containing the guest root directory. Do not select an EFI, swap, or data partition. If you cannot identify the root partition confidently, detach/dev/diskNand stop. -
Run a read-only diagnostic on the confirmed root partition:
sudo "$(brew --prefix e2fsprogs)/sbin/e2fsck" -fn /dev/rdiskNsY. ReplacerdiskNsYwith the partition you identified, and record the findings and exact exit status. Status0means no filesystem errors were found; detach/dev/diskNand do not run the repair pass. Exact status4means errors remain uncorrected and you may continue to step 5. Any other or combined status indicates a different failure; detach/dev/diskNand investigate instead of attempting repair. -
Detach the disk with
hdiutil detach /dev/diskN, then attach the same image for writing withhdiutil attach -nomount -imagekey diskimage-class=CRawDiskImage "$disk_image". Read the new device name and partition map again; macOS may assign a differentdiskN. Confirm the root partition again, then runsudo "$(brew --prefix e2fsprogs)/sbin/e2fsck" -fy /dev/rdiskNsYon that partition only. Exit status1means errors were corrected;0means no errors were found. For any other exit status or unresolved error, detach/dev/diskNbefore stopping to investigate. -
Run
sudo "$(brew --prefix e2fsprogs)/sbin/e2fsck" -fn /dev/rdiskNsYagain, record its result, and detach withhdiutil detach /dev/diskNregardless of the result. Runcoop start <instance>and verify SSH works only if the final check was clean and returned exit status0.
Keep the backup until the guest boots and its files are checked. If hdiutil cannot attach the disk, stop. If a later step fails, detach any attached disk before stopping; never boot or retry the instance while its disk remains attached to the host.
The Firecracker backend runs Firecracker microVMs with KVM hardware virtualization. Each instance is a lightweight VM with its own rootfs and Firecracker process. An open instance also has a TAP-backed virtio NIC; an isolated instance has neither.
- KVM access:
/dev/kvmmust exist and be readable/writable by the current user. Setup checks this and offers to fix permissions viasetfaclor by adding the user to thekvmgroup. - x86_64 or arm64 architecture: The Firecracker backend supports both. x86_64 is the primary test target; arm64 builds are produced but less exercised.
- curl and host utilities: Firecracker setup uses root-owned executables from
/usr/bin,/usr/sbin,/bin,/sbin,/usr/local/bin,/usr/local/sbin, and the NixOS system profile or sudo wrapper directories under/run. It checks the complete Coop-selected bootstrap family before setup changes storage, and checks the image-build family after any package installation but before downloading artifacts. A missing or rejected tool stops setup with its role and a repair hint. Setup does not search the invokingPATH. Starting a VM requires a running systemd system manager version 252 or newer plus trustedsystemd-runandsystemctlexecutables; there is no direct-launch fallback. Coop checks the manager version before creating or changing an instance. Older releases lack a qualified contract for coop's JSON unit listing and transient service behavior; upgrade the host before starting a Firecracker VM. The KVM integration matrix runs on Ubuntu 24.04 and 26.04; other hosts remain unverified. - System packages: Setup checks trusted
setfacl,unsquashfs, andmkfs.ext4paths, plus resolvedssh,scp, andrsyncclients. It also requires trustedinstallandmvtools to publish the managed binary pair. If tools are absent, it offers to install their Debian/Ubuntu packages (acl,squashfs-tools,e2fsprogs,openssh-client, andrsync) using a trusted systemapt-get. It checks ownership and writability of/etc/aptand files reached through apt configuration includes before automatic installation. Non-FHS OpenSSH/rsync installations can be selected with[ssh.host_tools]; see configuration. If trustedapt-getis unavailable, install the packages with your host's package manager, then reruncoop setup. The guest remains Ubuntu regardless of the host distribution. - Network control-plane tools:
sudo,ip,bridge,iptables, andsysctluse validated absolute system paths. Non-FHS hosts can provide exact per-tool paths under[network.host_tools]; see configuration.
coop setup prepares three artifacts:
- Firecracker and jailer: Setup downloads the pinned production v1.18.0 release, checks the architecture-specific archive digest, extracts both binaries, and publishes them together under
/var/lib/coop/firecracker/v1.18.0/. That directory and its ancestors must be root-owned and not writable by unprivileged users. Setup and launch verify both installed files against pinned digests. The VMM runs as root in a foreground systemd system service at this stage; the installed jailer is not yet used. - Guest kernel: Fetched from Firecracker's CI S3 bucket. The search starts at the managed release's minor version and can fall back to an older CI bucket when assets lag.
- Template rootfs: Built by downloading the Firecracker CI squashfs rootfs (Ubuntu-based), unpacking it, creating an ext4 image at the configured template size, and running an install script inside a chroot. The script installs Docker, GitHub CLI, Claude Code, Codex, Grok Build, and profile packages. It configures the
ubuntuuser with SSH keys and sets up systemd-networkd plus a socket-activated AF_VSOCK listener that forwards to loopback SSH. Newly created instances persist[vm].firecracker_control;vsockis opt-in, while existing and untagged instances retainlegacy_tcp. Image metadata records whether the listener is present, so selecting or restoring an older image in vsock mode fails before disk or instance state is changed and tells the user to rebuild it. Because committed disks contain guest-authored state, Firecracker commit repairs and verifies a staging disk before publishing fresh capability metadata. Both setup and commit mark publication incomplete before replacing the live disk; the marker prevents consumers from using mismatched disk and metadata. Plaincoop setup --image Xrefuses an image with this marker. Retry an interrupted commit from its source instance withcoop commit --image X --force, or explicitly rebuild an interrupted base image withcoop setup --image X --rebuild. A Firecracker image with metadata lacking the vsock listener capability needs explicit--rebuildeven if its recipe hashes differ. Rebuilding a committed checkpoint discards its contents.
All three steps are idempotent. Setup reuses a verified managed binary pair and does not overwrite an incomplete or modified root-owned release directory. If that directory fails verification, stop VMs using it, have an administrator remove that release directory, and rerun coop setup. An interrupted publication leaves a staging directory that a later setup removes after its timeout window; it does not replace a published pair. When coop pins a newer release, setup installs a separate versioned directory. Existing running VMs keep their executable, and stopped instance disks and image metadata are unchanged. Rolling back coop reuses the older installed pair if it still verifies. A custom or legacy firecracker_bin override must be removed before setup or launch. Older user-data-directory copies are no longer launched.
Creating an instance (coop up) follows this sequence:
- Copies the template rootfs to the instance directory using
cp --reflink=autofor copy-on-write on supported filesystems. - Mounts the copy and patches the guest network config with the instance's unique IP address, plus
/etc/hostnameand the matching/etc/hostsalias so the guest can resolve its own name. - Optionally resizes the rootfs if a larger disk was requested (truncate + e2fsck + resize2fs).
- Writes a Firecracker JSON config specifying the kernel, rootfs drive, vCPU/memory allocation, vsock device, and either the open-mode network interface or an empty interface list.
- In
openmode, creates and attaches a TAP device to the bridge (see TAP networking below). Isolated mode skips all per-instance host networking setup. - Verifies the installed root-owned Firecracker/jailer pair, then uses
systemd-runthroughsudoto start the VMM in a system service with--no-api. Systemd owns the service identity and process group; coop checks the service's main PID and cgroup before treating it as running. The VMM still runs as root and is not jailed. Firecracker's production binary supplies its default seccomp filters, but those filters do not provide jailer confinement. - Records the systemd main PID for compatibility and waits for SSH to become reachable.
- If
--mountwas specified, rsyncs the host directory into the guest. This is a one-time copy, not a live mount. Usecoop pushto update the guest; usecoop pull --dir <new-directory>to retrieve guest changes for review.
A running legacy direct-launch VM remains manageable until it stops. Its next
start uses the system service. A failed start or guest boot timeout retains the
instance disk. If systemd cannot confirm a service's process state, coop leaves
the PID, sockets, TAP, and disk for inspection instead of treating the VM as
stopped. The service unit name appears in coop status while a managed VMM is
running.
The code uses a typestate pattern (Configured then Running) to enforce valid lifecycle transitions at compile time.
Stopping a VM requests a guest reboot over SSH and waits up to ten seconds for
Firecracker to exit. If a managed VMM remains active, coop asks systemd to stop
its service and waits for the service cgroup to empty. A legacy direct-launch
VMM instead falls back to SIGTERM, then SIGKILL.
Each open Firecracker instance gets a dedicated TAP device (tap0, tap1, ...) derived from its instance index. Isolated mode boots without a NIC or TAP and retains control through vsock-backed SSH.
The network is configured as follows:
- A Linux bridge (
br0) is created if it does not already exist, with the configured host IP (default172.16.0.1/24). - IP forwarding is enabled via
sysctl. - iptables NAT masquerade and forwarding rules route guest traffic through the host's default network interface. The interface is auto-detected from the default route, or set explicitly via
network.host_ifacein the config. - A
FORWARD -i br0 -o br0 -j DROPrule is inserted at the head of the chain. If an existing rule has lost that precedence, startup fails until the host firewall configuration places it first. - Each instance's TAP device is created, attached to the bridge, marked as an isolated bridge port, and brought up.
- Guest IPs are assigned statically:
172.16.0.{index + 2}. Instance 0 gets172.16.0.2.
Open instances cannot reach each other by IP. Two controls enforce that and both are required: the isolated bridge-port flag blocks the direct L2 path, and the FORWARD rule blocks the L3 path a guest could otherwise take by routing through the host's bridge address. Each open guest still reaches the host and the internet. The flag is read back after being set, so a host that cannot apply it fails the VM start rather than booting an unisolated guest; this needs Linux ≥ 4.18 and iproute2 ≥ 4.19.
Isolation is applied per start. Because the kernel drops a frame only when both ports are isolated, a VM still running from before the upgrade leaves the whole bridge unisolated until it is stopped and started — not just itself.
docs/trust-model.md carries the full invariant and its known residuals (ARP/IP impersonation, IPv6, firewall reloads).
On teardown, any TAP left by a prior open boot is removed, including when the instance is now selected as isolated. If no TAP devices remain on the bridge, the bridge and all associated iptables rules are also removed.
The network section in config.toml controls Firecracker networking:
| Field | Default | Description |
|---|---|---|
host_ip |
172.16.0.1 |
IP address assigned to the bridge on the host side |
subnet_mask |
/24 |
CIDR subnet mask for the bridge network |
host_iface |
auto |
Host interface for NAT. auto detects the default route interface. Set explicitly if auto-detection fails. |
These settings are ignored on macOS. Lima handles its own networking. coop's generated Lima templates declare no networks: stanza, so each macOS guest gets its own user-mode NAT: instances are isolated from each other by construction there, with no shared bridge and nothing to enforce.
Resizing a stopped Firecracker instance's disk extends the rootfs image if needed, then runs e2fsck -fy and resize2fs to grow the filesystem in place. Retrying the same absolute size resumes an interrupted filesystem resize when the filesystem is repairable; it also checks a fully grown filesystem. Shrinking is not supported.
Memory and vCPU changes edit the machine-config block of the instance's per-instance JSON (vm_config.json), written atomically so a crash mid-write leaves the prior values intact. This JSON is authoritative: on every restart configure() regenerates the infra fields (kernel path, boot args, drive, network) from the global config so they roll forward, but preserves the on-disk mem_size_mib/vcpu_count rather than resetting them to the global [vm] defaults. Those defaults therefore only seed new instances. Firecracker does not boot the VM to apply the change; it takes effect on the next coop start (or immediately with --start).
The current Firecracker backend uses sudo for several operations:
- Starting the root-run VMM; creating an open-mode TAP is a separate privileged host operation
- Stopping the VM (the Firecracker process runs as root)
- Creating and manipulating TAP devices and bridge interfaces
- Managing iptables rules
- Rootfs operations during setup (chroot, mount, filesystem tools)
- Destroying instance directories (files owned by the root-owned Firecracker process)
The managed binary pair remains in /var/lib/coop/firecracker/v1.18.0/ after
coop destroy --all or coop uninstall, because it may be shared by other
users or installations. An administrator can remove it after all users have
stopped VMs that depend on it. Uninstall removes legacy copies from the user's
data directory; it does not remove an arbitrary firecracker_bin override.
coop supports cross-compiling from macOS to Linux for the Firecracker backend.
The project includes Cargo linker configuration for both release targets:
x86_64-unknown-linux-musl uses x86_64-linux-musl-gcc, and
aarch64-unknown-linux-musl uses aarch64-linux-musl-gcc. The matching cross
compiler must be installed and available on PATH; see
RELEASING.md for the target-scoped compiler and linker
variables used by the release preflight.
The integration test runner (tests/run-integration.sh --remote) automates this workflow: it cross-compiles a release build, copies the binary to the remote Linux host via scp, and runs the test suite there.
Both backends support the same core CLI commands and guest capabilities, with the intentional setup and mount differences listed below:
| Capability | Lima (macOS) | Firecracker (Linux) |
|---|---|---|
coop setup |
Builds golden image via builder VM | Installs binary + kernel, builds rootfs via chroot |
Explicit setup --extra-packages / --post-install |
Refused | Consumed by provisioning and image staleness checks |
coop up |
Creates or reconnects/restarts a project VM; --profile builds/starts a derived image |
Copies rootfs, configures the selected open-TAP or isolated-no-NIC mode, starts Firecracker; --profile builds/starts a derived image |
coop start |
Restarts a stopped Lima VM | Restarts a stopped Firecracker VM |
coop stop |
limactl stop |
Guest reboot over persisted SSH transport, then stop the systemd service and its cgroup; legacy direct processes retain the SIGTERM/SIGKILL fallback |
coop destroy |
limactl delete --force |
Kill process, remove any prior TAP, delete instance dir |
coop status |
Queries limactl list --json |
Queries systemd service and main PID (or legacy PID), then the guest via SSH |
coop logs |
Reads Lima's serial.log |
Reads Firecracker log file |
coop shell |
SSH to localhost on Lima-assigned port | SSH over the instance's persisted control mode: guest IP for legacy_tcp, Firecracker vsock ProxyCommand for vsock |
coop resize |
Disk: truncates Lima disk and updates lima.yaml disk:. Mem/vCPU: edits lima.yaml, validated via start |
Disk: extends rootfs image if needed, then runs fsck + resize2fs. Mem/vCPU: edits per-instance JSON |
| Resource monitoring | SSH query to guest | SSH query to guest |
| Docker in guest | Works (full kernel) | Works (with iptables-legacy workaround) |
--mount host mounts |
Live virtiofs (changes visible immediately) | One-time rsync sync (use push/pull to re-sync) |
| Needs sudo | No | Yes (VM start, stop, networking, rootfs ops) |