Skip to content

Remote-host sweep treats a tombstoned project's or workroom's hosts as unknown #296

Closed

Description

@joelmoss

🤖 Left by Claude Code. This was found by the adversarial review of #294 (#287). It's low likelihood and wasn't reproduced.

The bug

apply passes the tombstone-filtered project list to the remote-host adopter:

// AppStore.swift, apply
let fresh = applyingDeletionTombstones(sorted)
if RemoteWorkrooms.isEnabled { RemoteHosts.shared.adopt(fresh) }

RemoteHosts.adopt builds the sweep's keep-set from that same list. The sweep runs once per launch, on the first adopt:

// RemoteWorkrooms.swift, RemoteHosts.adopt
let shouldSweep = lock.withLock { defer { swept = true }; return !swept }
guard shouldSweep else { return }
let known = Set(recorded.compactMap(\.id))
Task.detached(priority: .utility) { for failure in await driver.sweep(keeping: known) { … } }

ContainerHostDriver.sweep(keeping:grace:) runs rm --force --volumes on every labelled container, and removes every labelled image, that isn't in known and is older than grace (20 minutes).

A project in deletingProjects (#294), or a workroom in deletingWorkrooms (#116), is missing from fresh. If the sweep's first run happens during that delete, the sweep treats the item's hosts as unrecorded and removes them.

When it bites

All of these must happen together:

  1. The sweep hasn't run yet this launch. For example, the remote-workrooms preview was turned on after launch, so no adopt had found anything to record.
  2. A delete of a project or workroom with remote hosts is in flight, and a reload runs during it.
  3. A host is older than 20 minutes.
  4. The delete then fails before taking that host down. Config still records the host, but the sweep has destroyed it.

If the delete succeeds, the sweep only removes what was being removed anyway.

Fix options

  • Build the keep-set from the unfiltered list. For example, pass sorted to adopt, or a separate known set built from it. The tombstone should hide the item from the UI, not from host bookkeeping. Check that adopting a tombstoned item's hosts is harmless. adopt only registers hosts it doesn't already hold, so it looks safe.
  • Or defer the sweep while any tombstone is set, and let the next adopt run it.

The first option is smaller and fixes the cause.

Related: #116, #253, #287, #294.

Activity

  1. added 2 commits that reference this issue on Oct 3, 2026
    bee8d92
    a391449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions