[DGX Station][Upgrade] strict pre-upgrade backup counts an uncapturable sandbox as neither failed nor skipped, so the gate never fires

[wangericnv]

Description

A registered sandbox that cannot be captured at all is counted as neither failed nor skipped by the strict pre-upgrade backup, so the strict gate does not fire. The summary line reports a clean backup while one sandbox was never backed up, and the installer continues past the backup phase instead of stopping before the legacy OpenShell gateway work.

An operator reading the summary would reasonably conclude every sandbox was captured. The sandbox that was not captured is mentioned only on a separate line, in different wording, and is not reflected in any of the three counters.

Platform scope: reproduced on DGX Station GB300 (Linux aarch64); the counting happens in the installer's backup phase and is not expected to be hardware specific.
Regression: Unknown — earlier builds not tested for this exact counting.
OpenShell issue: No — the miscount and the missing abort are on the NemoClaw installer side; the OpenShell gateway was correctly left at its old version.

Environment

Device:        NVIDIA DGX Station GB300
OS:            Ubuntu 24.04.4 LTS
Architecture:  aarch64
Starting from: nemoclaw v0.0.121, openshell 0.0.106 (gateway still needing the legacy upgrade)
Upgrading to:  the current maintained release via the documented installer

Steps to Reproduce

1. On a host running an older maintained release whose OpenShell gateway still needs the legacy upgrade, register two managed-image sandboxes:
   • keep-good — Ready, its pre-upgrade capture will succeed
   • keep-stuck — registered, but made uncapturable
2. Make keep-stuck uncapturable by removing its backing container out of band, so it can no longer be started for backup: docker rm -f <the keep-stuck container>
3. Confirm both are still registered: nemoclaw list
4. Run the documented in-place upgrade: curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
5. Read the pre-upgrade backup phase: the per-sandbox result lines, the summary counters, and the line printed when the installer stops.

Expected Result

The sandbox that could not be captured is reported as failed or skipped and is counted as such, for example Pre-upgrade backup: 1 backed up, 1 failed or ... 1 skipped, and the strict gate fires before any legacy gateway work, printing the documented abort line.

Actual Result

The backup phase prints:

Backing up 'keep-good'...
keep-good: 12 dirs, 1 files, stored under the gateway's rebuild-backups directory
Pre-upgrade backup: 1 backed up, 0 failed, 0 skipped
Backups stored in: <backup path>
1 recorded sandbox(es) were not found on their recorded gateway: keep-stuck.

So keep-stuck is counted in none of the three counters, and the summary reads as a clean backup. The installer does not stop here — it proceeds into the sandbox upgrade phase:

keep-good   v2026.7.1 unchanged; NemoClaw image v0.0.121 to v0.0.123  (running)
keep-stuck  v2026.7.1 unchanged; NemoClaw image v0.0.121 to v0.0.123  (stopped)
Backup recovery blocked:
  keep-stuck  no validated pre-upgrade backup was found

and only much later exits nonzero with a different message:

Installation incomplete: one or more existing sandboxes failed to upgrade.

The documented abort line about stopping before retiring the legacy OpenShell gateway never appears.

What did work: the legacy gateway was not retired (openshell stayed at 0.0.106), both sandboxes remained registered, and keep-good status stayed Ready. So the end state is safe; the defect is that the strict backup gate did not fire and its counters do not account for an uncapturable sandbox.

Related — checked and distinct

• **stopped sandboxes are started for backup but remain SSH-unreachable and block strict upgrade. That is the opposite outcome: there the gate fires and blocks; here a sandbox that cannot be captured at all does not make the gate fire. The conditions differ too — there the backing container still exists and only SSH is unreachable; here the backing container was removed so the sandbox cannot be started for backup at all. Worth fixing together, since both concern which uncapturable states the strict gate recognises.

[Thabhelo]

Picking this up.

Root cause on current main (src/lib/actions/maintenance.ts, backupAll): a registered sandbox whose container is confirmed absent is routed into the stranded-orphan list, which never increments failed or skipped. The summary line then reports a clean backup and the strict gate (failed > 0 || (requireAll && skipped > 0)) never fires, so the installer continues past the backup phase.

Plan: count confirmed-stranded sandboxes in the summary and make the strict gate fail on them, with regression tests in maintenance.test.ts for both the strict and non-strict paths. PR to follow.

[Thabhelo]

One correction after reading the history. The stranded path is a deliberate exemption from #7290 (follow-up to #6520): a record left behind by nemoclaw uninstall plus reinstall has nothing to back up, so strict backup-all warns and lets the installer reach its recovery phase instead of hard-aborting. This report is the other face of that exemption: the gateway is still alive, the container was removed out of band, and the upgrade phase then refuses the same sandbox ("no validated pre-upgrade backup was found") much later.

Two ways to close the gap, and I would like a maintainer call before opening the PR:

1. Count confirmed-stranded sandboxes in the summary (1 backed up, 0 failed, 0 skipped, 1 stranded) and make the strict gate fail on them, keeping the existing destroy/onboard guidance so the operator is told what to do at the abort. This matches the expected result above and fails fast instead of in the upgrade phase. It changes the fix(cli): let strict backup-all skip stranded orphan sandboxes #7290 contract for the uninstall-and-reinstall flow.
2. Keep the exemption but make the summary honest with the same stranded counter and a one-line warning that these sandboxes were not backed up. No gate change.

I am preparing option 1 on a branch with the summary change as a separate commit, so falling back to option 2 is a one-commit drop. Happy to go either way.

[Thabhelo]

Draft PR with both options is up at #12334: commit 1 is the summary count, commit 2 is the strict pre-upgrade gate (pre-uninstall keeps the #7290 behavior). Ready to mark it ready for review once a maintainer picks one.