[Ubuntu 24.04][Upgrade] v0.0.55 to v0.0.93 leaves sandbox stuck in Provisioning; nemoclaw rebuild fails with legacy registry entry error

[lyral-commits]

Warning

Regression — v0.0.55 to v0.0.76 upgrade was fixed (GH #6114, closed 2026-07-20); direct v0.0.55 to v0.0.93 upgrade re-introduces the same stuck-Provisioning symptom with an additional rebuild block not present in the original report.

Description

Upgrading NemoClaw directly from v0.0.55 (OpenShell 0.0.44) to v0.0.93 (OpenShell 0.0.85) leaves pre-existing sandboxes permanently stuck in Provisioning phase. Unlike the similar issue fixed for v0.0.55 to v0.0.76 (GH #6114), this regression also blocks manual recovery: running nemoclaw {sandbox} rebuild fails with "legacy registry entry cannot distinguish a managed image from a custom --from image". The only recovery path is to destroy the sandbox and re-onboard from scratch, losing any workspace state not covered by the automatic backup.

Platform scope: Reproduced on Ubuntu 24.04.4 LTS (x86_64) only; other platforms not tested.
Regression: Yes — v0.0.55 to v0.0.76 upgrade was fixed (GH #6114, closed 2026-07-20); direct v0.0.55 to v0.0.93 upgrade re-introduces the same stuck-Provisioning symptom with an additional rebuild block not present in the original report.

Environment

Device:        Ubuntu 24.04 x86_64 server
OS:            Ubuntu 24.04.4 LTS
Architecture:  x86_64
Node.js:       v22.23.1
npm:           10.9.8
Docker:        29.6.2
OpenShell CLI: 0.0.44 (source) / 0.0.85 (after upgrade)
NemoClaw:      v0.0.55 (source) / v0.0.93 (after upgrade)
OpenClaw:      v2026.5.22 (in sandbox, before upgrade)

Steps to Reproduce

1. Install NemoClaw v0.0.55:

   curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.55 bash

2. Complete onboard, creating sandbox "alpha" (openclaw agent, any provider). Confirm: nemoclaw --version shows v0.0.55; openshell --version shows 0.0.44; sandbox Phase: Ready.
3. Add a messaging channel (e.g. Slack) to trigger a sandbox rebuild, so the sandbox has an active channel before upgrade.
4. Write a workspace marker:

   nemoclaw alpha exec -- bash -c "echo upgrade-test > /sandbox/.openclaw-data/workspace/marker.txt"

5. Upgrade directly to v0.0.93:

   curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.93 bash

   When prompted "Confirm these were managed-image sandboxes? [y/N]:", answer y.
6. After upgrade completes, run: nemoclaw alpha status
7. Run: nemoclaw alpha rebuild --yes

Expected Result

• Step 6: nemoclaw alpha status shows Phase: Ready, OpenClaw version matches expected v2026.7.1.
• Step 7: Rebuild succeeds or is not needed; sandbox is accessible and workspace/channel config is preserved.

Actual Result

Step 6: nemoclaw alpha status shows Phase: Provisioning with:

OpenShell: 0.0.44 (docker)
Agent: OpenClaw version not verified (expected v2026.7.1)
Update: unable to verify sandbox OpenClaw version
        Run nemoclaw alpha rebuild if this sandbox predates the current install
Sandbox is stuck in Provisioning phase.

Sandbox container logs show repeated auth failures:

Policy fetch failed after 5 attempts: no sandbox token source available

Step 7: nemoclaw alpha rebuild --yes fails immediately with:

Rebuild preflight failed: recorded custom Dockerfile is invalid.
legacy registry entry cannot distinguish a managed image from a custom --from image
Sandbox is untouched — no data was lost.

The only recovery path is nemoclaw alpha destroy followed by re-onboarding.

Logs

Upgrade output (relevant excerpt):
[INFO]  Retiring OpenShell 0.0.44 gateway before installing current OpenShell...
[INFO]  Backing up 1 sandbox(es) before upgrading OpenShell...
  Backing up alpha: 11 dirs, 1 files backed up

Post-upgrade sandbox container stderr:
  Policy fetch failed after 5 attempts: no sandbox token source available

nemoclaw alpha rebuild --yes:
  Rebuild preflight failed: recorded custom Dockerfile is invalid.
  legacy registry entry cannot distinguish a managed image from a custom --from image

[nvshaxie]

Picking this up. Initial code reading on the rebuild half of the report (the nemoclaw alpha rebuild block), before I run the v0.0.55 → v0.0.93 repro:

isManagedImageRecoveryAllowed (src/lib/state/sandbox.ts:1919) admits a pre-fingerprint row only when allowLegacyManagedImageRecovery is passed, and hasPositiveManagedImageEvidence (src/lib/state/sandbox.ts:1907) is satisfied only by a non-empty nemoclawVersion on the registry entry. rebuild sets that flag only alongside a recovery manifest (src/lib/actions/sandbox/rebuild-preflight-phase.ts:110), whereas the upgrade path passes it directly (src/lib/actions/upgrade-sandboxes.ts:127).

So the operator y answer to "Confirm these were managed-image sandboxes?" appears to authorize the upgrade run only, without stamping the managed-image fingerprint onto the v0.0.55-era entry. A later standalone nemoclaw <sandbox> rebuild then has neither positive evidence nor a manifest and fails closed permanently — which matches the reported "only recovery path is destroy and re-onboard".

I will confirm this against a real v0.0.55 → v0.0.93 upgrade before proposing a fix. The stuck-Provisioning half (Policy fetch failed after 5 attempts: no sandbox token source available) has no source match in this repo and may sit on the OpenShell side; I will report back on whether the two halves need to be split.

[nvshaxie]

Follow-up on the rebuild half. My earlier comment read the missing nemoclawVersion stamp as the root cause; further code reading narrows that.

A successful upgrade does re-stamp the entry. upgrade-sandboxes recovery calls rebuildSandbox(name, ["--yes"], { recoveryManifest, allowLegacyManagedImageRecovery }) (src/lib/actions/upgrade-sandboxes.ts:493), rebuild recreates through onboard({ resume: true, nonInteractive: true, recreateSandbox: true }) (src/lib/actions/sandbox/rebuild-recreate-phase.ts:166), and createSandbox() owns the stamp (src/lib/onboard/sandbox-agent.ts:119). So the fingerprint guard is not what breaks the upgrade.

What makes it unrecoverable is that rebuild has no operator authorization path. allowLegacyManagedImageRecovery is true only when a recovery manifest is also present (src/lib/actions/sandbox/rebuild-preflight-phase.ts:110), and the only caller that sets it is upgrade-sandboxes, gated on NEMOCLAW_CONFIRMED_LEGACY_MANAGED_SANDBOXES (src/lib/actions/upgrade-sandboxes.ts:142). There is no CLI flag and no prompt. A standalone nemoclaw <sandbox> rebuild therefore always takes the pre-fingerprint branch in src/lib/actions/sandbox/rebuild-durable-config.ts:232 and fails closed with no way forward.

That makes the two reported halves one failure chain: the post-upgrade recreate does not complete (Policy fetch failed after 5 attempts: no sandbox token source available, sandbox left in Provisioning), so the v0.0.55-era entry is never stamped; and because rebuild exposes no equivalent confirmation, destroy plus re-onboard is the only path left — which matches the report.

Two separable fixes:

1. Root cause: why the recreate leaves the sandbox in Provisioning with no token source across the OpenShell 0.0.44 to 0.0.85 jump.
2. Recoverability: give a standalone rebuild the same per-sandbox managed-image confirmation the upgrade path already has, so an operator is not dead-ended.

I am still confirming (1) against a real v0.0.55 to v0.0.93 upgrade. (2) is testable without the full repro, but it loosens a fail-closed guard that deliberately refuses to distinguish a managed image from a custom --from image, so I would like maintainer direction on whether that confirmation belongs on rebuild before I propose it.

[nvshaxie]

Status check — the analysis above still holds at current main da1b10312, so this is not stale, and I am still blocked on one scope decision.

Re-verified on today's main (line numbers shifted, structure unchanged):

• isManagedImageRecoveryAllowed still admits a pre-fingerprint row only via allowLegacyManagedImageRecovery (src/lib/state/sandbox.ts:1921-1927).
• rebuild still derives that flag as opts.recoveryManifest !== undefined && opts.allowLegacyManagedImageRecovery === true (src/lib/actions/sandbox/rebuild-preflight-phase.ts:112-113), so a manifest is mandatory.
• NEMOCLAW_CONFIRMED_LEGACY_MANAGED_SANDBOXES is still read in exactly one place, src/lib/actions/upgrade-sandboxes.ts:142. There is no rebuild-side flag, prompt, or environment equivalent anywhere in src/.

Nothing merged since 2026-07-25 touches this path; the only commits in the relevant files are #7711 (inference reasoning-effort) and #7194 (baseline policy exclusions), neither of which changes the recovery guard.

The open question is unchanged and is a scope decision, not a technical one. Fix (2) — giving a standalone rebuild the same per-sandbox managed-image confirmation the upgrade path already has — deliberately loosens a guard that fails closed precisely because it cannot distinguish a managed image from a custom --from image. Under the Product Scope Gate that is a new supported operator surface, so I do not want to propose it without maintainer direction on whether the confirmation belongs on rebuild at all, and if so whether it should be a flag, an interactive prompt, or the same environment-variable shape the upgrade path uses.

@aaronerickson @cv — a yes/no on that surface is enough to unblock; I will take the shape decision from whichever of the three you prefer.

On fix (1), the root cause of the recreate leaving the sandbox in Provisioning with Policy fetch failed after 5 attempts: no sandbox token source available: I still cannot close this one out locally. Reproducing it needs a real v0.0.55-era sandbox carried across the OpenShell 0.0.44 → 0.0.85 jump, and the host available to me runs Docker but no OpenShell gateway or registered sandbox. If someone on QA still has the original upgrade environment from this report, the nemoclaw <sandbox> logs output around the failed policy fetch would let me finish the diagnosis without rebuilding that history from scratch.

[hulynn]

Reproduced on a new platform, still present in v0.0.114

Confirmed this is still reproducible on the current release, on a platform not previously covered by this report.

Environment: NVIDIA Jetson AGX Thor Developer Kit (aarch64, L4T R39.2). Upgraded in-place from a clean NemoClaw v0.0.55 install directly to v0.0.114 (official curl|bash installer, no manual steps), confirming these were managed-image sandboxes when prompted.

Result: identical to this report — the CLI/OpenShell upgrade itself completed successfully and the pre-existing sandbox's registry entry was preserved with no reported data loss, but the sandbox rebuild step failed at preflight with:

Rebuild preflight failed: recorded custom Dockerfile is invalid.
legacy registry entry cannot distinguish a managed image from a custom --from image

Same result on a direct retry of nemoclaw {sandbox} rebuild after the upgrade completed. No CLI-driven recovery path was found — the sandbox is left registered but permanently unable to rebuild.

This confirms the issue is not specific to Ubuntu 24.04 or to the v0.0.55→v0.0.93 upgrade path specifically — any sandbox that predates managed-image provenance tracking appears to hit this on upgrade to a current release, regardless of platform.

[hulynn]

Corroborating reproduction from today's v0.0.114 sanity pass, on a new platform: upgrading a v0.0.55-era sandbox all the way to v0.0.114 (further than the v0.0.93 originally reported) still hits the identical "legacy registry entry cannot distinguish a managed image from a custom --from image" error on rebuild, now confirmed on a Jetson AGX Thor (aarch64) host in addition to the original Ubuntu 24.04 x86_64 report. The sandbox in this run was independently confirmed to have been created with the standard managed image (no --from flag ever used), so this isn't a case of a genuinely custom-image sandbox being correctly rejected. The affected sandbox was left in place, stopped and un-upgraded, with no data loss reported by the product at any failed attempt.