Gateway-upgrade E2E races dashboard forwarding before the recreated sandbox is ready

[cv]

Investigation Summary

• All five gateway-upgrade shards passed at commit 48a81588.
• At the release candidate, three shards pass and two fail.
• Both failures recreate the sandbox, then start dashboard forwarding before OpenShell reports it ready.
• OpenShell returns sandbox is not ready; recovery deletes the sandbox.
• PR fix(onboard): return nonzero for incomplete flow #9020 exposes the incomplete upgrade with exit code 1 but does not cause the readiness failure.

Description

Upgrade recovery must wait for the recreated sandbox readiness boundary before it starts dashboard forwarding.

The Arm64 v0.0.55 and x86-64 v0.0.36 shards fail at the same point. Other upgrade shards pass at the same commit, which indicates a timing-sensitive readiness race.

Reproduction Steps

1. Run the v0.0.55 Arm64 or v0.0.36 x86-64 gateway-upgrade E2E target.
2. Let the installer recreate the existing sandbox.
3. Observe the readiness wait and dashboard-forward startup.
4. Observe that forwarding receives sandbox is not ready.
5. Observe that recovery deletes the recreated sandbox and returns exit code 1.

Environment

• Last executed pass for all shards: https://github.com/NVIDIA/NemoClaw/actions/runs/31734723431
• Current v0.0.55 Arm64 failure: https://github.com/NVIDIA/NemoClaw/actions/runs/31743399726/job/94603993942
• Current v0.0.36 x86-64 failure: https://github.com/NVIDIA/NemoClaw/actions/runs/31743399726/job/94603994229

Debug Output

Waiting for sandbox to become ready...
Failed to start dashboard forward on port 18789:
sandbox is not ready

Acceptance Evidence

• Recovery verifies the exact recreated sandbox identity and ready state before forwarding.
• The wait has a bounded timeout and preserves diagnostics on failure.
• A transient not-ready response does not delete a sandbox that is still starting.
• Deterministic tests cover delayed readiness and terminal failure.
• All five gateway-upgrade shards pass for the same exact candidate commit.

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[senthilr-nv]

Maintainer follow-up claim for the fresh-create variant:

Automatic main E2E run 31886082826, job 95018948364, failed after ordinary onboarding accepted the sandbox-list Ready row but dashboard forwarding still received OpenShell sandbox is not ready; the orphan was removed. The same signature previously appeared in jobs 94964859783 and 94959167339.

The closed #9050 fix protects recreated sandboxes with an identity plus sandbox exec -- true readiness proof. The ordinary fresh-create path still passes no executable-readiness check into waitForCreatedSandboxReadyWithTrace, so a stale Ready row can advance to dashboard forwarding. Exact diagnostic, target, and owning-path open-PR searches found no current owner.

I am claiming this root cause for a small follow-up: require the machine-readable executable-readiness boundary for ordinary fresh creation before dashboard forwarding, preserve bounded failure diagnostics and identity checks for recreated sandboxes, and add a deterministic regression. I will link the draft PR here after publication. No E2E rerun or manual dispatch was issued.