Legacy upgrade leaves healthy patched container with sandbox stuck Deleting

[rsliter]

Summary

The v0.0.89 x86-64 upgrade E2E rebuilt a healthy patched container, but the OpenShell sandbox remained in Deleting until the 900-second final handoff deadline expired. The installer then failed recovery and preserved the backup.

This is a product or integration regression with an intermittent trigger, not a flaky assertion. The same exact job passed in four recent full E2E runs.

Current evidence

• Run: https://github.com/NVIDIA/NemoClaw/actions/runs/32178707220
• Job: https://github.com/NVIDIA/NemoClaw/actions/runs/32178707220/job/95856132222
• Exact SHA: b2d1ce52a716444b083f6e3b8ed8bace1cba3240
• Test: Upgrade: migrates v0.0.89 state on x86-64 / GitHub read token
• Failing phase: upgrade the old working OpenClaw sandbox to the current OpenShell gateway.
• Phase duration before failure: 1,236 seconds.

The final handoff diagnostic reported:

OpenShell supervisor did not reach Ready (last phase: Deleting).
sandbox_phase=Deleting
patched_container_status=running
patched_container_running=true
patched_container_exit_code=0
patched_container_health=healthy

The installer waited the full 900-second handoff budget, failed recovery, retained the registry entry and backup, and exited 1. No retry ran.

Recurrence evidence

The same exact job passed recently:

• https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95696291170
• https://github.com/NVIDIA/NemoClaw/actions/runs/31827969357/job/94858195174
• https://github.com/NVIDIA/NemoClaw/actions/runs/31816437247/job/94821954853
• https://github.com/NVIDIA/NemoClaw/actions/runs/31767577321/job/94669311981

Closed #4664 and #5662 cover earlier GPU-patch handoff failures where a healthy container and OpenShell phase diverged. This occurrence is distinct because the sandbox remained in Deleting during a legacy-upgrade recovery path.

Required behavior

1. Determine whether NemoClaw's container handoff or the OpenShell state machine owns the persistent Deleting phase.
2. If OpenShell owns it, attach sanitized upstream evidence and record the upstream issue here.
3. A healthy patched container must not remain indefinitely detached from the sandbox lifecycle record.
4. Recovery must either converge to the same ready sandbox identity or fail with the preserved backup and a stable ownership diagnostic.
5. Add deterministic coverage at the smallest lifecycle-state boundary for a healthy container paired with a persistent Deleting sandbox phase.
6. Keep the live E2E assertion on restored survivor state and lifecycle readiness.

Non-goals

• Increasing the 900-second timeout without evidence.
• Retrying the complete workflow or upgrade mutation.
• Treating a healthy Docker container as success while OpenShell remains non-ready.
• Adding another live E2E target.

Release signal

Classification: product or integration regression with an intermittent trigger. It is terminal for the candidate and recovered zero times.

[ericksoa]

Additional same-SHA gateway reproduction and proposed design

This also reproduced in the gateway recovery qualification for #9140 / PR #9323, and the red-to-green retry isolates the failure as an event-ordering race rather than runner slowness or an assertion timeout.

Same-SHA evidence

• Workflow run: https://github.com/NVIDIA/NemoClaw/actions/runs/32367659836
• Exact candidate SHA: b85031a83345df2f86468d0fb16e0587f3265468
• Red attempt: https://github.com/NVIDIA/NemoClaw/actions/runs/32367659836/job/96421038598
• Green retry: https://github.com/NVIDIA/NemoClaw/actions/runs/32367659836/job/96435245781
• Test: gateway recovery restores /tmp guard chain after pod-recreate wipe (#2701)
• Boundary: legacy-keepalive container recreation followed by OpenShell readiness

In both attempts, the fixture successfully replaced the exact pinned container identity, proved the replacement executable through OpenShell, removed the backup, and returned success.

After it returned:

• Red attempt: all 12 readiness probes observed phase: Deleting; none recovered.
• Green attempt: the first probe observed phase: Error, and the second reached Ready.

The same code and candidate SHA therefore produced different controller outcomes after an otherwise successful handoff.

Confirmed code-level race

The immediate recreation path in recreateOpenShellDockerSandboxContainer:

1. waits for the replacement supervisor to become executable;
2. calls reconcileSupervisorReconnect;
3. on success, removes the old backup container as the final Docker lifecycle event;
4. returns success immediately afterward.

That final backup-removal event can cause OpenShell to demote the already-working replacement to Deleting.

The repository already documents and avoids this ordering in finalizeDockerGpuPatchBackup: it stops the replacement, removes the backup, then starts the exact replacement again so the replacement registration is the final authoritative lifecycle event. The immediate and deferred finalization paths currently implement different handoff semantics.

Proposed design

Unify immediate and deferred recreation behind one identity-pinned final-handoff operation:

1. Pin the transaction identities

   • Carry the exact old container ID, exact replacement container ID, original name, and backup name through the handoff.
   • Refuse to commit if any observed identity differs.

2. Prove the replacement before destructive commit

   • Confirm the replacement supervisor is reachable.
   • Keep the backup until that proof succeeds.

3. Use one authoritative commit ordering

   • Stop the exact replacement container.
   • Remove the exact backup container.
   • Start the exact replacement container.
   • This makes replacement registration, not backup deletion, the final lifecycle event.

4. Require a post-commit acknowledgement

   • Do not return success merely because docker start succeeded.
   • Return success only after OpenShell acknowledges the final replacement as Ready and, where the OpenShell contract exposes it, reports the expected runtime/container identity or generation.
   • If the current status API cannot bind readiness to identity, add a lifecycle receipt or generation field at this boundary rather than inferring ownership from elapsed time.

5. Classify controller states

   • Deleting after the final replacement start is a terminal ownership failure and should fail immediately with the exact transaction identities.
   • A transient Error may be observed only while the exact replacement is alive and controller registration is still progressing.
   • A deadline may remain as process protection, but neither additional sleep nor repeated workflow retries is an acceptance condition.

6. Make the E2E assert the production contract once

   • After the recreate helper returns, perform one openshell sandbox exec ... -- true assertion and verify the exact replacement identity.
   • Remove the current blind 12-attempt readiness loop. If the helper returns while the sandbox is Deleting, the E2E should fail immediately as a product-contract violation.

Acceptance criteria

• Immediate and deferred handoffs share the same finalization implementation and lifecycle ordering.
• Unit coverage proves stop(replacement) -> remove(backup) -> start(replacement) ordering using exact identities.
• Unit coverage proves Deleting is not retried as generic readiness.
• The recreate helper cannot report success before post-commit OpenShell acknowledgement.
• The gateway recovery E2E no longer uses a blind retry loop after helper success.
• The gateway recovery job passes repeated executions without increasing timeouts or retrying the workflow.
• Diagnostics preserve the old ID, replacement ID, backup name, last OpenShell phase, and rollback disposition.

Non-goals

• Increasing the readiness or handoff timeout.
• Retrying the complete E2E job.
• Treating Docker health alone as successful OpenShell recovery.
• Adding another E2E target.