[CLI] no in-place sandbox restart or reconcile command; operators must kubectl delete pod after every CR edit

[davidglogan]

nemoclaw <name> subcommands cover connect, status, logs, snapshot create/restore/list, rebuild, destroy, skill install, policy-add/list/remove. None of them reconcile the current CR into the running pod.

rebuild is the closest, but it is documented as "Upgrade sandbox to current agent version": heavyweight, touches the image, not intended for "the pod spec just changed, please roll the pod." destroy removes the sandbox, too destructive for this purpose. snapshot restore rebuilds from a snapshot, not from the live CR.

Result: when the Sandbox CR spec changes (for example, after a hostAliases patch that has no CLI surface; see companion issue), the operator's only option is:

docker exec openshell-cluster-nemoclaw kubectl -n openshell delete pod my-assistant
# wait for the operator to recreate the pod from the updated CR

Environment

• NemoClaw v0.0.17, OpenShell v0.0.26
• Sandbox: my-assistant on DietPi2 (x86_64, 64GB, MicroK8s)
• Context: added a new hostAliases entry to the CR; needed to force the operator to re-roll the pod

Expected behaviour

nemoclaw my-assistant restart [--wait]
# or
nemoclaw my-assistant reconcile [--wait]

Delete and recreate the pod (preserving the PVC / memory), wait for Ready, exit 0 once the new pod is serving. Keep snapshots as the heavy-surface backup / rollback primitive.

Actual behaviour

No such command. Operators who need an in-place pod refresh must kubectl delete pod inside the k3s container, which (a) the docs discourage and (b) requires knowing the cluster name and the namespace layout.

Impact

• CR-edit workflows end with a kubectl detour on the docker-exec path.
• Scripting a "hot-reload policy + DNS" workflow has no supported entrypoint.
• Diagnostic workflows (restart the pod, watch the fresh boot logs) currently need multi-step manual steps.

Suggested fix

Add nemoclaw <name> restart that:

1. Deletes the pod via the operator's authority, not direct kubectl.
2. Waits for the operator to recreate it.
3. Waits for the new pod to reach Ready.
4. Optionally prints the boot log tail.
5. Exits non-zero if anything times out.

Accept --wait-timeout <seconds> with a sensible default (300s).

Companion issues

Third in a cluster of three CLI-gap filings:

• [CLI] nemoclaw policy-add has no custom-preset surface; custom egress targets require dropping to openshell policy set #2039: nemoclaw policy-add has no custom-preset surface
• [CLI] no CLI surface for sandbox hostAliases management; operators must docker exec + kubectl patch #2040: no CLI surface for sandbox hostAliases management
• this issue: no in-place sandbox restart / reconcile command

All three describe the same pattern: the supported CLI covers the common cases, but custom-infrastructure edits drop below the supported surface. #2039 covers the egress-policy axis, #2040 covers the DNS axis, and this issue covers the reconcile axis. A workflow that edits the Sandbox CR (for example to add a hostAliases entry via #2040's workaround) typically needs #2039's policy-edit and this issue's reconcile-pod step to complete end-to-end.

Notes

Filed on 2026-04-17 alongside #2039 and #2040.

[perlowja]

Issue understanding

Thanks for laying this out. My reading is that #2041 is the orchestration gap: once sandbox state drifts, operators currently have to rely on incidental connect recovery or manually force the pod/service path back into shape. #2042 is the concrete gateway/port-forward failure mode, while #2595 and #2604 show the same underlying problem surfacing as misleading or incomplete status output.

Implementation offer

I can implement two first-class NemoClaw CLI commands at the orchestration layer:

• nemoclaw <name> restart: stop and start the sandbox components without data loss, then verify that gateway, forwards, tokens, and readiness all re-establish.
• nemoclaw <name> reconcile: diagnose state divergence, e.g. process alive but port-forward dead, gateway process running but token expired, selected gateway refusing connections, or local status stale, and repair the specific failed component without restart-from-scratch.
• Both commands would be non-destructive: no rebuild, no snapshot rollback.
• --json would emit schema_version: 1 plus per-component diagnosis/repair outcomes so scripts can distinguish repaired, already_healthy, failed, and skipped.
• Scope feels like roughly 600-800 LOC including focused pytest coverage.

Implementation sketch

@app.command("restart")
def restart(name: str, json: bool = False, wait: bool = True):
    result = orchestration.restart(name, wait=wait)
    emit(result, json=json)

def restart(name: str, wait: bool) -> RecoveryResult:
    stop_runtime_components(name, preserve_storage=True)
    start_runtime_components(name)
    return reconcile(name, wait=wait)

def diagnose(name: str) -> list[ComponentDiagnosis]:
    return [
        check_gateway_process(name),
        check_dashboard_forward(name),
        check_gateway_auth(name),
        check_selected_gateway(name),
    ]

def reconcile(name: str, wait: bool) -> RecoveryResult:
    diagnoses = diagnose(name)
    repairs = [repair(d) for d in diagnoses if d.state != "healthy"]
    return RecoveryResult(schema_version=1, sandbox=name, components=repairs)

@pytest.fixture
def sandbox_with_dead_forward(fake_runtime):
    fake_runtime.gateway.alive = True
    fake_runtime.forward("dashboard").alive = False
    fake_runtime.token.valid = True
    return fake_runtime

I would keep this out of lower-level sandbox primitives: the command should compose existing NemoClaw health checks and recovery paths, with explicit reporting and tests for idempotency. Recovery semantics are tricky to get right, especially when partial recovery succeeds, so I would make the JSON contract and exit-code behavior part of the test matrix rather than an afterthought.

Coordination ask

Want me to open a PR with both commands, or ship restart first and reconcile as a follow-up?

[davidglogan]

Thanks @perlowja, and apologies for the slow reply.

Quick answer to your coordination ask: I'd lean toward shipping restart first as a standalone PR, then reconcile as a follow-up. Two reasons.

1. restart is the narrower, immediately-useful surface. Most of my own CR-edit workflows would close out at that step alone.
2. The reconcile schema (component diagnosis + repair categories) is its own conversation, and easier to extend cleanly if it isn't bundled with restart.

The sketch in your comment lines up with what I had in mind when filing. The non-destructive guarantee, the explicit schema_version: 1 JSON contract, and the repaired / already_healthy / failed / skipped outcomes all read right.

Once you have a restart branch up, happy to exercise it against my setup and report back. Let me know when there's a branch to pull.

Thanks for picking this up.

[perlowja]

On PTO this week but will take up when I get back.

…

On Tue, May 12, 2026, 1:42 PM Dave Logan ***@***.***> wrote: *davidglogan* left a comment (NVIDIA/NemoClaw#2041) <#2041 (comment)> Thanks @perlowja <https://github.com/perlowja>, and apologies for the slow reply. Quick answer to your coordination ask: I'd lean toward shipping restart first as a standalone PR, then reconcile as a follow-up. Two reasons. 1. restart is the narrower, immediately-useful surface. Most of my own CR-edit workflows would close out at that step alone. 2. The reconcile schema (component diagnosis + repair categories) is its own conversation, and easier to extend cleanly if it isn't bundled with restart. The sketch in your comment lines up with what I had in mind when filing. The non-destructive guarantee, the explicit schema_version: 1 JSON contract, and the repaired / already_healthy / failed / skipped outcomes all read right. Once you have a restart branch up, happy to exercise it against my setup and report back. Let me know when there's a branch to pull. Thanks for picking this up. — Reply to this email directly, view it on GitHub <#2041 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BGW2SJEYPK7EX2KQN6RC4FL42NWC3AVCNFSM6AAAAACX5NQNT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMZTGY2TKOJQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>