[Linux][Upgrade] v0.0.55→v0.0.81 legacy-sandbox recovery drops Slack channel config + user data outside /sandbox/.openclaw

[hulynn]

Description

When upgrading NemoClaw in place from v0.0.55 to v0.0.81 (curl|bash), a pre-existing (legacy) sandbox is now recovered to Ready via the new legacy-recreate guard + pre-upgrade backup + restore. The earlier "stuck in Provisioning/Error" break is fixed (the sandbox comes up Ready and the agent runs). However the recovery is incomplete — it silently drops two kinds of pre-upgrade state:

1. The configured Slack messaging channel is LOST. After the upgrade, openclaw channels list reports "no configured chat channels" (before the upgrade it was "Slack default: enabled, configured, running, connected, health:healthy"). The sandbox's /sandbox/.openclaw/credentials directory is empty. Only the slack egress policy preset survives — the channel itself is gone, so messaging is broken until the user re-adds it.

2. Any user data stored OUTSIDE /sandbox/.openclaw is LOST. A file tree seeded at /sandbox/user-data (a marker plus a random blob) is absent from the recreated sandbox and is not present in the pre-upgrade backup (the backup captures only the .openclaw state: ~13 directories). So this data is unrecoverable — not even via a manual snapshot restore.

Net: the primary GH#6114 symptom (sandbox stuck / all data unreachable until manual rebuild) is fixed, but the legacy-recreate restore does not preserve messaging-channel config or non-.openclaw workspace files. A user upgrading in place silently loses their Slack channel and any files kept outside .openclaw.

Platform scope: Reproduced on Ubuntu 26.04 x86_64 (RTX PRO 6000). Root cause (the upgrade backup/restore scope only covers .openclaw state + egress policy presets) is platform-independent; other platforms (e.g. DGX Spark aarch64) are expected to be affected.

Regression: Not a classic regression — this is a residual gap of the v0.0.81 legacy-recreate recovery that fixed the GH#6114 stuck-Provisioning break; the recovery is incomplete on channel + non-.openclaw data.

Environment

Device:        RTX PRO 6000 test host (x86_64)
OS:            Ubuntu 26.04 LTS
Architecture:  x86_64
NemoClaw:      v0.0.55 -> v0.0.81 (curl|bash in-place upgrade)
OpenShell:     0.0.44 -> 0.0.72 (auto-upgrade during install)
OpenClaw:      2026.5.22 (v0.0.55 sandbox), recovered on v0.0.81
Docker:        29.5.3
GPU:           NVIDIA RTX PRO 6000 Blackwell Server Edition, driver 595.71.05

Steps to Reproduce

1. Install NemoClaw v0.0.55 clean (installer tag v0.0.55, non-interactive, third-party accepted, express skipped), then onboard an OpenClaw sandbox named my-assistant on the nvidia-prod (build) provider. The provider credential is supplied via the standard NVIDIA API key env var; the provider is selected with the standard provider env var.
2. Add a Slack messaging channel to the sandbox and rebuild. Export the Slack app token, bot token, and allowed-user id in the standard channel env vars, then run:

   nemoclaw my-assistant channels add slack
   nemoclaw my-assistant rebuild            # NVIDIA API key exported in the env

   Confirm: nemoclaw my-assistant exec -- openclaw channels status → "Slack default: enabled, configured, running, connected, health:healthy".
3. Seed user data OUTSIDE .openclaw and record integrity — write a marker file plus a random blob under /sandbox/user-data/ and note the blob's sha256:

   nemoclaw my-assistant exec -- bash -lc 'mkdir -p /sandbox/user-data/notes;
     echo MARKER > /sandbox/user-data/marker.txt;
     head -c 20000 /dev/urandom | base64 > /sandbox/user-data/notes/blob.b64;
     sha256sum /sandbox/user-data/notes/blob.b64'

4. Upgrade in place to v0.0.81 with the real user command (installer tag v0.0.81, third-party accepted, non-interactive, experimental OpenShell upgrade accepted). The upgrade halts requiring explicit legacy-recreate confirmation, so re-run it with the confirmation env var NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE set to ["my-assistant"] and the NVIDIA API key exported. The recovery then recreates and restores the sandbox.
5. After the upgrade completes and the sandbox is Ready, check preservation:

   nemoclaw my-assistant exec -- openclaw channels list
   nemoclaw my-assistant exec -- find /sandbox/user-data -type f
   ls ~/.nemoclaw/rebuild-backups/my-assistant/<timestamp>/

Expected Result

The legacy-recreate recovery restores the full pre-upgrade sandbox state — including any configured messaging channels (Slack) and all workspace/user files — so that after the in-place upgrade the Slack channel is still connected and the user's files are intact. An in-place upgrade must not silently drop a configured channel or user data; if some paths are intentionally not preserved, the CLI should warn and require confirmation before discarding them.

Actual Result

Sandbox recovers to Ready (stuck-Provisioning symptom fixed), .openclaw agent state is restored, and the agent runs (inference OK). But:

- openclaw channels list  ->  "no configured chat channels"
  (pre-upgrade it was: "Slack default: enabled, configured, running, connected, healthy")
  /sandbox/.openclaw/credentials is empty. Only the slack egress policy preset survived.
  => Slack messaging is broken after the upgrade; the channel must be re-added.

- find /sandbox/user-data  ->  (nothing; the directory is gone)
  The seeded marker + blob (sha256 f1a75863...a16938) are absent from the recreated
  sandbox AND absent from the pre-upgrade backup (backup contains only .openclaw state
  dirs: agents, canvas, credentials, cron, ...).
  => non-.openclaw user data is silently lost and unrecoverable.

Logs

# Upgrade recovery (restore) log:
  Found pre-upgrade backup for 'my-assistant'; it will be restored after recreation.
  Restoring workspace state from pre-upgrade backup...
  State restored (13 directories, 1 files)
  Policy presets restored: slack, npm, pypi, huggingface, brew, openclaw-pricing
  Deployment verified — gateway and dashboard are healthy.
  OpenClaw is ready
  Existing sandboxes were recovered and upgraded.

# Post-upgrade — Slack channel gone:
  $ openclaw channels list
  - no configured chat channels (run `openclaw channels list --all` to see installable channels)
  $ ls /sandbox/.openclaw/credentials/      # -> empty

# Post-upgrade — non-.openclaw user data gone (live sandbox AND backup):
  $ find /sandbox/user-data -type f          # -> (nothing)
  $ grep -rl UPGRADE-DATA-MARKER ~/.nemoclaw/rebuild-backups   # -> (no match)
  # backup dir top-level: agents canvas credentials cron devices extensions hooks identity ...

# Agent still works after upgrade:
  $ openclaw agent --agent main -m "reply with exactly UPGRADE-SMOKE-OK" --json
  UPGRADE-SMOKE-OK

# Fix direction: the legacy-recreate restore should also re-apply messaging-channel
# credentials/config (so channels survive the upgrade) and should back up + restore the
# full /sandbox workspace, not only .openclaw — or explicitly warn + confirm before
# discarding non-preserved paths.

[yanyunl1991]

Will take a look

[sandl99]

Recommended messaging migration bridge: v0.0.63

@hulynn @yanyunl1991
For the messaging portion of this upgrade, use v0.0.63 as the intermediate compatibility release:

1. Upgrade v0.0.55 -> v0.0.63.
2. While running v0.0.63, re-apply each configured messaging channel so NemoClaw writes the schema-v1 messaging.plan.
3. Rebuild and verify the plan and channel runtime.
4. Only then upgrade v0.0.63 -> v0.0.81.

v0.0.63 is the last release whose SandboxEntry contains both the legacy flat fields (messagingChannels, messagingChannelConfig, and disabledChannels) and the new messaging.plan: https://github.com/NVIDIA/NemoClaw/blob/v0.0.63/src/lib/state/registry.ts#L68-L91

v0.0.64 is the first plan-only release. PR #5123 removed the legacy flat-field and conflict-backfill compatibility path: #5123

Important: merely installing or rebuilding on v0.0.63 is not sufficient for a legacy-only registry row. The v0.0.63 rebuild planner deliberately returns no plan when only messagingChannels exists: https://github.com/NVIDIA/NemoClaw/blob/v0.0.63/src/lib/messaging/compiler/workflow-planner.test.ts#L654-L665

For this Slack case, export the same Slack inputs and refresh the channel while on v0.0.63:

export SLACK_BOT_TOKEN=<redacted>
export SLACK_APP_TOKEN=<redacted>
export SLACK_ALLOWED_USERS=<member-id>
# Optional: export SLACK_ALLOWED_CHANNELS=<channel-id>

nemoclaw my-assistant channels add slack
nemoclaw my-assistant rebuild

Before continuing to v0.0.81, verify that ~/.nemoclaw/sandboxes.json contains:

• .sandboxes["my-assistant"].messaging.schemaVersion == 1
• a slack entry under .messaging.plan.channels
• openclaw channels status reports Slack configured and healthy

This is a good bridge because both v0.0.55 and v0.0.63 still use OpenShell 0.0.44, so the messaging state can be rewritten before the later OpenShell 0.0.72 recreation boundary.

This recommendation addresses only the messaging-schema loss. It does not protect /sandbox/user-data or other paths outside /sandbox/.openclaw; those paths still need an explicit external backup before the final upgrade. Treat this as the recommended QA/workaround path pending a live v0.0.55 -> v0.0.63 -> v0.0.81 validation.

[yanyunl1991]

Live validation of the v0.0.63 messaging bridge (Ubuntu x86_64, build provider)

Ran the full v0.0.55 → v0.0.63 → v0.0.81 path on real hardware to validate the recommended bridge.

Root cause confirmed. On v0.0.55 the sandbox registry row stores the channel in the legacy flat fields only — messagingChannels: ["slack"], messagingChannelConfig: {...} — with no messaging.plan. A direct v0.0.55→v0.0.81 upgrade (plan-only since #5123) therefore has no plan to re-apply, so the channel is dropped and openclaw channels list reports "no configured chat channels".

The bridge concept works — on a clean channel. A fresh v0.0.63 onboard + nemoclaw <sb> channels add slack (non-interactive, real tokens) does write messaging.schemaVersion == 1 + messaging.plan.channels: ["slack"], and openclaw channels status reports running, connected, health:healthy. ✅

But the documented step is insufficient for a v0.0.55→v0.0.63 migrated row. On the migrated sandbox, channels add slack short-circuits (it never prints "Registered slack bridge") because the residual legacy state makes it treat the channel as already configured, so no messaging.plan is written — verified across plain add / remove+add / provider-removal+add / full-remove+add, with real tokens. Notably channels remove does not clear messagingChannelConfig, so the residual persists. To make the bridge work on a migrated row it needs an explicit "fully clear legacy channel state" step, or a fix to channels remove/add so the re-apply is not short-circuited by migrated residue.

/sandbox/user-data still unprotected (confirmed). The pre-upgrade backup manifest scopes to dir: /sandbox/.openclaw only; /sandbox/user-data is not captured (stateDirs are all under .openclaw). Non-.openclaw paths must be backed up externally before the final upgrade.

[hulynn]

QA validation of the recommended v0.0.55 → v0.0.63 → v0.0.81 messaging bridge — the bridge preserves Slack ✅ (user data outside .openclaw still lost ❌)

Ran the recommended bridge live on Ubuntu 26.04 x86_64 (RTX PRO 6000). The bridge works for the messaging channel — Slack survives the final upgrade.

Baseline (v0.0.55, legacy-only row)

• sandboxes.json: messagingChannels: ["slack"] + messagingChannelConfig, and no messaging / messaging.plan (schemaVersion absent) — confirms the legacy-only starting point.
• openclaw channels status → Slack enabled, configured, connected, health:healthy.

v0.0.55 → v0.0.63 (both on OpenShell 0.0.44)

• After the upgrade the row is still legacy-only (messaging.schemaVersion absent), but the Slack channel still works (v0.0.63 still reads the legacy flat fields).
• Re-applied the channel on v0.0.63 (channels add slack + rebuild) → this wrote the schema-v1 plan. Verified the recommended checkpoints:
  • .sandboxes["my-assistant"].messaging.schemaVersion == 1 ✅
  • a slack entry under .messaging.plan.channels ✅
  • openclaw channels status → Slack configured + healthy ✅

v0.0.63 → v0.0.81 (OpenShell 0.0.44 → 0.0.72 recreation boundary)

• messaging.schemaVersion stays 1, 1 channel remains under .messaging.plan.channels ✅
• openclaw channels status → Slack enabled, configured, running, connected, health:healthy ✅ — Slack survived, in contrast to the direct v0.0.55 → v0.0.81 path which drops it.

Data gap (unchanged, as called out in the recommendation): /sandbox/user-data (seeded before the final upgrade) is gone afterward — the bridge does not protect paths outside /sandbox/.openclaw. An explicit external backup is still required.

One extra observation (possibly environmental): on this run the v0.0.63 → v0.0.81 installer's automatic sandbox recovery failed at Rebuild preflight failed: the replacement sandbox image did not build / Failed to recover 'my-assistant': Replacement sandbox image preflight failed, leaving the sandbox at Phase: Unspecified. A manual nemoclaw my-assistant rebuild afterward recovered it to Ready with Slack + the schema-v1 plan intact (the results above are from after that manual rebuild). Flagging in case the auto-recovery image-build preflight is fragile at the OpenShell 0.0.44 → 0.0.72 boundary; it could also be a transient build/network hiccup on our side.

Net: the messaging-schema bridge is a valid workaround for the Slack-channel loss. The /sandbox/user-data loss remains and is what this issue tracks.

[yanyunl1991]

PR #7093 addresses the /sandbox/user-data (non-.openclaw) data-loss facet this issue tracks.

What it does: the pre-upgrade backup taken before a legacy-managed sandbox recreate scopes to the sandbox's recorded managed-state root only (e.g. /sandbox/.openclaw). Files kept elsewhere (e.g. /sandbox/user-data) were dropped by the recreate with no prior notice. The PR prints a warning in the prepared-backup-recovery summary — shown in both upgrade-sandboxes --check and the real recreate, before the destructive step — naming the exact managed-state root that will be restored so users can back up other paths externally first. The warned path is derived per-sandbox from the recovery manifest, so it is accurate across agents (e.g. .openclaw vs .hermes).

Scope: deliberately a warning only; it does not change backup scope or restore behavior. Widening the managed backup to capture arbitrary sandbox paths is a larger lifecycle-ownership decision, left out of this change.

Messaging-channel facet: the Slack-channel loss on the direct v0.0.55 → v0.0.81 path is resolved by the recommended v0.0.63 migration bridge (which writes the schema-v1 messaging.plan that #5123 made authoritative), independently validated live in this thread. This PR therefore does not Fixes that facet.

Verification: confirmed end-to-end on a real host — forced a registered sandbox non-Ready with a validated backup and confirmed the live binary prints the warning in the upgrade-sandboxes --check output; unit tests drive the real recovery path (including the per-agent managed-state root); typecheck:cli clean.