[Ubuntu 22.04][Agent&Skills] every agent turn on a fresh sandbox fails on legacy credential migration, and the doctor --fix it prescribes cannot run while the Gateway is up

Description

On a freshly onboarded sandbox, every openclaw agent turn fails because the agent's auth profile store reports that it needs legacy credential migration. The remedy the product names, openclaw doctor --fix, cannot run while the Gateway is up — and the Gateway is exactly what the agent turn needs — so the two instructions form a closed loop with no documented way out.

The sandbox itself is healthy: it reaches Ready, its inference route answers, and a direct model round-trip through inference.local works. Only the openclaw agent path is blocked.

"Legacy credential migration" on a store created minutes earlier by a fresh onboard also looks wrong on its face — there is no legacy state to migrate from.

Platform scope: Reproduced on Ubuntu 22.04 x86_64 with the native Podman gateway runtime; other platforms and the Docker runtime not tested.
Regression: Unknown — earlier versions not tested for this path.
OpenShell issue: No. The gateway starts and serves; the failure is in OpenClaw's auth profile store and its doctor maintenance gate.

Environment

NemoClaw:      v0.0.128
OpenShell CLI: 0.0.116
OpenClaw:      2026.9.1 (ad6fe23)
Podman:        5.7.1 rootless, cgroups v2
OS:            Ubuntu 22.04.5 LTS, x86_64, 16 vCPU / 62.8 GiB
Provider:      NVIDIA Endpoints (nvidia-prod), nvidia/nemotron-3-super-120b-a12b
Sandbox GPU:   disabled (--no-sandbox-gpu)

Steps to Reproduce

# 1. Onboard a fresh sandbox on the native Podman runtime.
NEMOCLAW_GATEWAY_RUNTIME=podman nemoclaw onboard --fresh --no-sandbox-gpu \
  --name {sandbox} --non-interactive --yes

# 2. Confirm it is healthy.
nemoclaw {sandbox} status          # Phase: Ready, Inference: healthy

# 3. Run one agent turn through the Gateway.
nemoclaw {sandbox} exec -- sh -lc 'openclaw agent --json -m "say OK"'

# 4. Follow the instruction the failure gives.
nemoclaw {sandbox} exec -- sh -lc 'openclaw doctor --fix'

Expected Result

Step 3 returns an agent reply. If a migration genuinely is required on a freshly created store, step 4 performs it, or the message names a remedy that can actually be run.

Actual Result

Step 3:

{
  "ok": false,
  "origin": "gateway",
  "error": {
    "type": "cli_error",
    "message": "Auth profile store $OPENCLAW_HOME/.openclaw/agents/main/agent/openclaw-agent.sqlite requires legacy credential migration; run openclaw doctor --fix. | AUTH_PROFILE_MIGRATION_REQUIRED"
  }
}

Step 4:

┌  OpenClaw doctor
Doctor could not enter maintenance. Stop the Gateway through its service owner, then run
openclaw doctor --fix. StateDatabaseCoordinatorContentionError: another OpenClaw process owns
gateway-lifecycle

So the agent says run doctor, and doctor says stop the Gateway first — while the Gateway is the component serving the agent. Neither message names a supported way to stop and restart it around the migration.

Reproduced on two independently onboarded sandboxes on the same host, with identical output both times. The second sandbox was created after the first was destroyed and its gateway stopped, so this is not leftover state from one bad sandbox.

Meanwhile the sandbox is demonstrably fine:

Phase: Ready
Inference: healthy (https://inference.local/v1/models)
GET https://inference.local/v1/models  ->  200, full model list

Logs

[openclaw] Could not start the CLI.
[openclaw] Reason: Auth profile store $OPENCLAW_HOME/.openclaw/agents/main/agent/openclaw-agent.sqlite
           requires legacy credential migration; run openclaw doctor --fix. | AUTH_PROFILE_MIGRATION_REQUIRED
[openclaw] Try: openclaw doctor

Note the literal $OPENCLAW_HOME in the message — the variable is not expanded, so the path shown to the user is not a path they can act on.

Acceptance criteria

The original fresh-sandbox AUTH_PROFILE_MIGRATION_REQUIRED agent-turn failure is fixed by #12237 and was reverified on v0.0.131. This issue remains open for the separately observed maintenance dead end that the original reproduction exposed:

When a supported runtime path requires repair, the product must not prescribe openclaw doctor --fix unless the user has a supported way to enter maintenance, run the repair, and return the sandbox to Ready.
The supported repair flow must stop or coordinate with the Gateway through its service owner, run doctor --fix without StateDatabaseCoordinatorContentionError, restart or reconcile the Gateway, and preserve user-managed credentials, sessions, and workspace state.
After repair, a normal agent turn must succeed through the Gateway. A healthy inference probe alone is not sufficient evidence.
Verification must record the NemoClaw and OpenClaw versions, sandbox image identity, runtime, doctor --fix exit status, post-repair agent-turn result, and cleanup outcome, without exposing credentials.
Close this issue only after both the already-verified auth-profile regression and this supported maintenance flow pass. If the maintenance flow is intentionally unsupported, remove the unusable prescription and provide actionable recovery guidance instead.

Independently reproduced this on a second host with a different container driver (Docker gateway runtime, Ubuntu 26.04) — same symptom, same root-cause shape: fresh onboard reaches Ready with a healthy inference route, openclaw doctor immediately flags an "Auth profile SQLite migration" / "Secret runtime degradation" finding for openclaw-agent.sqlite, openclaw secrets reload doesn't clear it, and openclaw doctor --fix fails with the same Gateway-lifecycle contention error.

Filed as NVB#6826525 (cross-referenced back here).

I've had the same issue since upgrading to v0.0.128

fix(openclaw): avoid legacy auth profiles on managed routes (NVIDIA#1…

Regression reproduced on NemoClaw v0.0.129 with the Docker runtime during release sanity. A freshly onboarded OpenClaw sandbox reaches Ready and the inference smoke succeeds, but its first TUI agent turn fails with AUTH_PROFILE_MIGRATION_REQUIRED and instructs the user to run openclaw doctor --fix. This is the same signature as the original report, now on Docker rather than only Podman. Evidence: https://gitlab-master.nvidia.com/cloud-service-qa/nemoclaw/nemoclaw-test/-/jobs/459001588

Reproduced a related variant on DGX Spark (aarch64), Docker runtime, NemoClaw v0.0.129 / OpenClaw 2026.9.1 (ad6fe23) — but failing on the session store rather than the auth profile store.

Setup: self-managed local vLLM (Nemotron 3.5 Lightning) via the local vLLM provider, Telegram channel. Reproduced on a fully fresh install (destroy --yes --cleanup-gateway, then onboard --fresh), before any workspace restore.

Every turn routed to the main session fails:

lane task error: lane=session:agent:main:main error="Persisted legacy session transcripts require doctor/import migration before runtime use"
Embedded agent failed before reply: Persisted legacy session transcripts require doctor/import migration before runtime use

Likely why it differs from the auth variant: local vLLM needs no credentials, so the turn never touches the auth profile store and trips on the session store instead. Same shape — fresh 9.1 state read as legacy — in a different subsystem.

It is scoped to the main session. On the same sandbox:

Session key	Result
`agent:main:main`	fails — Telegram DMs, default webchat thread, `cron-nested` lane
`agent:main:dashboard-…`	works normally

Workaround for anyone blocked: starting a new conversation in the dashboard creates a separate session that works. Channels that route DMs into the main session (Telegram) stay unusable, and the generator emits no session.dmScope to route around it.

Nothing to migrate by OpenClaw's own measure:

session-sqlite inspect: 1 target(s), 0 legacy entries, 1 sqlite entries, 0 issue(s)
session-sqlite dry-run: 0 target(s), 0 legacy entries, 0 sqlite entries, 0 issue(s)

No *.jsonl exists anywhere under /sandbox/.openclaw.

Same closed loop on every repair path: doctor --fix and doctor --session-sqlite recover both refuse while the Gateway owns the state directory; openclaw sessions delete --agent main --yes agent:main:main returns Cannot delete the main session; and nemoclaw <sandbox> gateway is not a command, so there is no supported stop.

Doctor also prints Removed retired agents.entries.*.default markers on every run. It never persists because the config is sealed, and the generated openclaw.json carries "default": true on the main agent entry.

Possibly relevant upstream: openclaw/openclaw#150110 (keep legacy session repair out of startup) merged 2026-09-16 and is not in 2026.9.1 — verified by ancestry, it first ships in 2026.9.5. Worth checking against the session variant when qualifying the next pin.

@wangericnv

Investigation: the original managed-inference auth failure is addressed by #12237, which is included in v0.0.130 and v0.0.131. The v0.0.129 tag predates that fix by seven commits, so the v0.0.129 reproduction that prompted reopening did not test the fix.

NemoClaw startup generated a legacy auth-profiles.json even for managed inference.local routes, where OpenShell already handles authentication. OpenClaw rejected that profile and prescribed Doctor, which could not acquire maintenance ownership while the Gateway was running. #12237 prevents the unnecessary profile, clears NVIDIA credential aliases before child processes start, and removes only the selected provider's matching NemoClaw-generated legacy entry while preserving user-managed profiles.

Validation for this investigation:

Verified release ancestry and the startup implementation in the release tags.
Ran 18 isolated component probes against v0.0.129, v0.0.130, and v0.0.131. These reproduced the old profile creation and verified the fixed managed-route behavior, user-profile preservation, direct-route behavior, and rejection of unsafe files.
Confirmed existing PR evidence includes successful native agent turns on Docker and Podman. Those are earlier PR results, not a new release retest; the overall workflow also had failed jobs.
No fresh sandbox E2E was run during this investigation. The repository Vitest suite could not run because checkout dependencies were unavailable.

The separate session-store variant is tracked in #12549. Its reporter confirmed resolution after upgrading to v0.0.130 and closed that issue.

@wangericnv, please retest the original reproduction using a v0.0.131 sandbox image, recording the NemoClaw/OpenClaw versions, sandbox image identity, and the actual agent-turn result. A healthy inference probe alone does not verify this failure. If the agent turn succeeds without AUTH_PROFILE_MIGRATION_REQUIRED, this issue can close as fixed in v0.0.130. If it still fails, please attach the redacted error and those runtime identities so we can distinguish a remaining defect from an older sandbox image.

QA retest on v0.0.131, following the report's own Steps to Reproduce. The original failure does not reproduce.

Runtime identities

NemoClaw:      v0.0.131
OpenShell CLI: openshell 0.0.116
OpenClaw:      2026.9.2 (3928bad)
Podman:        5.7.1 rootless, cgroups v2
OS:            Ubuntu 22.04.5 LTS, x86_64, 16 vCPU / 62.8 GiB
Provider:      NVIDIA Endpoints (nvidia-prod), nvidia/nemotron-3-super-120b-a12b
Sandbox GPU:   disabled (--no-sandbox-gpu)

Sandbox image identity:

ghcr.io/nvidia/nemoclaw/openclaw-sandbox@sha256:a061adb546d9a17f106bfbc1864fdda54f268abd145868ca5d8fa39aba4ec9c7

The sandbox was created fresh on this build — onboard --fresh on a host with no
registered sandboxes — so this is not an older sandbox image.

Step 2 — status

Phase: Ready
Agent:     OpenClaw v2026.9.2
Model:     nvidia/nemotron-3-super-120b-a12b
Provider:  nvidia-prod
Inference: healthy (https://inference.local/v1/models)

Step 3 — the agent turn

This is the part that previously failed. It now succeeds, exit code 0:

{
  "runId": "4ed528f8-2e6b-49a6-8cd0-d5c376f0dbe7",
  "status": "ok",
  "result": {
    "payloads": [ { "text": "OK", "mediaUrl": null } ],
    "meta": {
      "durationMs": 2749,
      "agentMeta": {
        "provider": "inference",
        "model": "nvidia/nemotron-3-super-120b-a12b",
        "agentHarnessId": "openclaw"
      }
    }
  }
}

A second turn on the same sandbox also returned normally ("status": "ok",
"text": "READY"). AUTH_PROFILE_MIGRATION_REQUIRED occurs 0 times across
both runs.

The mechanism #12237 changed is confirmed gone

The legacy profile that the agent used to reject is simply not created any more.
Both candidate locations are absent inside the sandbox:

$OPENCLAW_HOME/.openclaw/auth-profiles.json                     absent
$OPENCLAW_HOME/.openclaw/agents/main/agent/auth-profiles.json   absent

The agent store directory holds only the SQLite database and its WAL/SHM
companions.

One honest caveat — the second half of the report still reproduces

Step 4 of the original report was openclaw doctor --fix, and that half has not
changed. Run against this healthy sandbox it still exits 1:

┌  OpenClaw doctor
Doctor could not enter maintenance. Stop the Gateway through its service owner,
then run openclaw doctor --fix. StateDatabaseCoordinatorContentionError:
another OpenClaw process owns gateway-lifecycle

So Doctor is still unable to enter maintenance while the Gateway is running, and
still names no supported way to stop and restart it around a migration. What has
changed is that nothing prescribes Doctor on this path any more, because the
profile that triggered the prescription is never written. The dead end is
therefore unreachable through this reproduction, but it is not fixed — if any
other path ever prescribes doctor --fix on a running Gateway, the same loop
returns. Whether that deserves its own issue is a maintainer call; I did not open
one.

Verdict

The reported failure — every agent turn failing with
AUTH_PROFILE_MIGRATION_REQUIRED — is fixed, consistent with #12237 landing
in v0.0.130. From this reproduction's point of view this issue can close as fixed
in v0.0.130.

Navigation Menu

Repository navigation

[Ubuntu 22.04][Agent&Skills] every agent turn on a fresh sandbox fails on legacy credential migration, and the doctor --fix it prescribes cannot run while the Gateway is up #12254

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Logs

Related

Acceptance criteria

Runtime identities

Step 2 — status

Step 3 — the agent turn

The mechanism #12237 changed is confirmed gone

One honest caveat — the second half of the report still reproduces

Verdict

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Ubuntu 22.04][Agent&Skills] every agent turn on a fresh sandbox fails on legacy credential migration, and the doctor --fix it prescribes cannot run while the Gateway is up #12254

Description

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Logs

Related

Acceptance criteria

Activity

lyral-commits commented on Sep 23, 2026

oidebrett commented on Sep 23, 2026

mercl-lau commented on Sep 28, 2026

convexchaos commented on Oct 1, 2026

rsliter commented on Oct 7, 2026

wangericnv commented on Oct 8, 2026

Runtime identities

Step 2 — status

Step 3 — the agent turn

The mechanism #12237 changed is confirmed gone

One honest caveat — the second half of the report still reproduces

Verdict

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions