From 6128b82ffd1260d247e9fcfc370649d7f953cf08 Mon Sep 17 00:00:00 2001 From: Miyoung Choi Date: Fri, 17 Apr 2026 12:19:31 -0700 Subject: [PATCH] docs: catch up documentation for v0.0.18 changes Refresh user-facing docs against the 34 commits merged between v0.0.17 and v0.0.18. Highlights: - Replace the Ollama 0.0.0.0 binding guidance with the new authenticated reverse proxy on 127.0.0.1:11435 (#1922). - Document the compatible-endpoint provider defaulting to /v1/chat/completions and the NEMOCLAW_PREFERRED_API=openai-responses opt-in (#1984). - Add the new nemoclaw upgrade-sandboxes command with --check, --auto, and --yes flags (#1943). - Note the cross-sandbox messaging overlap warning and 409 detection in nemoclaw status (#1953). - Document the messaging-token rotation auto-rebuild flow (#1967). - Cover new troubleshooting entries for the Ollama auth proxy, IPv6 localhost resolution, orphan SSH port-forward cleanup on re-onboard, and rotated messaging credentials (#1978, #1950). - Note tar failure exit code for nemoclaw debug --output (#1770) and the orphaned openshell process cleanup in nemoclaw uninstall (#1940). Also: - Extend docs/.docs-skip to exclude the experimental sandbox-mgmt shields and config commands (#1976). - Fix a sphinx-autobuild infinite rebuild loop in docs/conf.py by writing docs/project.json only when its contents change. - Bump the docs version switcher preferred entry to 0.0.18. - Regenerate nemoclaw-user-* agent skills from docs/. Signed-off-by: Miyoung Choi Made-with: Cursor --- .../SKILL.md | 77 ++++++++++------ .../nemoclaw-user-deploy-remote/SKILL.md | 8 ++ .../references/commands.md | 27 ++++++ .../references/troubleshooting.md | 91 +++++++++++++++---- docs/.docs-skip | 16 ++++ docs/conf.py | 6 +- docs/deployment/set-up-telegram-bridge.md | 8 ++ docs/inference/switch-inference-providers.md | 8 +- docs/inference/use-local-inference.md | 69 ++++++++------ docs/project.json | 2 +- docs/reference/commands.md | 27 ++++++ docs/reference/troubleshooting.md | 91 +++++++++++++++---- docs/versions1.json | 4 + 13 files changed, 338 insertions(+), 96 deletions(-) diff --git a/.agents/skills/nemoclaw-user-configure-inference/SKILL.md b/.agents/skills/nemoclaw-user-configure-inference/SKILL.md index 22f79a7c53e..46067f73f52 100644 --- a/.agents/skills/nemoclaw-user-configure-inference/SKILL.md +++ b/.agents/skills/nemoclaw-user-configure-inference/SKILL.md @@ -143,11 +143,13 @@ Select the same provider and endpoint again. The updated streaming probe will detect incomplete `/v1/responses` support and select `/v1/chat/completions` automatically. -To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API` -before onboarding: +For the compatible-endpoint provider, NemoClaw uses `/v1/chat/completions` by +default, so no env var is required to keep the safe path. +To opt in to `/v1/responses` for a backend you have verified end to end, set +`NEMOCLAW_PREFERRED_API` before onboarding: ```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard +$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard ``` > **Note:** `NEMOCLAW_INFERENCE_API_OVERRIDE` patches the config at container startup but @@ -233,19 +235,35 @@ Select **Local Ollama** from the provider list. NemoClaw lists installed models or offers starter models if none are installed. It pulls the selected model, loads it into memory, and validates it before continuing. -### Linux with Docker +### Authenticated Reverse Proxy -On Linux hosts that run NemoClaw with Docker, the sandbox reaches Ollama through -`http://host.openshell.internal:11434`, not the host shell's `localhost` socket. -If Ollama is already running, make sure it listens on `0.0.0.0:11434` instead of -`127.0.0.1:11434`. +NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated +reverse proxy on `0.0.0.0:11435`. +Containers and other hosts on the local network reach Ollama only through the +proxy, which validates a Bearer token before forwarding requests. +Ollama itself is never exposed without authentication. -```console -$ OLLAMA_HOST=0.0.0.0:11434 ollama serve -``` +The onboard wizard manages the proxy automatically: + +- Generates a random 24-byte token on first run and stores it in + `~/.nemoclaw/ollama-proxy-token` with `0600` permissions. +- Starts the proxy after Ollama and verifies it before continuing. +- Cleans up stale proxy processes from previous runs. +- Reuses the persisted token after a host reboot so you do not need to re-run + onboard. + +The sandbox provider is configured to use proxy port `11435` with the generated +token as its `OPENAI_API_KEY` credential. +OpenShell's L7 proxy injects the token at egress, so the agent inside the +sandbox never sees the token directly. + +`GET /api/tags` is exempt from authentication so container health checks +continue to work. +All other endpoints (including `POST /api/tags`) require the Bearer token. -If Ollama only binds loopback, NemoClaw can detect it on the host, but the -sandbox-side validation step fails because containers cannot reach it. +If Ollama is already running on a non-loopback address when you start onboard, +the wizard restarts it on `127.0.0.1:11434` so the proxy is the only network +path to the model server. ### Non-Interactive Setup @@ -265,8 +283,9 @@ If `NEMOCLAW_MODEL` is not set, NemoClaw selects a default model based on availa ## Step 6: OpenAI-Compatible Server This option works with any server that implements `/v1/chat/completions`, including vLLM, TensorRT-LLM, llama.cpp, LocalAI, and others. -If the server also supports `/v1/responses`, NemoClaw only favors that path when onboarding can verify tool-calling behavior that matches what OpenClaw actually sends. -Otherwise NemoClaw falls back to `/v1/chat/completions`. +For compatible endpoints, NemoClaw uses `/v1/chat/completions` by default. +This avoids a class of failures where local backends accept `/v1/responses` requests but silently drop the system prompt and tool definitions. +To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API=openai-responses` before running onboard. Start your model server. The examples below use vLLM, but any OpenAI-compatible server works. @@ -288,8 +307,8 @@ The wizard prompts for an API key. If your server does not require authentication, enter any non-empty string (for example, `dummy`). NemoClaw validates the endpoint by sending a test inference request before continuing. -For OpenAI-compatible endpoints, the validation prefers `/responses` only when the probe produces a compatible function or tool call. -Endpoints that return `200 OK` on `/responses` but do not format tool calls the way OpenClaw expects are configured to use `/chat/completions` instead. +The wizard probes `/v1/chat/completions` by default for the compatible-endpoint provider. +If you set `NEMOCLAW_PREFERRED_API=openai-responses`, NemoClaw probes `/v1/responses` instead and only selects it when the response includes the streaming events OpenClaw requires. ### Non-Interactive Setup @@ -310,27 +329,27 @@ $ NEMOCLAW_PROVIDER=custom \ | `NEMOCLAW_MODEL` | Model ID as reported by the server. | | `COMPATIBLE_API_KEY` | API key for the endpoint. Use any non-empty value if authentication is not required. | -### Forcing Chat Completions API +### Selecting the API Path -Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but do -not emit the granular streaming events that OpenClaw requires. -NemoClaw tests streaming events during onboarding and falls back to -`/v1/chat/completions` automatically when it detects incomplete streaming. +For the compatible-endpoint provider, `/v1/chat/completions` is the default. +NemoClaw tests streaming events during onboarding and uses chat completions +without probing the Responses API. -If you need to bypass the `/v1/responses` probe entirely, set -`NEMOCLAW_PREFERRED_API` before running onboard: +To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API` before running onboard: ```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard +$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard ``` -Set this variable to make the wizard skip the `/v1/responses` probe and use -`/v1/chat/completions` directly. -You can use it in both interactive and non-interactive mode. +The wizard then probes `/v1/responses` and only selects it when streaming +support is complete. +If the probe fails, the wizard falls back to `/v1/chat/completions` +automatically. +You can use this variable in both interactive and non-interactive mode. | Variable | Values | Default | |---|---|---| -| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `chat-completions` | unset (auto-detect) | +| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `openai-responses` | `openai-completions` for compatible endpoints | If you already onboarded and the sandbox is failing at runtime, re-run `nemoclaw onboard` to re-probe the endpoint and bake the correct API path diff --git a/.agents/skills/nemoclaw-user-deploy-remote/SKILL.md b/.agents/skills/nemoclaw-user-deploy-remote/SKILL.md index 486cb70da09..025268d78ea 100644 --- a/.agents/skills/nemoclaw-user-deploy-remote/SKILL.md +++ b/.agents/skills/nemoclaw-user-deploy-remote/SKILL.md @@ -179,6 +179,14 @@ Channel entries in `/sandbox/.openclaw/openclaw.json` are fixed at image build t If you add or change `TELEGRAM_BOT_TOKEN` (or toggle channels) after a sandbox already exists, you typically need to run `nemoclaw onboard` again so the image and provider attachments are rebuilt with the new settings. +NemoClaw stores a SHA-256 hash of each messaging token in the sandbox registry at creation time. +When you re-run `nemoclaw onboard --non-interactive` with a new token, NemoClaw detects the change, backs up workspace state, deletes the sandbox, recreates it with the new credential, and restores the backup. +This makes credential rotation safe to script. + +Telegram, Discord, and Slack each allow only one active consumer per bot token. +If you enable a messaging channel and another sandbox already uses the same token, onboard prompts you to confirm before continuing in interactive mode and exits non-zero in non-interactive mode. +`nemoclaw status` also reports cross-sandbox overlaps so you can resolve duplicates before messages start dropping. + For a full first-time flow, refer to Quickstart (see the `nemoclaw-user-get-started` skill). ## Step 12: Confirm Delivery diff --git a/.agents/skills/nemoclaw-user-reference/references/commands.md b/.agents/skills/nemoclaw-user-reference/references/commands.md index dd93fc1d14a..7f8e9c5fc74 100644 --- a/.agents/skills/nemoclaw-user-reference/references/commands.md +++ b/.agents/skills/nemoclaw-user-reference/references/commands.md @@ -186,6 +186,9 @@ If the backend is down, the output includes an `Inference: unreachable` line wit The Policy section displays the live enforced policy (fetched via `openshell policy get --full`), which reflects presets added or removed after sandbox creation. If the sandbox is running an outdated agent version, the output includes an `Update` line with the available version and a `nemoclaw rebuild` hint. +When other sandboxes have the same messaging channel enabled (Telegram, Discord, or Slack) with the same bot token, the output includes a cross-sandbox overlap warning so you can resolve the conflict before messages start dropping. +The command also tails `/tmp/gateway.log` inside the default sandbox and flags Telegram `409 Conflict` errors that indicate a duplicate consumer for the bot token. + ```console $ nemoclaw my-assistant status ``` @@ -292,6 +295,25 @@ $ nemoclaw my-assistant rebuild [--yes] [--verbose] The sandbox must be running for the backup step to succeed. After restore, the command runs `openclaw doctor --fix` for cross-version structure repair. +### `nemoclaw upgrade-sandboxes` + +Rebuild sandboxes whose base image is older than the one currently pinned by NemoClaw. +NemoClaw resolves the digest of `ghcr.io/nvidia/nemoclaw/sandbox-base:latest` from the registry, then compares it against the digest each sandbox was created with. +Sandboxes that match the current digest are left alone. + +```console +$ nemoclaw upgrade-sandboxes [--check] [--auto] [--yes] +``` + +| Flag | Description | +|------|-------------| +| `--check` | List stale sandboxes without rebuilding any of them. Exits non-zero if any are stale. | +| `--auto` | Rebuild every stale sandbox without prompting. Used by the installer to upgrade in place. | +| `--yes` | Skip the confirmation prompt for the rebuild plan. | + +Each rebuild reuses the same workspace backup-and-restore flow as `nemoclaw rebuild`, so workspace files survive the upgrade. +If the registry is unreachable (offline or firewalled hosts), NemoClaw falls back to the unpinned `:latest` tag and reports that the digest could not be resolved instead of failing. + ### `nemoclaw backup-all` Back up all registered running sandboxes to `~/.nemoclaw/rebuild-backups/`. @@ -395,6 +417,8 @@ $ nemoclaw debug [--quick] [--sandbox NAME] [--output PATH] | `--sandbox NAME` | Target a specific sandbox (default: auto-detect) | | `--output PATH` | Write diagnostics tarball to the given path | +If `--output` is set and the tarball cannot be written (for example, the destination directory is missing or read-only), the command exits non-zero so scripts can detect the failure. + ### `nemoclaw credentials list` List the names of all credentials stored in `~/.nemoclaw/credentials.json`. @@ -422,6 +446,9 @@ $ nemoclaw credentials reset NVIDIA_API_KEY Run `uninstall.sh` to remove NemoClaw sandboxes, gateway resources, related images and containers, and local state. The CLI uses the local `uninstall.sh` first and falls back to the hosted script if the local file is unavailable. +Uninstall also stops any orphaned `openshell` host processes left behind by previous onboard or destroy cycles, including `openshell sandbox create`, `openshell ssh-proxy`, and SSH sessions spawned by OpenShell. +Earlier releases only stopped `openshell forward` processes, so those orphans accumulated across runs. + | Flag | Effect | |---|---| | `--yes` | Skip the confirmation prompt | diff --git a/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md b/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md index ef224b30d32..caa146c90ad 100644 --- a/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md +++ b/.agents/skills/nemoclaw-user-reference/references/troubleshooting.md @@ -216,6 +216,43 @@ If neither is found, verify that Colima is running: $ colima status ``` +### Re-onboard fails because port 18789 is held by SSH + +After destroying a sandbox and gateway, the SSH port-forward process for the +dashboard can be left running. +Re-running onboard then fails preflight with `Port 18789 is not available. +Blocked by: ssh`. + +Current NemoClaw detects this case and kills the orphaned SSH process +automatically before retrying the port check. +If you see the error on an older release, identify the SSH process and +terminate it manually: + +```console +$ sudo lsof -i :18789 +$ kill +``` + +Then re-run `nemoclaw onboard`. + +### Updated messaging token is not picked up + +Re-running `nemoclaw onboard --non-interactive` with a new +`TELEGRAM_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, or `SLACK_BOT_TOKEN` previously +reported success while the sandbox kept polling with the old credential. +Current NemoClaw stores SHA-256 hashes of messaging credentials in the +sandbox registry at creation time and detects when a token has changed. +When rotation is detected, NemoClaw automatically backs up workspace state, +deletes the sandbox, recreates it with the new credential, and restores the +backup. + +If you suspect a sandbox is still using a stale token, re-run onboarding so +the credential check runs: + +```console +$ nemoclaw onboard --non-interactive +``` + ### Sandbox creation killed by OOM (exit 137) On systems with 8 GB RAM or less and no swap configured, the sandbox image push can exhaust available memory and get killed by the Linux OOM killer (exit code 137). @@ -338,25 +375,23 @@ $ nemoclaw onboard ### Agent fails at runtime after onboarding succeeds with a compatible endpoint -Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` and pass -the onboarding validation probe, but their streaming mode is incomplete. +Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their +streaming mode is incomplete. OpenClaw requires granular streaming events like `response.output_text.delta` that these backends do not emit. -NemoClaw now tests streaming events during the `/v1/responses` probe and falls -back to `/v1/chat/completions` automatically. -If you onboarded before this check was added, re-run onboarding so the wizard -re-probes the endpoint and bakes the correct API path into the image: +For the compatible-endpoint provider, NemoClaw now defaults to +`/v1/chat/completions` and skips the Responses API probe entirely unless you +opt in. +If you onboarded an older release that selected `/v1/responses`, re-run +onboarding so the wizard rebuilds the image with chat completions: ```console $ nemoclaw onboard ``` -To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API`: - -```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard -``` +If you previously set `NEMOCLAW_PREFERRED_API=openai-responses` to force the +Responses API, unset it before re-running onboard. Do not rely on `NEMOCLAW_INFERENCE_API_OVERRIDE` alone — it patches the config at container startup but does not update the Dockerfile ARG baked into the @@ -499,16 +534,36 @@ $ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard If you need to run multiple sandboxes at different ports at the same time, see [Running multiple sandboxes simultaneously](#running-multiple-sandboxes-simultaneously). -### Ollama network exposure warning during onboard +### Ollama auth proxy did not start + +NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated +reverse proxy on `0.0.0.0:11435` so the sandbox can reach Ollama without +exposing it to the local network. +If the proxy fails to start, onboarding exits before configuring inference. + +Check whether the proxy port is occupied by another process: + +```console +$ sudo lsof -i :11435 +``` + +Stop the conflicting process and re-run `nemoclaw onboard`. +The wizard cleans up stale proxy processes from previous runs automatically, +so most failures resolve by retrying. -When you select a local Ollama provider, onboarding binds Ollama to `0.0.0.0` so the Docker sandbox can reach the host. -This exposes the unauthenticated Ollama API to the local network. -NemoClaw prints a warning during onboarding to alert you to this risk. +The proxy token is persisted to `~/.nemoclaw/ollama-proxy-token` with `0600` +permissions. +If the file is missing or unreadable after a host reboot, re-running +`nemoclaw onboard` regenerates it. -On trusted private networks, the warning is informational. -On shared or public networks (airports, coffee shops), any adjacent device can send prompts to and enumerate models on your Ollama instance. +### Local inference health check resolves to IPv6 -The warning is suppressed on WSL, where Ollama binds to `127.0.0.1` instead. +Local inference health checks now use `127.0.0.1` instead of `localhost`. +On systems where `localhost` resolves to `::1` first, older NemoClaw releases +could probe the wrong address and report the local backend as unreachable +even when it was running. +If you see this on a current NemoClaw release, verify that the local backend +binds an IPv4 address and not only `::1`. ### Blueprint run failed diff --git a/docs/.docs-skip b/docs/.docs-skip index 5da76cd4b02..769edaf4f14 100644 --- a/docs/.docs-skip +++ b/docs/.docs-skip @@ -30,8 +30,24 @@ skip-features: - "--dangerously-skip-permissions" # experimental flag, pending UX review - "openclaw-sandbox-permissive.yaml" # permissive policy file, same feature + - "shields and config commands" # experimental sandbox-mgmt commands (#1976), not advertised + - "src/lib/shields.ts" # shields implementation, experimental + - "src/lib/shields-timer.ts" # shields auto-restore timer, experimental + - "src/lib/shields-audit.ts" # shields audit trail, experimental + - "src/lib/sandbox-config.ts" # config get/set/rotate-token, experimental + - "test/shields" # shields tests, experimental + - "test/config-set.test.ts" # config set tests, experimental + - "test/config-rotate-token.test.ts" # config rotate-token tests, experimental + - "test/e2e/test-shields-config.sh" # shields/config E2E, experimental + - "shields-status" # plugin shields-status command, experimental + - "config-show" # plugin config-show command, experimental skip-terms: - "dangerously-skip-permissions" # do not mention this flag anywhere in docs - "permissive mode" # do not reference this concept in docs - "Hermes" + - "shields down" # experimental shields command, not advertised + - "shields up" # experimental shields command, not advertised + - "shields status" # experimental shields command, not advertised + - "config rotate-token" # experimental config command, not advertised + - "rotate-token" # experimental config command, not advertised diff --git a/docs/conf.py b/docs/conf.py index 562c23302e4..30e6e7701d7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -139,5 +139,9 @@ # Keep project.json in sync with the resolved release version so the # static copy served alongside the docs always reports the correct version. +# Write only when the contents change so sphinx-autobuild does not detect +# a self-induced source change and rebuild in an infinite loop. _project_json = Path(__file__).parent / "project.json" -_project_json.write_text(json.dumps({"name": "nemoclaw", "version": release}) + "\n") +_project_json_contents = json.dumps({"name": "nemoclaw", "version": release}) + "\n" +if not _project_json.exists() or _project_json.read_text() != _project_json_contents: + _project_json.write_text(_project_json_contents) diff --git a/docs/deployment/set-up-telegram-bridge.md b/docs/deployment/set-up-telegram-bridge.md index 5e3345b8e19..d13993a10ba 100644 --- a/docs/deployment/set-up-telegram-bridge.md +++ b/docs/deployment/set-up-telegram-bridge.md @@ -71,6 +71,14 @@ Channel entries in `/sandbox/.openclaw/openclaw.json` are fixed at image build t If you add or change `TELEGRAM_BOT_TOKEN` (or toggle channels) after a sandbox already exists, you typically need to run `nemoclaw onboard` again so the image and provider attachments are rebuilt with the new settings. +NemoClaw stores a SHA-256 hash of each messaging token in the sandbox registry at creation time. +When you re-run `nemoclaw onboard --non-interactive` with a new token, NemoClaw detects the change, backs up workspace state, deletes the sandbox, recreates it with the new credential, and restores the backup. +This makes credential rotation safe to script. + +Telegram, Discord, and Slack each allow only one active consumer per bot token. +If you enable a messaging channel and another sandbox already uses the same token, onboard prompts you to confirm before continuing in interactive mode and exits non-zero in non-interactive mode. +`nemoclaw status` also reports cross-sandbox overlaps so you can resolve duplicates before messages start dropping. + For a full first-time flow, refer to [Quickstart](../get-started/quickstart.md). ## Confirm Delivery diff --git a/docs/inference/switch-inference-providers.md b/docs/inference/switch-inference-providers.md index 2ac5df5ad3a..8c3014a1724 100644 --- a/docs/inference/switch-inference-providers.md +++ b/docs/inference/switch-inference-providers.md @@ -88,11 +88,13 @@ Select the same provider and endpoint again. The updated streaming probe will detect incomplete `/v1/responses` support and select `/v1/chat/completions` automatically. -To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API` -before onboarding: +For the compatible-endpoint provider, NemoClaw uses `/v1/chat/completions` by +default, so no env var is required to keep the safe path. +To opt in to `/v1/responses` for a backend you have verified end to end, set +`NEMOCLAW_PREFERRED_API` before onboarding: ```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard +$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard ``` :::{note} diff --git a/docs/inference/use-local-inference.md b/docs/inference/use-local-inference.md index 037e0471b5a..242580559a1 100644 --- a/docs/inference/use-local-inference.md +++ b/docs/inference/use-local-inference.md @@ -53,19 +53,35 @@ Select **Local Ollama** from the provider list. NemoClaw lists installed models or offers starter models if none are installed. It pulls the selected model, loads it into memory, and validates it before continuing. -### Linux with Docker +### Authenticated Reverse Proxy -On Linux hosts that run NemoClaw with Docker, the sandbox reaches Ollama through -`http://host.openshell.internal:11434`, not the host shell's `localhost` socket. -If Ollama is already running, make sure it listens on `0.0.0.0:11434` instead of -`127.0.0.1:11434`. +NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated +reverse proxy on `0.0.0.0:11435`. +Containers and other hosts on the local network reach Ollama only through the +proxy, which validates a Bearer token before forwarding requests. +Ollama itself is never exposed without authentication. -```console -$ OLLAMA_HOST=0.0.0.0:11434 ollama serve -``` +The onboard wizard manages the proxy automatically: + +- Generates a random 24-byte token on first run and stores it in + `~/.nemoclaw/ollama-proxy-token` with `0600` permissions. +- Starts the proxy after Ollama and verifies it before continuing. +- Cleans up stale proxy processes from previous runs. +- Reuses the persisted token after a host reboot so you do not need to re-run + onboard. + +The sandbox provider is configured to use proxy port `11435` with the generated +token as its `OPENAI_API_KEY` credential. +OpenShell's L7 proxy injects the token at egress, so the agent inside the +sandbox never sees the token directly. + +`GET /api/tags` is exempt from authentication so container health checks +continue to work. +All other endpoints (including `POST /api/tags`) require the Bearer token. -If Ollama only binds loopback, NemoClaw can detect it on the host, but the -sandbox-side validation step fails because containers cannot reach it. +If Ollama is already running on a non-loopback address when you start onboard, +the wizard restarts it on `127.0.0.1:11434` so the proxy is the only network +path to the model server. ### Non-Interactive Setup @@ -85,8 +101,9 @@ If `NEMOCLAW_MODEL` is not set, NemoClaw selects a default model based on availa ## OpenAI-Compatible Server This option works with any server that implements `/v1/chat/completions`, including vLLM, TensorRT-LLM, llama.cpp, LocalAI, and others. -If the server also supports `/v1/responses`, NemoClaw only favors that path when onboarding can verify tool-calling behavior that matches what OpenClaw actually sends. -Otherwise NemoClaw falls back to `/v1/chat/completions`. +For compatible endpoints, NemoClaw uses `/v1/chat/completions` by default. +This avoids a class of failures where local backends accept `/v1/responses` requests but silently drop the system prompt and tool definitions. +To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API=openai-responses` before running onboard. Start your model server. The examples below use vLLM, but any OpenAI-compatible server works. @@ -108,8 +125,8 @@ The wizard prompts for an API key. If your server does not require authentication, enter any non-empty string (for example, `dummy`). NemoClaw validates the endpoint by sending a test inference request before continuing. -For OpenAI-compatible endpoints, the validation prefers `/responses` only when the probe produces a compatible function or tool call. -Endpoints that return `200 OK` on `/responses` but do not format tool calls the way OpenClaw expects are configured to use `/chat/completions` instead. +The wizard probes `/v1/chat/completions` by default for the compatible-endpoint provider. +If you set `NEMOCLAW_PREFERRED_API=openai-responses`, NemoClaw probes `/v1/responses` instead and only selects it when the response includes the streaming events OpenClaw requires. ### Non-Interactive Setup @@ -130,27 +147,27 @@ $ NEMOCLAW_PROVIDER=custom \ | `NEMOCLAW_MODEL` | Model ID as reported by the server. | | `COMPATIBLE_API_KEY` | API key for the endpoint. Use any non-empty value if authentication is not required. | -### Forcing Chat Completions API +### Selecting the API Path -Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but do -not emit the granular streaming events that OpenClaw requires. -NemoClaw tests streaming events during onboarding and falls back to -`/v1/chat/completions` automatically when it detects incomplete streaming. +For the compatible-endpoint provider, `/v1/chat/completions` is the default. +NemoClaw tests streaming events during onboarding and uses chat completions +without probing the Responses API. -If you need to bypass the `/v1/responses` probe entirely, set -`NEMOCLAW_PREFERRED_API` before running onboard: +To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API` before running onboard: ```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard +$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard ``` -Set this variable to make the wizard skip the `/v1/responses` probe and use -`/v1/chat/completions` directly. -You can use it in both interactive and non-interactive mode. +The wizard then probes `/v1/responses` and only selects it when streaming +support is complete. +If the probe fails, the wizard falls back to `/v1/chat/completions` +automatically. +You can use this variable in both interactive and non-interactive mode. | Variable | Values | Default | |---|---|---| -| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `chat-completions` | unset (auto-detect) | +| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `openai-responses` | `openai-completions` for compatible endpoints | If you already onboarded and the sandbox is failing at runtime, re-run `nemoclaw onboard` to re-probe the endpoint and bake the correct API path diff --git a/docs/project.json b/docs/project.json index b6bd97868bf..510b800de1c 100644 --- a/docs/project.json +++ b/docs/project.json @@ -1 +1 @@ -{"name": "nemoclaw", "version": "0.0.17"} +{"name": "nemoclaw", "version": "0.0.18"} diff --git a/docs/reference/commands.md b/docs/reference/commands.md index 90d71c2f452..6304e9f129f 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -210,6 +210,9 @@ If the backend is down, the output includes an `Inference: unreachable` line wit The Policy section displays the live enforced policy (fetched via `openshell policy get --full`), which reflects presets added or removed after sandbox creation. If the sandbox is running an outdated agent version, the output includes an `Update` line with the available version and a `nemoclaw rebuild` hint. +When other sandboxes have the same messaging channel enabled (Telegram, Discord, or Slack) with the same bot token, the output includes a cross-sandbox overlap warning so you can resolve the conflict before messages start dropping. +The command also tails `/tmp/gateway.log` inside the default sandbox and flags Telegram `409 Conflict` errors that indicate a duplicate consumer for the bot token. + ```console $ nemoclaw my-assistant status ``` @@ -318,6 +321,25 @@ $ nemoclaw my-assistant rebuild [--yes] [--verbose] The sandbox must be running for the backup step to succeed. After restore, the command runs `openclaw doctor --fix` for cross-version structure repair. +### `nemoclaw upgrade-sandboxes` + +Rebuild sandboxes whose base image is older than the one currently pinned by NemoClaw. +NemoClaw resolves the digest of `ghcr.io/nvidia/nemoclaw/sandbox-base:latest` from the registry, then compares it against the digest each sandbox was created with. +Sandboxes that match the current digest are left alone. + +```console +$ nemoclaw upgrade-sandboxes [--check] [--auto] [--yes] +``` + +| Flag | Description | +|------|-------------| +| `--check` | List stale sandboxes without rebuilding any of them. Exits non-zero if any are stale. | +| `--auto` | Rebuild every stale sandbox without prompting. Used by the installer to upgrade in place. | +| `--yes` | Skip the confirmation prompt for the rebuild plan. | + +Each rebuild reuses the same workspace backup-and-restore flow as `nemoclaw rebuild`, so workspace files survive the upgrade. +If the registry is unreachable (offline or firewalled hosts), NemoClaw falls back to the unpinned `:latest` tag and reports that the digest could not be resolved instead of failing. + ### `nemoclaw backup-all` Back up all registered running sandboxes to `~/.nemoclaw/rebuild-backups/`. @@ -423,6 +445,8 @@ $ nemoclaw debug [--quick] [--sandbox NAME] [--output PATH] | `--sandbox NAME` | Target a specific sandbox (default: auto-detect) | | `--output PATH` | Write diagnostics tarball to the given path | +If `--output` is set and the tarball cannot be written (for example, the destination directory is missing or read-only), the command exits non-zero so scripts can detect the failure. + ### `nemoclaw credentials list` List the names of all credentials stored in `~/.nemoclaw/credentials.json`. @@ -450,6 +474,9 @@ $ nemoclaw credentials reset NVIDIA_API_KEY Run `uninstall.sh` to remove NemoClaw sandboxes, gateway resources, related images and containers, and local state. The CLI uses the local `uninstall.sh` first and falls back to the hosted script if the local file is unavailable. +Uninstall also stops any orphaned `openshell` host processes left behind by previous onboard or destroy cycles, including `openshell sandbox create`, `openshell ssh-proxy`, and SSH sessions spawned by OpenShell. +Earlier releases only stopped `openshell forward` processes, so those orphans accumulated across runs. + | Flag | Effect | |---|---| | `--yes` | Skip the confirmation prompt | diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index c1bf19f2794..4e25d424a6c 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -242,6 +242,43 @@ If neither is found, verify that Colima is running: $ colima status ``` +### Re-onboard fails because port 18789 is held by SSH + +After destroying a sandbox and gateway, the SSH port-forward process for the +dashboard can be left running. +Re-running onboard then fails preflight with `Port 18789 is not available. +Blocked by: ssh`. + +Current NemoClaw detects this case and kills the orphaned SSH process +automatically before retrying the port check. +If you see the error on an older release, identify the SSH process and +terminate it manually: + +```console +$ sudo lsof -i :18789 +$ kill +``` + +Then re-run `nemoclaw onboard`. + +### Updated messaging token is not picked up + +Re-running `nemoclaw onboard --non-interactive` with a new +`TELEGRAM_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, or `SLACK_BOT_TOKEN` previously +reported success while the sandbox kept polling with the old credential. +Current NemoClaw stores SHA-256 hashes of messaging credentials in the +sandbox registry at creation time and detects when a token has changed. +When rotation is detected, NemoClaw automatically backs up workspace state, +deletes the sandbox, recreates it with the new credential, and restores the +backup. + +If you suspect a sandbox is still using a stale token, re-run onboarding so +the credential check runs: + +```console +$ nemoclaw onboard --non-interactive +``` + ### Sandbox creation killed by OOM (exit 137) On systems with 8 GB RAM or less and no swap configured, the sandbox image push can exhaust available memory and get killed by the Linux OOM killer (exit code 137). @@ -368,25 +405,23 @@ $ nemoclaw onboard ### Agent fails at runtime after onboarding succeeds with a compatible endpoint -Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` and pass -the onboarding validation probe, but their streaming mode is incomplete. +Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their +streaming mode is incomplete. OpenClaw requires granular streaming events like `response.output_text.delta` that these backends do not emit. -NemoClaw now tests streaming events during the `/v1/responses` probe and falls -back to `/v1/chat/completions` automatically. -If you onboarded before this check was added, re-run onboarding so the wizard -re-probes the endpoint and bakes the correct API path into the image: +For the compatible-endpoint provider, NemoClaw now defaults to +`/v1/chat/completions` and skips the Responses API probe entirely unless you +opt in. +If you onboarded an older release that selected `/v1/responses`, re-run +onboarding so the wizard rebuilds the image with chat completions: ```console $ nemoclaw onboard ``` -To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API`: - -```console -$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard -``` +If you previously set `NEMOCLAW_PREFERRED_API=openai-responses` to force the +Responses API, unset it before re-running onboard. Do not rely on `NEMOCLAW_INFERENCE_API_OVERRIDE` alone — it patches the config at container startup but does not update the Dockerfile ARG baked into the @@ -529,16 +564,36 @@ $ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard If you need to run multiple sandboxes at different ports at the same time, see [Running multiple sandboxes simultaneously](#running-multiple-sandboxes-simultaneously). -### Ollama network exposure warning during onboard +### Ollama auth proxy did not start + +NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated +reverse proxy on `0.0.0.0:11435` so the sandbox can reach Ollama without +exposing it to the local network. +If the proxy fails to start, onboarding exits before configuring inference. + +Check whether the proxy port is occupied by another process: + +```console +$ sudo lsof -i :11435 +``` + +Stop the conflicting process and re-run `nemoclaw onboard`. +The wizard cleans up stale proxy processes from previous runs automatically, +so most failures resolve by retrying. -When you select a local Ollama provider, onboarding binds Ollama to `0.0.0.0` so the Docker sandbox can reach the host. -This exposes the unauthenticated Ollama API to the local network. -NemoClaw prints a warning during onboarding to alert you to this risk. +The proxy token is persisted to `~/.nemoclaw/ollama-proxy-token` with `0600` +permissions. +If the file is missing or unreadable after a host reboot, re-running +`nemoclaw onboard` regenerates it. -On trusted private networks, the warning is informational. -On shared or public networks (airports, coffee shops), any adjacent device can send prompts to and enumerate models on your Ollama instance. +### Local inference health check resolves to IPv6 -The warning is suppressed on WSL, where Ollama binds to `127.0.0.1` instead. +Local inference health checks now use `127.0.0.1` instead of `localhost`. +On systems where `localhost` resolves to `::1` first, older NemoClaw releases +could probe the wrong address and report the local backend as unreachable +even when it was running. +If you see this on a current NemoClaw release, verify that the local backend +binds an IPv4 address and not only `::1`. ### Blueprint run failed diff --git a/docs/versions1.json b/docs/versions1.json index 9e9620ea806..693b32d82ba 100644 --- a/docs/versions1.json +++ b/docs/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": true, + "version": "0.0.18", + "url": "https://docs.nvidia.com/nemoclaw/0.0.18/" + }, + { "version": "0.0.17", "url": "https://docs.nvidia.com/nemoclaw/0.0.17/" },