Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 48 additions & 29 deletions .agents/skills/nemoclaw-user-configure-inference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,11 +143,13 @@ Select the same provider and endpoint again.
The updated streaming probe will detect incomplete `/v1/responses` support
and select `/v1/chat/completions` automatically.

To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API`
before onboarding:
For the compatible-endpoint provider, NemoClaw uses `/v1/chat/completions` by
default, so no env var is required to keep the safe path.
To opt in to `/v1/responses` for a backend you have verified end to end, set
`NEMOCLAW_PREFERRED_API` before onboarding:

```console
$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard
$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard
```

> **Note:** `NEMOCLAW_INFERENCE_API_OVERRIDE` patches the config at container startup but
Expand Down Expand Up @@ -233,19 +235,35 @@ Select **Local Ollama** from the provider list.
NemoClaw lists installed models or offers starter models if none are installed.
It pulls the selected model, loads it into memory, and validates it before continuing.

### Linux with Docker
### Authenticated Reverse Proxy

On Linux hosts that run NemoClaw with Docker, the sandbox reaches Ollama through
`http://host.openshell.internal:11434`, not the host shell's `localhost` socket.
If Ollama is already running, make sure it listens on `0.0.0.0:11434` instead of
`127.0.0.1:11434`.
NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated
reverse proxy on `0.0.0.0:11435`.
Containers and other hosts on the local network reach Ollama only through the
proxy, which validates a Bearer token before forwarding requests.
Ollama itself is never exposed without authentication.

```console
$ OLLAMA_HOST=0.0.0.0:11434 ollama serve
```
The onboard wizard manages the proxy automatically:

- Generates a random 24-byte token on first run and stores it in
`~/.nemoclaw/ollama-proxy-token` with `0600` permissions.
- Starts the proxy after Ollama and verifies it before continuing.
- Cleans up stale proxy processes from previous runs.
- Reuses the persisted token after a host reboot so you do not need to re-run
onboard.

The sandbox provider is configured to use proxy port `11435` with the generated
token as its `OPENAI_API_KEY` credential.
OpenShell's L7 proxy injects the token at egress, so the agent inside the
sandbox never sees the token directly.

`GET /api/tags` is exempt from authentication so container health checks
continue to work.
All other endpoints (including `POST /api/tags`) require the Bearer token.

If Ollama only binds loopback, NemoClaw can detect it on the host, but the
sandbox-side validation step fails because containers cannot reach it.
If Ollama is already running on a non-loopback address when you start onboard,
the wizard restarts it on `127.0.0.1:11434` so the proxy is the only network
path to the model server.

### Non-Interactive Setup

Expand All @@ -265,8 +283,9 @@ If `NEMOCLAW_MODEL` is not set, NemoClaw selects a default model based on availa
## Step 6: OpenAI-Compatible Server

This option works with any server that implements `/v1/chat/completions`, including vLLM, TensorRT-LLM, llama.cpp, LocalAI, and others.
If the server also supports `/v1/responses`, NemoClaw only favors that path when onboarding can verify tool-calling behavior that matches what OpenClaw actually sends.
Otherwise NemoClaw falls back to `/v1/chat/completions`.
For compatible endpoints, NemoClaw uses `/v1/chat/completions` by default.
This avoids a class of failures where local backends accept `/v1/responses` requests but silently drop the system prompt and tool definitions.
To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API=openai-responses` before running onboard.

Start your model server.
The examples below use vLLM, but any OpenAI-compatible server works.
Expand All @@ -288,8 +307,8 @@ The wizard prompts for an API key.
If your server does not require authentication, enter any non-empty string (for example, `dummy`).

NemoClaw validates the endpoint by sending a test inference request before continuing.
For OpenAI-compatible endpoints, the validation prefers `/responses` only when the probe produces a compatible function or tool call.
Endpoints that return `200 OK` on `/responses` but do not format tool calls the way OpenClaw expects are configured to use `/chat/completions` instead.
The wizard probes `/v1/chat/completions` by default for the compatible-endpoint provider.
If you set `NEMOCLAW_PREFERRED_API=openai-responses`, NemoClaw probes `/v1/responses` instead and only selects it when the response includes the streaming events OpenClaw requires.

### Non-Interactive Setup

Expand All @@ -310,27 +329,27 @@ $ NEMOCLAW_PROVIDER=custom \
| `NEMOCLAW_MODEL` | Model ID as reported by the server. |
| `COMPATIBLE_API_KEY` | API key for the endpoint. Use any non-empty value if authentication is not required. |

### Forcing Chat Completions API
### Selecting the API Path

Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but do
not emit the granular streaming events that OpenClaw requires.
NemoClaw tests streaming events during onboarding and falls back to
`/v1/chat/completions` automatically when it detects incomplete streaming.
For the compatible-endpoint provider, `/v1/chat/completions` is the default.
NemoClaw tests streaming events during onboarding and uses chat completions
without probing the Responses API.

If you need to bypass the `/v1/responses` probe entirely, set
`NEMOCLAW_PREFERRED_API` before running onboard:
To opt in to `/v1/responses`, set `NEMOCLAW_PREFERRED_API` before running onboard:

```console
$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard
$ NEMOCLAW_PREFERRED_API=openai-responses nemoclaw onboard
```

Set this variable to make the wizard skip the `/v1/responses` probe and use
`/v1/chat/completions` directly.
You can use it in both interactive and non-interactive mode.
The wizard then probes `/v1/responses` and only selects it when streaming
support is complete.
If the probe fails, the wizard falls back to `/v1/chat/completions`
automatically.
You can use this variable in both interactive and non-interactive mode.

| Variable | Values | Default |
|---|---|---|
| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `chat-completions` | unset (auto-detect) |
| `NEMOCLAW_PREFERRED_API` | `openai-completions`, `openai-responses` | `openai-completions` for compatible endpoints |

If you already onboarded and the sandbox is failing at runtime, re-run
`nemoclaw onboard` to re-probe the endpoint and bake the correct API path
Expand Down
8 changes: 8 additions & 0 deletions .agents/skills/nemoclaw-user-deploy-remote/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,14 @@ Channel entries in `/sandbox/.openclaw/openclaw.json` are fixed at image build t

If you add or change `TELEGRAM_BOT_TOKEN` (or toggle channels) after a sandbox already exists, you typically need to run `nemoclaw onboard` again so the image and provider attachments are rebuilt with the new settings.

NemoClaw stores a SHA-256 hash of each messaging token in the sandbox registry at creation time.
When you re-run `nemoclaw onboard --non-interactive` with a new token, NemoClaw detects the change, backs up workspace state, deletes the sandbox, recreates it with the new credential, and restores the backup.
This makes credential rotation safe to script.

Telegram, Discord, and Slack each allow only one active consumer per bot token.
If you enable a messaging channel and another sandbox already uses the same token, onboard prompts you to confirm before continuing in interactive mode and exits non-zero in non-interactive mode.
`nemoclaw status` also reports cross-sandbox overlaps so you can resolve duplicates before messages start dropping.

For a full first-time flow, refer to Quickstart (see the `nemoclaw-user-get-started` skill).

## Step 12: Confirm Delivery
Expand Down
27 changes: 27 additions & 0 deletions .agents/skills/nemoclaw-user-reference/references/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,9 @@ If the backend is down, the output includes an `Inference: unreachable` line wit
The Policy section displays the live enforced policy (fetched via `openshell policy get --full`), which reflects presets added or removed after sandbox creation.
If the sandbox is running an outdated agent version, the output includes an `Update` line with the available version and a `nemoclaw <name> rebuild` hint.

When other sandboxes have the same messaging channel enabled (Telegram, Discord, or Slack) with the same bot token, the output includes a cross-sandbox overlap warning so you can resolve the conflict before messages start dropping.
The command also tails `/tmp/gateway.log` inside the default sandbox and flags Telegram `409 Conflict` errors that indicate a duplicate consumer for the bot token.

```console
$ nemoclaw my-assistant status
```
Expand Down Expand Up @@ -292,6 +295,25 @@ $ nemoclaw my-assistant rebuild [--yes] [--verbose]
The sandbox must be running for the backup step to succeed.
After restore, the command runs `openclaw doctor --fix` for cross-version structure repair.

### `nemoclaw upgrade-sandboxes`

Rebuild sandboxes whose base image is older than the one currently pinned by NemoClaw.
NemoClaw resolves the digest of `ghcr.io/nvidia/nemoclaw/sandbox-base:latest` from the registry, then compares it against the digest each sandbox was created with.
Sandboxes that match the current digest are left alone.

```console
$ nemoclaw upgrade-sandboxes [--check] [--auto] [--yes]
```

| Flag | Description |
|------|-------------|
| `--check` | List stale sandboxes without rebuilding any of them. Exits non-zero if any are stale. |
| `--auto` | Rebuild every stale sandbox without prompting. Used by the installer to upgrade in place. |
| `--yes` | Skip the confirmation prompt for the rebuild plan. |

Each rebuild reuses the same workspace backup-and-restore flow as `nemoclaw <name> rebuild`, so workspace files survive the upgrade.
If the registry is unreachable (offline or firewalled hosts), NemoClaw falls back to the unpinned `:latest` tag and reports that the digest could not be resolved instead of failing.

### `nemoclaw backup-all`

Back up all registered running sandboxes to `~/.nemoclaw/rebuild-backups/`.
Expand Down Expand Up @@ -395,6 +417,8 @@ $ nemoclaw debug [--quick] [--sandbox NAME] [--output PATH]
| `--sandbox NAME` | Target a specific sandbox (default: auto-detect) |
| `--output PATH` | Write diagnostics tarball to the given path |

If `--output` is set and the tarball cannot be written (for example, the destination directory is missing or read-only), the command exits non-zero so scripts can detect the failure.

### `nemoclaw credentials list`

List the names of all credentials stored in `~/.nemoclaw/credentials.json`.
Expand Down Expand Up @@ -422,6 +446,9 @@ $ nemoclaw credentials reset NVIDIA_API_KEY
Run `uninstall.sh` to remove NemoClaw sandboxes, gateway resources, related images and containers, and local state.
The CLI uses the local `uninstall.sh` first and falls back to the hosted script if the local file is unavailable.

Uninstall also stops any orphaned `openshell` host processes left behind by previous onboard or destroy cycles, including `openshell sandbox create`, `openshell ssh-proxy`, and SSH sessions spawned by OpenShell.
Earlier releases only stopped `openshell forward` processes, so those orphans accumulated across runs.

| Flag | Effect |
|---|---|
| `--yes` | Skip the confirmation prompt |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,43 @@ If neither is found, verify that Colima is running:
$ colima status
```

### Re-onboard fails because port 18789 is held by SSH

After destroying a sandbox and gateway, the SSH port-forward process for the
dashboard can be left running.
Re-running onboard then fails preflight with `Port 18789 is not available.
Blocked by: ssh`.

Current NemoClaw detects this case and kills the orphaned SSH process
automatically before retrying the port check.
If you see the error on an older release, identify the SSH process and
terminate it manually:

```console
$ sudo lsof -i :18789
$ kill <PID>
```

Then re-run `nemoclaw onboard`.

### Updated messaging token is not picked up

Re-running `nemoclaw onboard --non-interactive` with a new
`TELEGRAM_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, or `SLACK_BOT_TOKEN` previously
reported success while the sandbox kept polling with the old credential.
Current NemoClaw stores SHA-256 hashes of messaging credentials in the
sandbox registry at creation time and detects when a token has changed.
When rotation is detected, NemoClaw automatically backs up workspace state,
deletes the sandbox, recreates it with the new credential, and restores the
backup.

If you suspect a sandbox is still using a stale token, re-run onboarding so
the credential check runs:

```console
$ nemoclaw onboard --non-interactive
```

### Sandbox creation killed by OOM (exit 137)

On systems with 8 GB RAM or less and no swap configured, the sandbox image push can exhaust available memory and get killed by the Linux OOM killer (exit code 137).
Expand Down Expand Up @@ -338,25 +375,23 @@ $ nemoclaw onboard

### Agent fails at runtime after onboarding succeeds with a compatible endpoint

Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` and pass
the onboarding validation probe, but their streaming mode is incomplete.
Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their
streaming mode is incomplete.
OpenClaw requires granular streaming events like `response.output_text.delta`
that these backends do not emit.

NemoClaw now tests streaming events during the `/v1/responses` probe and falls
back to `/v1/chat/completions` automatically.
If you onboarded before this check was added, re-run onboarding so the wizard
re-probes the endpoint and bakes the correct API path into the image:
For the compatible-endpoint provider, NemoClaw now defaults to
`/v1/chat/completions` and skips the Responses API probe entirely unless you
opt in.
If you onboarded an older release that selected `/v1/responses`, re-run
onboarding so the wizard rebuilds the image with chat completions:

```console
$ nemoclaw onboard
```

To force `/v1/chat/completions` without re-probing, set `NEMOCLAW_PREFERRED_API`:

```console
$ NEMOCLAW_PREFERRED_API=openai-completions nemoclaw onboard
```
If you previously set `NEMOCLAW_PREFERRED_API=openai-responses` to force the
Responses API, unset it before re-running onboard.

Do not rely on `NEMOCLAW_INFERENCE_API_OVERRIDE` alone — it patches the config
at container startup but does not update the Dockerfile ARG baked into the
Expand Down Expand Up @@ -499,16 +534,36 @@ $ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard
If you need to run multiple sandboxes at different ports at the same time, see
[Running multiple sandboxes simultaneously](#running-multiple-sandboxes-simultaneously).

### Ollama network exposure warning during onboard
### Ollama auth proxy did not start

NemoClaw keeps Ollama bound to `127.0.0.1:11434` and starts a token-gated
reverse proxy on `0.0.0.0:11435` so the sandbox can reach Ollama without
exposing it to the local network.
If the proxy fails to start, onboarding exits before configuring inference.

Check whether the proxy port is occupied by another process:

```console
$ sudo lsof -i :11435
```

Stop the conflicting process and re-run `nemoclaw onboard`.
The wizard cleans up stale proxy processes from previous runs automatically,
so most failures resolve by retrying.

When you select a local Ollama provider, onboarding binds Ollama to `0.0.0.0` so the Docker sandbox can reach the host.
This exposes the unauthenticated Ollama API to the local network.
NemoClaw prints a warning during onboarding to alert you to this risk.
The proxy token is persisted to `~/.nemoclaw/ollama-proxy-token` with `0600`
permissions.
If the file is missing or unreadable after a host reboot, re-running
`nemoclaw onboard` regenerates it.

On trusted private networks, the warning is informational.
On shared or public networks (airports, coffee shops), any adjacent device can send prompts to and enumerate models on your Ollama instance.
### Local inference health check resolves to IPv6

The warning is suppressed on WSL, where Ollama binds to `127.0.0.1` instead.
Local inference health checks now use `127.0.0.1` instead of `localhost`.
On systems where `localhost` resolves to `::1` first, older NemoClaw releases
could probe the wrong address and report the local backend as unreachable
even when it was running.
If you see this on a current NemoClaw release, verify that the local backend
binds an IPv4 address and not only `::1`.

### Blueprint run failed

Expand Down
16 changes: 16 additions & 0 deletions docs/.docs-skip
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,24 @@
skip-features:
- "--dangerously-skip-permissions" # experimental flag, pending UX review
- "openclaw-sandbox-permissive.yaml" # permissive policy file, same feature
- "shields and config commands" # experimental sandbox-mgmt commands (#1976), not advertised
- "src/lib/shields.ts" # shields implementation, experimental
- "src/lib/shields-timer.ts" # shields auto-restore timer, experimental
- "src/lib/shields-audit.ts" # shields audit trail, experimental
- "src/lib/sandbox-config.ts" # config get/set/rotate-token, experimental
- "test/shields" # shields tests, experimental
- "test/config-set.test.ts" # config set tests, experimental
- "test/config-rotate-token.test.ts" # config rotate-token tests, experimental
- "test/e2e/test-shields-config.sh" # shields/config E2E, experimental
- "shields-status" # plugin shields-status command, experimental
- "config-show" # plugin config-show command, experimental

skip-terms:
- "dangerously-skip-permissions" # do not mention this flag anywhere in docs
- "permissive mode" # do not reference this concept in docs
- "Hermes"
- "shields down" # experimental shields command, not advertised
- "shields up" # experimental shields command, not advertised
- "shields status" # experimental shields command, not advertised
- "config rotate-token" # experimental config command, not advertised
- "rotate-token" # experimental config command, not advertised
6 changes: 5 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,5 +139,9 @@

# Keep project.json in sync with the resolved release version so the
# static copy served alongside the docs always reports the correct version.
# Write only when the contents change so sphinx-autobuild does not detect
# a self-induced source change and rebuild in an infinite loop.
_project_json = Path(__file__).parent / "project.json"
_project_json.write_text(json.dumps({"name": "nemoclaw", "version": release}) + "\n")
_project_json_contents = json.dumps({"name": "nemoclaw", "version": release}) + "\n"
if not _project_json.exists() or _project_json.read_text() != _project_json_contents:
_project_json.write_text(_project_json_contents)
8 changes: 8 additions & 0 deletions docs/deployment/set-up-telegram-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,14 @@ Channel entries in `/sandbox/.openclaw/openclaw.json` are fixed at image build t

If you add or change `TELEGRAM_BOT_TOKEN` (or toggle channels) after a sandbox already exists, you typically need to run `nemoclaw onboard` again so the image and provider attachments are rebuilt with the new settings.

NemoClaw stores a SHA-256 hash of each messaging token in the sandbox registry at creation time.
When you re-run `nemoclaw onboard --non-interactive` with a new token, NemoClaw detects the change, backs up workspace state, deletes the sandbox, recreates it with the new credential, and restores the backup.
This makes credential rotation safe to script.

Telegram, Discord, and Slack each allow only one active consumer per bot token.
If you enable a messaging channel and another sandbox already uses the same token, onboard prompts you to confirm before continuing in interactive mode and exits non-zero in non-interactive mode.
`nemoclaw status` also reports cross-sandbox overlaps so you can resolve duplicates before messages start dropping.

For a full first-time flow, refer to [Quickstart](../get-started/quickstart.md).

## Confirm Delivery
Expand Down
Loading
Loading