diff --git a/README.md b/README.md index 2931415eb05..28357d8f020 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![Security Policy](https://img.shields.io/badge/Security-Report%20a%20Vulnerability-red)](SECURITY.md) [![Project Status](https://img.shields.io/badge/status-alpha-orange)](docs/about/release-notes.md) -NVIDIA NemoClaw is an open source stack that simplifies running [OpenClaw](https://openclaw.ai) always-on assistants safely. It installs the [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) runtime, part of [NVIDIA Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest), a secure environment for running autonomous agents, and open source models like [NVIDIA Nemotron](https://build.nvidia.com). +NVIDIA NemoClaw is an open source stack that simplifies running [OpenClaw](https://openclaw.ai) always-on assistants safely. It installs the [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) runtime, part of [NVIDIA Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest), a secure environment for running autonomous agents, with inference routed through [NVIDIA cloud](https://build.nvidia.com). > **Alpha software** > @@ -101,28 +101,25 @@ NemoClaw installs the NVIDIA OpenShell runtime and Nemotron models, then uses a | **Plugin** | TypeScript CLI commands for launch, connect, status, and logs. | | **Blueprint** | Versioned Python artifact that orchestrates sandbox creation, policy, and inference setup. | | **Sandbox** | Isolated OpenShell container running OpenClaw with policy-enforced egress and filesystem. | -| **Inference** | NVIDIA-routed model calls (cloud, local NIM, or vLLM), transparent to the agent. | +| **Inference** | NVIDIA cloud model calls, routed through the OpenShell gateway, transparent to the agent. | The blueprint lifecycle follows four stages: resolve the artifact, verify its digest, plan the resources, and apply through the OpenShell CLI. When something goes wrong, errors may originate from either NemoClaw or the OpenShell layer underneath. Run `nemoclaw status` for NemoClaw-level health and `openshell sandbox list` to check the underlying sandbox state. -## Inference Profiles +--- -Inference requests from the agent never leave the sandbox directly. OpenShell intercepts every call and routes it to the configured provider. NemoClaw ships with three profiles: +## Inference -| Profile | Provider | Model | Use Case | -|-------------|--------------|--------------------------------------|-------------------------------------------------| -| `default` | NVIDIA cloud | `nvidia/nemotron-3-super-120b-a12b` | Production. Requires an NVIDIA API key. | -| `nim-local` | Local NIM | `nvidia/nemotron-3-super-120b-a12b` | On-premises. NIM deployed as a local container. | -| `vllm` | vLLM | `nvidia/nemotron-3-nano-30b-a3b` | Local development. vLLM on the host. | +Inference requests from the agent never leave the sandbox directly. OpenShell intercepts every call and routes it to the NVIDIA cloud provider. -Select a profile at launch with `--profile`, or switch at runtime without restarting the sandbox: +| Provider | Model | Use Case | +|--------------|--------------------------------------|-------------------------------------------------| +| NVIDIA cloud | `nvidia/nemotron-3-super-120b-a12b` | Production. Requires an NVIDIA API key. | -```console -$ openclaw nemoclaw launch --profile vllm -$ openshell inference set --provider vllm-local --model nvidia/nemotron-3-nano-30b-a3b -``` +Get an API key from [build.nvidia.com](https://build.nvidia.com). The `nemoclaw onboard` command prompts for this key during setup. + +--- ## Protection Layers @@ -137,6 +134,8 @@ The sandbox starts with a strict baseline policy that controls network egress an When the agent tries to reach an unlisted host, OpenShell blocks the request and surfaces it in the TUI for operator approval. +--- + ## Key Commands ### Host commands (`nemoclaw`) @@ -167,6 +166,7 @@ See the full [CLI reference](https://docs.nvidia.com/nemoclaw/latest/reference/c > - The `openclaw nemoclaw` plugin commands are under active development. Use the `nemoclaw` host CLI as the primary interface. > - Setup may require manual workarounds on some platforms. File an issue if you encounter blockers. +--- ## Learn More @@ -175,7 +175,7 @@ Refer to the documentation for more information on NemoClaw. - [Overview](https://docs.nvidia.com/nemoclaw/latest/about/overview.html): what NemoClaw does and how it fits together - [How It Works](https://docs.nvidia.com/nemoclaw/latest/about/how-it-works.html): plugin, blueprint, and sandbox lifecycle - [Architecture](https://docs.nvidia.com/nemoclaw/latest/reference/architecture.html): plugin structure, blueprint lifecycle, and sandbox environment -- [Inference Profiles](https://docs.nvidia.com/nemoclaw/latest/reference/inference-profiles.html): NVIDIA cloud, NIM, and vLLM configuration +- [Inference Profiles](https://docs.nvidia.com/nemoclaw/latest/reference/inference-profiles.html): NVIDIA cloud inference configuration - [Network Policies](https://docs.nvidia.com/nemoclaw/latest/reference/network-policies.html): egress control and policy customization - [CLI Commands](https://docs.nvidia.com/nemoclaw/latest/reference/commands.html): full command reference diff --git a/docs/about/how-it-works.md b/docs/about/how-it-works.md index 110db6fad5d..39285f46b4f 100644 --- a/docs/about/how-it-works.md +++ b/docs/about/how-it-works.md @@ -49,13 +49,7 @@ After the sandbox starts, the agent runs inside it with all network, filesystem, Inference requests from the agent never leave the sandbox directly. OpenShell intercepts every inference call and routes it to the configured provider. -NemoClaw ships with three inference profiles: - -- NVIDIA cloud, which routes to Nemotron 3 Super 120B through [build.nvidia.com](https://build.nvidia.com). -- Local NIM, which routes to a NIM container on your local network. -- Local vLLM, which routes to a vLLM server on localhost for offline development. - -You can switch providers at runtime without restarting the sandbox. +NemoClaw routes inference to NVIDIA cloud, specifically Nemotron 3 Super 120B through [build.nvidia.com](https://build.nvidia.com). You can switch models at runtime without restarting the sandbox. ## Network and Filesystem Policy diff --git a/docs/about/overview.md b/docs/about/overview.md index 6dee26db57d..9a723624741 100644 --- a/docs/about/overview.md +++ b/docs/about/overview.md @@ -26,7 +26,7 @@ It moves OpenClaw into a sandboxed environment where every network request, file ## What NemoClaw Does - Sandboxes OpenClaw. Creates an OpenShell sandbox pre-configured for OpenClaw, with strict filesystem and network policies applied from the first boot. -- Routes inference through NVIDIA. Configures OpenShell inference routing so agent traffic flows through cloud-hosted Nemotron 3 Super 120B, a local NIM service, or vLLM, depending on the selected profile. +- Routes inference through NVIDIA cloud. Configures OpenShell inference routing so agent traffic flows through cloud-hosted Nemotron 3 Super 120B via [build.nvidia.com](https://build.nvidia.com). - Manages the lifecycle. Handles blueprint versioning, digest verification, and sandbox setup. ## How It Fits Together diff --git a/docs/index.md b/docs/index.md index 8c1bdbe8633..d111d02b87b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -24,7 +24,7 @@ status: published [![License](https://img.shields.io/badge/License-Apache_2.0-blue)](https://github.com/NVIDIA/NemoClaw/blob/main/LICENSE) NemoClaw is the OpenClaw plugin for [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell). -It runs OpenClaw inside a sandboxed environment with NVIDIA inference, such as Nemotron 3 Super 120B through [build.nvidia.com](https://build.nvidia.com) or local vLLM. +It runs OpenClaw inside a sandboxed environment with NVIDIA cloud inference, such as Nemotron 3 Super 120B through [build.nvidia.com](https://build.nvidia.com). The sandbox enforces strict network policies and operator-controlled egress approval. ## Get Started @@ -128,7 +128,7 @@ CLI commands for launching, connecting, monitoring, and managing sandboxes. :link: reference/inference-profiles :link-type: doc -Switch between NVIDIA cloud, local NIM, and vLLM inference backends. +NVIDIA cloud inference configuration and available models. +++ {bdg-secondary}`Reference` @@ -203,8 +203,6 @@ Quickstart :hidden: Switch Inference Providers -Set Up Local vLLM -Set Up a Local NIM Service ``` ```{toctree} diff --git a/docs/inference/set-up-local-nim.md b/docs/inference/set-up-local-nim.md deleted file mode 100644 index 8fc6af5f76c..00000000000 --- a/docs/inference/set-up-local-nim.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: - page: "Set Up a Local NIM Service for NemoClaw On-Premises Inference" - nav: "Set Up Local NIM" -description: "Route NemoClaw inference to an NVIDIA NIM container on your network." -keywords: ["nemoclaw local nim setup", "on-premises nim inference nemoclaw"] -topics: ["generative_ai", "ai_agents"] -tags: ["openclaw", "openshell", "nim", "inference_routing", "containers"] -content: - type: how_to - difficulty: intermediate - audience: ["developer", "engineer"] -status: published ---- - - - -# Set Up a Local NIM Service - -Route inference to an NVIDIA NIM container running on your local network. -This profile keeps all inference on-premises while using the same models available through NVIDIA cloud. - -## Prerequisites - -- A running NemoClaw sandbox. -- Docker or a compatible container runtime. -- A GPU with enough VRAM for the target model. -- An NVIDIA NGC API key for pulling NIM container images. - -## Pull and Start the NIM Container - -Pull the NIM container image and start it on port 8000: - -```console -$ docker run -d --gpus all \ - -p 8000:8000 \ - --name nim-service \ - nvcr.io/nim/nvidia/nemotron-3-super-120b-a12b:latest -``` - -Wait until the container logs show that the model is loaded and ready to serve requests. -Check the logs with the following command: - -```console -$ docker logs -f nim-service -``` - -## Verify the NIM Service - -Confirm that the NIM container is reachable: - -```console -$ curl http://nim-service.local:8000/v1/models -``` - -If `nim-service.local` does not resolve, use the container IP address or configure DNS to resolve the hostname. - -## Set the NIM API Key - -Export the `NIM_API_KEY` environment variable. -The `nim-local` profile uses this variable to authenticate with the NIM service: - -```console -$ export NIM_API_KEY= -``` - -## Switch to the NIM Provider - -Set the active inference provider to `nim-local`: - -```console -$ openshell inference set --provider nim-local --model nvidia/nemotron-3-super-120b-a12b -``` - -The change takes effect immediately. -No sandbox restart is required. - -## Verify Inference Routing - -Check the sandbox status to confirm the active provider: - -```console -$ openclaw nemoclaw status -``` - -The output should show `nim-local` as the active provider with the endpoint `nim-service.local:8000`. - -## Network Policy - -The sandbox network policy includes a `nim_service` entry that allows traffic to `nim-service.local:8000`. -This entry is part of the baseline policy in `nemoclaw-blueprint/policies/openclaw-sandbox.yaml`. -No additional policy changes are required. - -## Select the NIM Profile at Launch - -To start a new sandbox with the `nim-local` profile, pass `--profile nim-local` during launch: - -```console -$ openclaw nemoclaw launch --profile nim-local -``` - -## Related Topics - -- [Switch Inference Providers](switch-inference-providers.md) to change providers at runtime. -- [Inference Profiles](../reference/inference-profiles.md) for the full profile specification. -- [Network Policies](../reference/network-policies.md) for the `nim_service` policy entry. diff --git a/docs/inference/set-up-local-vllm.md b/docs/inference/set-up-local-vllm.md deleted file mode 100644 index 7834361872d..00000000000 --- a/docs/inference/set-up-local-vllm.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: - page: "Set Up Local vLLM for NemoClaw Offline Development" - nav: "Set Up Local vLLM" -description: "Run NemoClaw offline by routing inference to a local vLLM server." -keywords: ["nemoclaw local vllm setup", "offline inference vllm nemoclaw"] -topics: ["generative_ai", "ai_agents"] -tags: ["openclaw", "openshell", "vllm", "inference_routing", "local_development"] -content: - type: how_to - difficulty: intermediate - audience: ["developer", "engineer"] -status: published ---- - - - -# Set Up Local vLLM for Offline Development - -Run NemoClaw entirely offline by routing inference to a vLLM server on your local machine. -This profile does not require an NVIDIA API key or network access to cloud endpoints. -OpenShell routes requests from the gateway to the host, so the provider endpoint must be reachable from outside the host loopback interface. - -## Prerequisites - -- A running NemoClaw sandbox. -- A GPU with enough VRAM for the target model. -- Python 3.10 or later. -- The `vllm` Python package installed. - -## Start the vLLM Server - -Start a vLLM server on port 8000 with the Nemotron 3 Nano 30B model: - -```console -$ python -m vllm.entrypoints.openai.api_server \ - --model nvidia/nemotron-3-nano-30b-a3b \ - --host 0.0.0.0 \ - --port 8000 -``` - -Wait until the server logs indicate that the model is loaded and ready to accept requests. - -## Verify the Server - -Confirm that the vLLM server is reachable by querying its models endpoint: - -```console -$ curl http://localhost:8000/v1/models -``` - -The response should list `nvidia/nemotron-3-nano-30b-a3b` as an available model. - -## Gateway Reachability - -OpenShell configures local OpenAI-compatible providers with a host-visible endpoint such as `http://host.openshell.internal:8000/v1`. -Do not use `localhost` or `127.0.0.1` in the provider record, because those addresses resolve inside the gateway runtime, not on your workstation. - -If your host uses UFW, allow the vLLM port from the Docker bridge subnet before switching providers: - -```console -$ sudo ufw allow proto tcp from 172.18.0.0/16 to any port 8000 comment 'Allow OpenShell cluster to vLLM' -$ sudo ufw allow proto tcp from 172.17.0.0/16 to any port 8000 comment 'Allow Docker bridge to vLLM' -``` - -## Switch to the vLLM Provider - -Set the active inference provider to `vllm-local`: - -```console -$ openshell inference set --provider vllm-local --model nvidia/nemotron-3-nano-30b-a3b -``` - -The `vllm` profile uses the `OPENAI_API_KEY` environment variable. -For local use, this defaults to `dummy` and does not need to be set. -The provider endpoint is `http://host.openshell.internal:8000/v1`. - -## Test Inference - -Run a test inference request from inside the sandbox: - -```console -$ nemoclaw my-assistant connect -$ openclaw agent --agent main --local -m "Hello, are you running locally?" --session-id test -``` - -## Select the vLLM Profile at Launch - -To start a new sandbox with the vLLM profile from the beginning, pass `--profile vllm` during launch: - -```console -$ openclaw nemoclaw launch --profile vllm -``` - -## Related Topics - -- [Switch Inference Providers](switch-inference-providers.md) to change providers at runtime. -- [Inference Profiles](../reference/inference-profiles.md) for the full profile specification. diff --git a/docs/inference/switch-inference-providers.md b/docs/inference/switch-inference-providers.md index abb4636a853..c750c72d76f 100644 --- a/docs/inference/switch-inference-providers.md +++ b/docs/inference/switch-inference-providers.md @@ -1,11 +1,11 @@ --- title: - page: "Switch NemoClaw Inference Providers at Runtime" - nav: "Switch Inference Providers" -description: "Change the active inference provider without restarting the sandbox." -keywords: ["switch nemoclaw inference provider", "change inference runtime"] + page: "Switch NemoClaw Inference Models at Runtime" + nav: "Switch Inference Models" +description: "Change the active inference model without restarting the sandbox." +keywords: ["switch nemoclaw inference model", "change inference runtime"] topics: ["generative_ai", "ai_agents"] -tags: ["openclaw", "openshell", "inference_routing", "nim", "vllm"] +tags: ["openclaw", "openshell", "inference_routing"] content: type: how_to difficulty: technical_beginner @@ -18,9 +18,9 @@ status: published SPDX-License-Identifier: Apache-2.0 --> -# Switch Inference Providers at Runtime +# Switch Inference Models at Runtime -Change the active inference provider while the sandbox is running. +Change the active inference model while the sandbox is running. No restart is required. ## Prerequisites @@ -28,7 +28,7 @@ No restart is required. - A running NemoClaw sandbox. - The OpenShell CLI on your `PATH`. -## Switch to NVIDIA Cloud +## Switch to a Different Model Set the provider to `nvidia-nim` and specify a model from [build.nvidia.com](https://build.nvidia.com): @@ -36,33 +36,10 @@ Set the provider to `nvidia-nim` and specify a model from [build.nvidia.com](htt $ openshell inference set --provider nvidia-nim --model nvidia/nemotron-3-super-120b-a12b ``` -This profile requires the `NVIDIA_API_KEY` environment variable. +This requires the `NVIDIA_API_KEY` environment variable. The `nemoclaw onboard` command stores this key in `~/.nemoclaw/credentials.json` on first run. -## Switch to Local vLLM - -Set the provider to `vllm-local` and specify a model served by vLLM on the host: - -```console -$ openshell inference set --provider vllm-local --model nvidia/nemotron-3-nano-30b-a3b -``` - -The vLLM server must be running before you switch. -Bind the server to `0.0.0.0` and make sure the host firewall allows the bridge subnet to reach port `8000`. -Refer to [Set Up Local vLLM](set-up-local-vllm.md) for setup instructions. - -## Switch to Local NIM - -Set the provider to `nim-local` and specify a model served by a NIM container on your network: - -```console -$ openshell inference set --provider nim-local --model nvidia/nemotron-3-super-120b-a12b -``` - -This profile requires the `NIM_API_KEY` environment variable. -Refer to [Set Up a Local NIM Service](set-up-local-nim.md) for setup instructions. - -## Verify the Active Provider +## Verify the Active Model Run the status command to confirm the change: @@ -93,5 +70,3 @@ You can switch to any of these models at runtime. ## Related Topics - [Inference Profiles](../reference/inference-profiles.md) for full profile configuration details. -- [Set Up Local vLLM](set-up-local-vllm.md) for offline development with vLLM. -- [Set Up a Local NIM Service](set-up-local-nim.md) for on-premises NIM deployment. diff --git a/docs/monitoring/monitor-sandbox-activity.md b/docs/monitoring/monitor-sandbox-activity.md index c2f6c104ce6..6b358d067e3 100644 --- a/docs/monitoring/monitor-sandbox-activity.md +++ b/docs/monitoring/monitor-sandbox-activity.md @@ -105,7 +105,6 @@ If the request fails, check the following: 1. Run `openclaw nemoclaw status` to confirm the active provider and endpoint. 2. Run `openclaw nemoclaw logs -f` to view error messages from the blueprint runner. 3. Verify that the inference endpoint is reachable from the host. - For local vLLM or NIM, confirm that the server is running. ## Common Issues diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index fb6d59f28a9..b5bd02d9c5f 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -96,10 +96,7 @@ Inference requests from the agent never leave the sandbox directly. OpenShell intercepts them and routes to the configured provider: ```text -Agent (sandbox) ──▶ OpenShell gateway ──▶ Provider - ├── NVIDIA cloud (default) - ├── Local NIM service - └── vLLM on localhost +Agent (sandbox) ──▶ OpenShell gateway ──▶ NVIDIA cloud (build.nvidia.com) ``` Refer to [Inference Profiles](../reference/inference-profiles.md) for provider configuration details. diff --git a/docs/reference/inference-profiles.md b/docs/reference/inference-profiles.md index 9cead94f59d..2b6457acf7d 100644 --- a/docs/reference/inference-profiles.md +++ b/docs/reference/inference-profiles.md @@ -1,11 +1,11 @@ --- title: - page: "NemoClaw Inference Profiles — NVIDIA Cloud, NIM, and vLLM" + page: "NemoClaw Inference Profiles — NVIDIA Cloud" nav: "Inference Profiles" -description: "Configuration reference for NVIDIA cloud, local NIM, and vLLM profiles." -keywords: ["nemoclaw inference profiles", "nemoclaw nvidia nim vllm provider"] +description: "Configuration reference for NVIDIA cloud inference profiles." +keywords: ["nemoclaw inference profiles", "nemoclaw nvidia cloud provider"] topics: ["generative_ai", "ai_agents"] -tags: ["openclaw", "openshell", "inference_routing", "nim", "vllm", "llms"] +tags: ["openclaw", "openshell", "inference_routing", "llms"] content: type: reference difficulty: intermediate @@ -20,9 +20,9 @@ status: published # Inference Profiles -NemoClaw ships with three inference profiles defined in `blueprint.yaml`. -Each profile configures an OpenShell inference provider and model route. -The agent inside the sandbox uses whichever profile is active. +NemoClaw ships with an inference profile defined in `blueprint.yaml`. +The profile configures an OpenShell inference provider and model route. +The agent inside the sandbox uses whichever model is active. Inference requests are routed transparently through the OpenShell gateway. ## Profile Summary @@ -30,8 +30,6 @@ Inference requests are routed transparently through the OpenShell gateway. | Profile | Provider | Model | Endpoint | Use Case | |---|---|---|---|---| | `default` | NVIDIA cloud | `nvidia/nemotron-3-super-120b-a12b` | `integrate.api.nvidia.com` | Production. Requires an NVIDIA API key. | -| `nim-local` | Local NIM service | `nvidia/nemotron-3-super-120b-a12b` | `nim-service.local:8000` | On-premises. NIM deployed as a local pod. | -| `vllm` | vLLM | `nvidia/nemotron-3-nano-30b-a3b` | `host.openshell.internal:8000` | Local development. vLLM on the host. | ## Available Models @@ -44,8 +42,7 @@ The `nvidia-nim` provider registers the following models from [build.nvidia.com] | `nvidia/llama-3.3-nemotron-super-49b-v1.5` | Nemotron Super 49B v1.5 | 131,072 | 4,096 | | `nvidia/nemotron-3-nano-30b-a3b` | Nemotron 3 Nano 30B | 131,072 | 4,096 | -The `default` and `nim-local` profiles use Nemotron 3 Super 120B. -The `vllm` profile uses Nemotron 3 Nano 30B. +The default profile uses Nemotron 3 Super 120B. You can switch to any model in the catalog at runtime. ## `default` -- NVIDIA Cloud @@ -64,51 +61,12 @@ The `nemoclaw onboard` command prompts for this key and stores it in `~/.nemocla $ openshell inference set --provider nvidia-nim --model nvidia/nemotron-3-super-120b-a12b ``` -## `nim-local` -- Local NIM Service +## Switching Models at Runtime -Routes inference to a NIM container running on the local network. - -- **Provider type:** `openai`, which uses the OpenAI-compatible API -- **Endpoint:** `http://nim-service.local:8000/v1` -- **Model:** `nvidia/nemotron-3-super-120b-a12b` -- **Credential:** `NIM_API_KEY` environment variable - -The sandbox network policy includes a `nim_service` entry that allows traffic to `nim-service.local:8000`. - -```console -$ openshell inference set --provider nim-local --model nvidia/nemotron-3-super-120b-a12b -``` - -## `vllm` -- Local vLLM - -Routes inference to a vLLM server running on the host and exposed to the OpenShell gateway. - -- **Provider type:** `openai`, which uses the OpenAI-compatible API -- **Endpoint:** `http://host.openshell.internal:8000/v1` -- **Model:** `nvidia/nemotron-3-nano-30b-a3b` -- **Credential:** `OPENAI_API_KEY` environment variable. Defaults to `dummy` for local use. - -Start the vLLM server with a non-loopback bind address such as `0.0.0.0`. -On Linux hosts with UFW enabled, allow the Docker bridge subnet to reach port `8000`. - -```console -$ openshell inference set --provider vllm-local --model nvidia/nemotron-3-nano-30b-a3b -``` - -## Selecting a Profile at Launch - -Pass `--profile` when launching to select a profile: - -```console -$ openclaw nemoclaw launch --profile vllm -``` - -## Switching Profiles at Runtime - -After the sandbox is running, switch inference providers with the OpenShell CLI: +After the sandbox is running, switch models with the OpenShell CLI: ```console -$ openshell inference set --provider --model +$ openshell inference set --provider nvidia-nim --model ``` The change takes effect immediately.