Skip to content

nemoclaw onboard fails on Ubuntu 24.04/Zorin OS 18: k3s cannot pull images from Docker Hub (nested container networking) #455

Closed

Description

@Earththing

Description

Disclaimer: I don't really know what is going on, this is an AI's assessment of my issues:

Environment:

OS: Zorin OS 18 (Ubuntu 24.04 Noble base)
Docker: 29.3.0
OpenShell: 0.0.10
NemoClaw: latest main
GPU: None (cloud inference)

What happened:
nemoclaw onboard always fails at the "Initializing environment" step with K8s namespace not ready. The k3s instance inside the OpenShell gateway container cannot pull images from Docker Hub or registry.k8s.io — all requests time out with i/o timeout.
Root cause identified:
The k3s containerd inside the gateway container cannot reach external registries. The following images fail to pull:

rancher/mirrored-pause:3.6
rancher/klipper-helm:v0.9.14-build20260210
rancher/local-path-provisioner:v0.0.34
registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0

Manually importing these via docker save | ctr images import fixes the issue temporarily, but onboard.js always calls openshell gateway destroy before starting, wiping the imported images every run.
Workaround attempted:
Pre-seeding images into k3s containerd works IF done fast enough after gateway start, but the onboard.js code destroys and recreates the gateway every run making this impossible to automate reliably.
Suggested fix:
Either (1) bundle required images inside the gateway container image, or (2) add a pre-pull hook that seeds images before k3s initialization begins, or (3) don't unconditionally destroy the gateway if it's already healthy.

Reproduction Steps

Steps to Reproduce:

Fresh install of Zorin OS 18 (Ubuntu 24.04 base) with no GPU
Install Docker Engine following official Ubuntu instructions
Clone NemoClaw and run installer:

git clone https://github.com/NVIDIA/NemoClaw.git
cd NemoClaw
./install.sh

Install OpenShell:

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

Run nemoclaw setup-spark to fix Ubuntu 24.04 cgroup v2 issues (required or Docker networking breaks)
Run nemoclaw onboard

Expected result:
Onboarding completes successfully through all 7 steps.
Actual result:
Fails at step 2/7 "Initializing environment" with:
× K8s namespace not ready
╰─▶ timed out waiting for namespace 'openshell' to exist
To confirm root cause, run after gateway container starts:
docker logs openshell-cluster-nemoclaw 2>&1 | grep "i/o timeout"
You will see repeated failures pulling rancher/mirrored-pause:3.6 from registry-1.docker.io with TCP timeout to 174.129.222.113:443, even though the container itself has internet access (ping 8.8.8.8 works).

Environment

OS: Zorin OS 18 (Ubuntu 24.04 Noble base)
Docker: 29.3.0
OpenShell: 0.0.10
NemoClaw: latest main
GPU: None (cloud inference)

Logs

Checklist

  • I confirmed this bug is reproducible
  • I searched existing issues and this is not a duplicate

Activity

  1. wscurran commented on Mar 25, 2026

    @wscurran
    Contributor

    ✨ Thanks for submitting this issue about the onboard process failure on Ubuntu 24.04/Zorin OS 18, I'll investigate the k3s and Docker Hub issue to understand what's causing the problem with the K8s and OpenShell setup.


    Possibly related open PRs:

  2. wscurran commented on Apr 13, 2026

    @wscurran
    Contributor

    The gateway-destroy-on-retry behavior that made the image pre-seeding workaround impossible was addressed in #961 (onboard resiliency hardening, merged March 27). The nested container networking timeout from Docker Hub is an environment-specific issue that may or may not persist on current versions — OpenShell has moved significantly past 0.0.10. If you're still hitting k3s image pull timeouts on the current version, please reopen with docker logs openshell-cluster-nemoclaw output and we'll investigate the specific registry routing.

  3. added
    area: sandboxOpenShell sandbox lifecycle, runtime, config, or recovery
    platform: k8sAffects Kubernetes deployments
    and removed on Jun 3, 2026
  4. added theissue type on Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryplatform: k8sAffects Kubernetes deploymentsplatform: ubuntuAffects Ubuntu Linux environments

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions