Skip to content

[Nemoclaw][All Platforms] Proxy test conflates wiring with proxy policy: Node https.get reports ERR_PROXY_TUNNEL when upstream proxy returns 403 #3836

Closed

Description

@zNeill

Description

Description
With NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT set to a custom HTTP proxy, NemoClaw correctly wires proxy environment variables and the Node ESM proxy wrapper, but the verification step using https.get("https://api.telegram.org/") fails with ERR_PROXY_TUNNEL because the upstream proxy returns HTTP/1.1 403 Forbidden on the CONNECT request. The existing test’s Expected section assumes a successful HTTP status (200/301/302) from Telegram and treats anything else as a failure, even though the proxy path is correct and the failure is due to proxy policy, not NemoClaw’s proxy wiring. This makes the test conflate “proxy wiring is broken” with “proxy is intentionally blocking the destination.”

Environment

  • Platform: Linux (e.g. Ubuntu 22.04 / 24.04 / 26.04), in a NemoClaw sandbox.
  • NemoClaw: Installed and working (e.g. via curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash).
  • OpenShell gateway: Running as per NemoClaw Quickstart.
  • Proxy configuration:
    • NEMOCLAW_PROXY_HOST=10.200.0.1
    • NEMOCLAW_PROXY_PORT=3128
    • Proxy reachable from the sandbox, but configured to deny CONNECT tunnels to api.telegram.org:443 with HTTP 403 Forbidden.
  • Node: v22.x inside the sandbox (uses EnvHttpProxyAgent when HTTP_PROXY / HTTPS_PROXY are set).

Steps to Reproduce

Preconditions

  1. Set the proxy environment for NemoClaw:
     
    bash export NEMOCLAW_PROXY_HOST=10.200.0.1 export NEMOCLAW_PROXY_PORT=3128
  2. Install NemoClaw and onboard at least one sandbox (e.g. prachi-new-sb), ensuring the sandbox entrypoint completes successfully.
  3. Ensure no host‑level HTTP_PROXY / HTTPS_PROXY overrides are set beyond the NemoClaw proxy config.

Repro steps

  1. Connect to the sandbox:
     
    bash nemoclaw connect
  2. Inside the sandbox, verify proxy env and config files:
     
    bash echo "$HTTP_PROXY" echo "$HTTPS_PROXY" ls -la /etc/profile.d/ 2>/dev/null || ls -la /sandbox/.profile.d/ 2>/dev/null stat -c '%U:%G %a' $(ls /etc/profile.d/nemoclaw-proxy.sh 2>/dev/null || ls /sandbox/.profile.d/nemoclaw-proxy.sh 2>/dev/null) grep -i proxy ~/.bashrc ~/.profile 2>&1 echo "$NO_PROXY" echo "$no_proxy"
  3. Verify Node wrapper:
     
    bash echo "$NODE_OPTIONS" echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js'
  4. Run the Node HTTPS probe:
     
    bash node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));'
  5. Confirm proxy behavior with curl:
     
    bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/

Expected Result (per current test case)

  • HTTP_PROXY and HTTPS_PROXY inside the sandbox reflect the custom proxy:
     
    bash http://10.200.0.1:3128 http://10.200.0.1:3128
  • A root‑owned, read‑only NemoClaw proxy script exists in a sourced config directory, e.g.:
     
    bash ls -la /etc/profile.d/ -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh root:root 444
  • ~/.bashrc and ~/.profile do not contain inline proxy exports or # BEGIN NEMOCLAW PROXY markers; they only source a runtime env file, e.g.:
     
    bash /sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh
  • NO_PROXY and no_proxy contain local bypass hosts and match exactly:
     
    bash localhost,127.0.0.1,::1,10.200.0.1 localhost,127.0.0.1,::1,10.200.0.1
  • NODE_OPTIONS includes the proxy wrapper:
     
    bash --require /tmp/nemoclaw-http-proxy-fix.js ...

    and

     
    bash

    echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js


  • Node HTTPS probe:

    • Expected by the test: STATUS: (e.g. STATUS:200 or redirect) and not a low‑level error like ECONNREFUSED or WebSocket 1006.
  • curl probe:
    • Expected: HTTP:200/301/302, indicating successful proxy tunnel and request to Telegram.

Actual Result

In the sandbox:

  1. Env and proxy script:
     
    bash echo $HTTPS_PROXY # http://10.200.0.1:3128 echo $HTTP_PROXY # http://10.200.0.1:3128 ls -la /etc/profile.d/ total 12 drwxr-xr-x 1 root root 4096 May 16 20:13 . drwxr-xr-x 1 root root 4096 May 19 16:59 .. -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh # root:root 444 grep -i proxy ~/.bashrc ~/.profile 2>&1 /sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh echo "$NO_PROXY" echo "$no_proxy" # localhost,127.0.0.1,::1,10.200.0.1 # localhost,127.0.0.1,::1,10.200.0.1
  2. Node wrapper:
     
    bash echo "$NODE_OPTIONS" --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js nemoclaw-http-proxy-fix.js
  3. Node HTTPS probe:
     
    bash node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));' (node:864) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time. ERR:ERR_PROXY_TUNNEL
  4. curl via proxy:
     
    bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/ * Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1,10.200.0.1' * Trying 10.200.0.1:3128... * CONNECT tunnel: HTTP/1.1 negotiated * Establish HTTP proxy tunnel to api.telegram.org:443 > CONNECT api.telegram.org:443 HTTP/1.1 > Host: api.telegram.org:443 > User-Agent: curl/8.14.1 > Proxy-Connection: Keep-Alive < < HTTP/1.1 403 Forbidden < Content-Type: application/json < Content-Length: 89 < Connection: close < * CONNECT tunnel failed, response 403 * closing connection #0 curl: (56) CONNECT tunnel failed, response 403

Key observations:

  • The proxy wiring is correct: Node and curl both route through the proxy; the CONNECT request and no_proxy behavior are as expected.
  • The failure (ERR_PROXY_TUNNEL and HTTP/1.1 403 Forbidden) is due to the upstream proxy deliberately denying tunnels to api.telegram.org:443.
  • The current Expected section for this test implicitly assumes that the proxy will allow Telegram and that a 2xx/3xx HTTP status will be returned; it treats a proxy‑generated 403 as a test failure even though the underlying NemoClaw/OpenShell behavior is correct.

In other words, the test conflates:

  1. “Proxy wiring is broken / bypassed” (which would be a NemoClaw bug), with
  2. “Proxy path works but proxy policy blocks this host” (which is an environment/proxy policy issue).

Because the test step 6 currently expects a non‑error STATUS and not an ERR_PROXY_TUNNEL, this scenario registers as a failure even though the NemoClaw/OpenShell proxy integration is functioning as designed.

A fix would be to adjust the test’s Expected result to:

  • Treat any outcome where Node/curl go through the proxy (CONNECT via 10.200.0.1:3128) as a wiring success, regardless of whether the proxy returns 2xx/3xx or 4xx, and
  • Reserve test failure for cases where Node/curl attempt direct connections to api.telegram.org that bypass the proxy (e.g., ECONNREFUSED to api.telegram.org:443 with no proxy CONNECT shown in the logs).

Bug Details

Field Value
Priority Unprioritized
Action Dev - Open - To fix
Disposition Open issue
Module Machine Learning - NemoClaw
Keyword NemoClaw, NEMOCLAW_GH_SYNC_APPROVAL

[NVB#6192899]

Activity

  1. self-assigned this
    on May 19, 2026
  2. chengjiew commented on May 19, 2026

    @chengjiew
    Contributor

    I'll take a first look.

  3. added a commit that references this issue on May 22, 2026
    a62fab6
  4. chengjiew commented on May 23, 2026

    @chengjiew
    Contributor

    Hi @kagura-agent
    If you want to take action for this issue, please help to notify me and assign you as engineer next time. Thanks!

  5. removed their assignment
    on May 23, 2026
  6. kagura-chen commented on May 23, 2026

    @kagura-chen
    Contributor

    Hi @chengjiew, thanks for the heads up! I'll make sure to comment and request assignment before starting work on issues going forward.

    I already have PR #3880 up for this one — would you be able to assign me on this issue and take a look when you get a chance? Thanks!

  7. kagura-chen commented on Jun 3, 2026

    @kagura-chen
    Contributor

    Hi, I previously submitted PR #3880 for this but closed it after 14 days with no review. I'm going to unassign myself so someone else can pick this up. Happy to help if needed — feel free to re-assign me.

  8. 48 remaining items

  9. sandl99 commented on Jun 26, 2026

    @sandl99
    Collaborator

    Hi, thanks for the detailed report. We tried to reproduce this in a NemoClaw sandbox using the Telegram Bot API path that the policy allows, and the request returned successfully:

    STATUS:200
    

    The earlier STATUS:403 result appears to come from probing https://api.telegram.org/ directly, which is outside the supported Telegram Bot API policy paths. The allowed shape is https://api.telegram.org/bot<TOKEN>/getMe, and that path is working in the latest NemoClaw behavior we tested.

    Could you please retry with the latest NemoClaw version? If you still see ERR_PROXY_TUNNEL for an allowed api.telegram.org/bot*/... request, please reopen or file a fresh issue with the updated sandbox logs and proxy details. Closing this as not reproducible for now.

  10. kagura-chen commented on Jun 26, 2026

    @kagura-chen
    Contributor

    Thanks for investigating and the clear explanation @sandl99! You're right — the test was probing https://api.telegram.org/ directly rather than the supported /bot<TOKEN>/getMe path. That explains the 403.

    I'll retest with the correct allowed path shape on the latest NemoClaw version. If the issue persists for valid bot API requests, I'll open a fresh issue with updated logs. Appreciate the follow-up! 🙏

  11. PrachiShevate-nv commented on Jun 26, 2026

    @PrachiShevate-nv

    @sandl99 @kagura-agent
    Retested on the latest NemoClaw build (v0.0.68) used in our environment. Reopening this issue for tracking purposes.

    Inside the sandbox, all proxy wiring matches the current test’s expectations: HTTP_PROXY/HTTPS_PROXY are http://10.200.0.1:3128, NO_PROXY/no_proxy are localhost,127.0.0.1,::1,10.200.0.1, /etc/profile.d/nemoclaw-proxy.sh is root:root 444, and NODE_OPTIONS includes nemoclaw-http-proxy-fix.js.

    For the Node HTTPS probe to https://api.telegram.org/, we still see ERR_PROXY_TUNNEL. A direct curl --proxy http://10.200.0.1:3128 https://api.telegram.org/ CONNECT also returns HTTP/1.1 403 Forbidden, so the upstream proxy at 10.200.0.1:3128 is denying the tunnel before Telegram is reached. This matches the “Actual Result” already captured in the bug and appears to be a proxy/infra policy issue, not a NemoClaw wiring bug.

    Given that, we agree the wiring is correct and the failure is due to the proxy’s 403 behavior. If you have a recommended allowed api.telegram.org/bot/... path and a proxy configuration where that path returns STATUS:200/3xx instead of ERR_PROXY_TUNNEL, we can re‑verify there; on our current proxy, even the baseline https://api.telegram.org/ CONNECT is blocked with 403.

    Logs: Please consider the striked out logs, for some reason they show me strikethrough

    sandbox@65dfb7ec49aa:$ echo "$HTTP_PROXY"
    http://10.200.0.1:3128
    sandbox@65dfb7ec49aa:
    $ echo "$HTTPS_PROXY"
    http://10.200.0.1:3128
    sandbox@65dfb7ec49aa:$ ls -la /etc/profile.d/ 2>/dev/null || ls -la /sandbox/.profile.d/ 2>/dev/null stat -c '%U:%G %a' $(ls /etc/profile.d/nemoclaw-proxy.sh 2>/dev/null || ls /sandbox/.profile.d/nemoclaw-proxy.sh 2>/dev/null) grep -i proxy /.bashrc /.profile 2>&1 echo "$NO_PROXY" echo "$no_proxy"
    total 20
    drwxr-xr-x 1 root root 4096 Jun 25 18:38 .
    drwxr-xr-x 1 root root 4096 Jun 26 19:54 ..
    -rw-r--r-- 1 root root 122 Jun 25 18:38 nemoclaw-linuxbrew.sh
    -r--r--r-- 1 root root 141 Jun 25 18:36 nemoclaw-proxy.sh
    sandbox@65dfb7ec49aa:
    $ echo "$NODE_OPTIONS" echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js'
    nemoclaw-http-proxy-fix.js
    nemoclaw-http-proxy-fix.js
    sandbox@65dfb7ec49aa:
    $ node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));'
    (node:1575) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time.
    (Use node --trace-warnings ... to show where the warning was created)
    ERR:ERR_PROXY_TUNNEL
    sandbox@65dfb7ec49aa:
    $ curl: bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/
    bash: curl:: command not found
    sandbox@65dfb7ec49aa:~$ curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/

    • Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1,10.200.0.1'
    • Trying 10.200.0.1:3128...
    • CONNECT tunnel: HTTP/1.1 negotiated
    • allocate connect buffer
    • Establish HTTP proxy tunnel to api.telegram.org:443

    CONNECT api.telegram.org:443 HTTP/1.1
    Host: api.telegram.org:443
    User-Agent: curl/8.14.1
    Proxy-Connection: Keep-Alive

    < HTTP/1.1 403 Forbidden
    < Content-Type: application/json
    < Content-Length: 89
    < Connection: close
    <

    • CONNECT tunnel failed, response 403
    • closing connection #0
      curl: (56) CONNECT tunnel failed, response 403
      sandbox@65dfb7ec49aa:~$
  12. kagura-chen commented on Jun 26, 2026

    @kagura-chen
    Contributor

    Thanks for the thorough retest @PrachiShevate-nv! The logs are very clear.

    This confirms the original thesis of the issue — the proxy wiring itself is correct (env vars set, nemoclaw-http-proxy-fix.js loaded, CONNECT tunnel established), but the upstream proxy at 10.200.0.1:3128 denies the CONNECT to api.telegram.org:443 with a 403. Node then surfaces this as ERR_PROXY_TUNNEL, which conflates a policy denial with a wiring failure from the user's perspective.

    The actionable improvement here would be for the proxy test (or its error reporting) to distinguish between:

    1. Wiring failure — proxy unreachable, env not set, fix script missing
    2. Policy denial — proxy reached and tunnel attempted, but upstream returned 403/407

    For validation on an allowed path: @sandl99's earlier test showed https://api.telegram.org/bot<TOKEN>/getMe returning STATUS:200, so the bot API paths appear to be in the proxy's allowlist — only the bare / root is blocked.

  13. added a commit that references this issue on Jul 3, 2026
    293ddb2
  14. added a commit that references this issue on Jul 12, 2026
    d78ca78
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA TeamPRRPRR recommendation trackingVDRLinked to VDR findingarea: networkingDNS, proxy, TLS, ports, host aliases, or connectivity

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions