Repository navigation
[Nemoclaw][All Platforms] Proxy test conflates wiring with proxy policy: Node https.get reports ERR_PROXY_TUNNEL when upstream proxy returns 403 #3836
Description
Description
Description
With NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT set to a custom HTTP proxy, NemoClaw correctly wires proxy environment variables and the Node ESM proxy wrapper, but the verification step using https.get("https://api.telegram.org/") fails with ERR_PROXY_TUNNEL because the upstream proxy returns HTTP/1.1 403 Forbidden on the CONNECT request. The existing test’s Expected section assumes a successful HTTP status (200/301/302) from Telegram and treats anything else as a failure, even though the proxy path is correct and the failure is due to proxy policy, not NemoClaw’s proxy wiring. This makes the test conflate “proxy wiring is broken” with “proxy is intentionally blocking the destination.”
Environment
- Platform: Linux (e.g. Ubuntu 22.04 / 24.04 / 26.04), in a NemoClaw sandbox.
-
NemoClaw: Installed and working (e.g. via
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash). - OpenShell gateway: Running as per NemoClaw Quickstart.
-
Proxy configuration:
-
NEMOCLAW_PROXY_HOST=10.200.0.1 -
NEMOCLAW_PROXY_PORT=3128 -
Proxy reachable from the sandbox, but configured to deny CONNECT tunnels to
api.telegram.org:443with HTTP 403 Forbidden.
-
-
Node: v22.x inside the sandbox (uses
EnvHttpProxyAgentwhenHTTP_PROXY/HTTPS_PROXYare set).
Steps to Reproduce
Preconditions
-
Set the proxy environment for NemoClaw:
bashexport NEMOCLAW_PROXY_HOST=10.200.0.1 export NEMOCLAW_PROXY_PORT=3128 -
Install NemoClaw and onboard at least one sandbox (e.g.
prachi-new-sb), ensuring the sandbox entrypoint completes successfully. -
Ensure no host‑level
HTTP_PROXY/HTTPS_PROXYoverrides are set beyond the NemoClaw proxy config.
Repro steps
-
Connect to the sandbox:
bashnemoclaw connect -
Inside the sandbox, verify proxy env and config files:
bashecho "$HTTP_PROXY" echo "$HTTPS_PROXY" ls -la /etc/profile.d/ 2>/dev/null || ls -la /sandbox/.profile.d/ 2>/dev/null stat -c '%U:%G %a' $(ls /etc/profile.d/nemoclaw-proxy.sh 2>/dev/null || ls /sandbox/.profile.d/nemoclaw-proxy.sh 2>/dev/null) grep -i proxy ~/.bashrc ~/.profile 2>&1 echo "$NO_PROXY" echo "$no_proxy" -
Verify Node wrapper:
bashecho "$NODE_OPTIONS" echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' -
Run the Node HTTPS probe:
bashnode -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));' -
Confirm proxy behavior with curl:
bashcurl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/
Expected Result (per current test case)
-
HTTP_PROXYandHTTPS_PROXYinside the sandbox reflect the custom proxy:
bashhttp://10.200.0.1:3128 http://10.200.0.1:3128 -
A root‑owned, read‑only NemoClaw proxy script exists in a sourced config directory, e.g.:
bashls -la /etc/profile.d/ -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh root:root 444 -
~/.bashrcand~/.profiledo not contain inline proxy exports or# BEGIN NEMOCLAW PROXYmarkers; they only source a runtime env file, e.g.:
bash/sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh -
NO_PROXYandno_proxycontain local bypass hosts and match exactly:
bashlocalhost,127.0.0.1,::1,10.200.0.1 localhost,127.0.0.1,::1,10.200.0.1 -
NODE_OPTIONSincludes the proxy wrapper:
bash--require /tmp/nemoclaw-http-proxy-fix.js ...and
bashecho "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js
Node HTTPS probe:-
Expected by the test:
STATUS:(e.g.STATUS:200or redirect) and not a low‑level error likeECONNREFUSEDor WebSocket 1006.
-
Expected by the test:
-
curl probe:
-
Expected:
HTTP:200/301/302, indicating successful proxy tunnel and request to Telegram.
-
Expected:
Actual Result
In the sandbox:
-
Env and proxy script:
bashecho $HTTPS_PROXY # http://10.200.0.1:3128 echo $HTTP_PROXY # http://10.200.0.1:3128 ls -la /etc/profile.d/ total 12 drwxr-xr-x 1 root root 4096 May 16 20:13 . drwxr-xr-x 1 root root 4096 May 19 16:59 .. -r--r--r-- 1 root root 141 May 16 20:13 nemoclaw-proxy.sh stat -c '%U:%G %a' /etc/profile.d/nemoclaw-proxy.sh # root:root 444 grep -i proxy ~/.bashrc ~/.profile 2>&1 /sandbox/.bashrc:# Source runtime proxy config /sandbox/.bashrc:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh /sandbox/.profile:# Source runtime proxy config /sandbox/.profile:[ -f /tmp/nemoclaw-proxy-env.sh ] && . /tmp/nemoclaw-proxy-env.sh echo "$NO_PROXY" echo "$no_proxy" # localhost,127.0.0.1,::1,10.200.0.1 # localhost,127.0.0.1,::1,10.200.0.1 -
Node wrapper:
bashecho "$NODE_OPTIONS" --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js --require /tmp/nemoclaw-sandbox-safety-net.js --require /tmp/nemoclaw-http-proxy-fix.js --require /tmp/nemoclaw-nemotron-inference-fix.js --require /tmp/nemoclaw-seccomp-guard.js --require /tmp/nemoclaw-ciao-network-guard.js echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js' nemoclaw-http-proxy-fix.js nemoclaw-http-proxy-fix.js -
Node HTTPS probe:
bashnode -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));' (node:864) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time. ERR:ERR_PROXY_TUNNEL -
curl via proxy:
bashcurl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/ * Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1,10.200.0.1' * Trying 10.200.0.1:3128... * CONNECT tunnel: HTTP/1.1 negotiated * Establish HTTP proxy tunnel to api.telegram.org:443 > CONNECT api.telegram.org:443 HTTP/1.1 > Host: api.telegram.org:443 > User-Agent: curl/8.14.1 > Proxy-Connection: Keep-Alive < < HTTP/1.1 403 Forbidden < Content-Type: application/json < Content-Length: 89 < Connection: close < * CONNECT tunnel failed, response 403 * closing connection #0 curl: (56) CONNECT tunnel failed, response 403
Key observations:
-
The proxy wiring is correct: Node and curl both route through the proxy; the CONNECT request and
no_proxybehavior are as expected. -
The failure (
ERR_PROXY_TUNNELandHTTP/1.1 403 Forbidden) is due to the upstream proxy deliberately denying tunnels toapi.telegram.org:443. - The current Expected section for this test implicitly assumes that the proxy will allow Telegram and that a 2xx/3xx HTTP status will be returned; it treats a proxy‑generated 403 as a test failure even though the underlying NemoClaw/OpenShell behavior is correct.
In other words, the test conflates:
- “Proxy wiring is broken / bypassed” (which would be a NemoClaw bug), with
- “Proxy path works but proxy policy blocks this host” (which is an environment/proxy policy issue).
Because the test step 6 currently expects a non‑error STATUS and not an ERR_PROXY_TUNNEL, this scenario registers as a failure even though the NemoClaw/OpenShell proxy integration is functioning as designed.
A fix would be to adjust the test’s Expected result to:
- Treat any outcome where Node/curl go through the proxy (CONNECT via 10.200.0.1:3128) as a wiring success, regardless of whether the proxy returns 2xx/3xx or 4xx, and
-
Reserve test failure for cases where Node/curl attempt direct connections to
api.telegram.orgthat bypass the proxy (e.g., ECONNREFUSED toapi.telegram.org:443with no proxy CONNECT shown in the logs).
Bug Details
| Field | Value |
|---|---|
| Priority | Unprioritized |
| Action | Dev - Open - To fix |
| Disposition | Open issue |
| Module | Machine Learning - NemoClaw |
| Keyword | NemoClaw, NEMOCLAW_GH_SYNC_APPROVAL |
[NVB#6192899]
Activity
chengjiew commentedon May 19, 2026
I'll take a first look.
fix(e2e): treat ERR_PROXY_TUNNEL as proxy wiring success in M12 test
chengjiew commentedon May 23, 2026
Hi @kagura-agent
If you want to take action for this issue, please help to notify me and assign you as engineer next time. Thanks!
kagura-chen commentedon May 23, 2026
Hi @chengjiew, thanks for the heads up! I'll make sure to comment and request assignment before starting work on issues going forward.
I already have PR #3880 up for this one — would you be able to assign me on this issue and take a look when you get a chance? Thanks!
kagura-chen commentedon Jun 3, 2026
Hi, I previously submitted PR #3880 for this but closed it after 14 days with no review. I'm going to unassign myself so someone else can pick this up. Happy to help if needed — feel free to re-assign me.
48 remaining items
sandl99 commentedon Jun 26, 2026
Hi, thanks for the detailed report. We tried to reproduce this in a NemoClaw sandbox using the Telegram Bot API path that the policy allows, and the request returned successfully:
The earlier
STATUS:403result appears to come from probinghttps://api.telegram.org/directly, which is outside the supported Telegram Bot API policy paths. The allowed shape ishttps://api.telegram.org/bot<TOKEN>/getMe, and that path is working in the latest NemoClaw behavior we tested.Could you please retry with the latest NemoClaw version? If you still see
ERR_PROXY_TUNNELfor an allowedapi.telegram.org/bot*/...request, please reopen or file a fresh issue with the updated sandbox logs and proxy details. Closing this as not reproducible for now.kagura-chen commentedon Jun 26, 2026
Thanks for investigating and the clear explanation @sandl99! You're right — the test was probing
https://api.telegram.org/directly rather than the supported/bot<TOKEN>/getMepath. That explains the 403.I'll retest with the correct allowed path shape on the latest NemoClaw version. If the issue persists for valid bot API requests, I'll open a fresh issue with updated logs. Appreciate the follow-up! 🙏
PrachiShevate-nv commentedon Jun 26, 2026
@sandl99 @kagura-agent
Retested on the latest NemoClaw build (v0.0.68) used in our environment. Reopening this issue for tracking purposes.
Inside the sandbox, all proxy wiring matches the current test’s expectations: HTTP_PROXY/HTTPS_PROXY are http://10.200.0.1:3128, NO_PROXY/no_proxy are localhost,127.0.0.1,::1,10.200.0.1, /etc/profile.d/nemoclaw-proxy.sh is root:root 444, and NODE_OPTIONS includes nemoclaw-http-proxy-fix.js.
For the Node HTTPS probe to https://api.telegram.org/, we still see ERR_PROXY_TUNNEL. A direct curl --proxy http://10.200.0.1:3128 https://api.telegram.org/ CONNECT also returns HTTP/1.1 403 Forbidden, so the upstream proxy at 10.200.0.1:3128 is denying the tunnel before Telegram is reached. This matches the “Actual Result” already captured in the bug and appears to be a proxy/infra policy issue, not a NemoClaw wiring bug.
Given that, we agree the wiring is correct and the failure is due to the proxy’s 403 behavior. If you have a recommended allowed api.telegram.org/bot/... path and a proxy configuration where that path returns STATUS:200/3xx instead of ERR_PROXY_TUNNEL, we can re‑verify there; on our current proxy, even the baseline https://api.telegram.org/ CONNECT is blocked with 403.
Logs: Please consider the striked out logs, for some reason they show me strikethrough
sandbox@65dfb7ec49aa:
$ echo "$HTTP_PROXY"$ echo "$HTTPS_PROXY"http://10.200.0.1:3128
sandbox@65dfb7ec49aa:
http://10.200.0.1:3128
sandbox@65dfb7ec49aa:
$ ls -la /etc/profile.d/ 2>/dev/null || ls -la /sandbox/.profile.d/ 2>/dev/null stat -c '%U:%G %a' $(ls /etc/profile.d/nemoclaw-proxy.sh 2>/dev/null || ls /sandbox/.profile.d/nemoclaw-proxy.sh 2>/dev/null) grep -i proxy$ curl: bash curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org//.bashrc$ node -e 'const https=require("https"); https.get("https://api.telegram.org/", r=>console.log("STATUS:"+r.statusCode)).on("error", e=>console.log("ERR:"+e.code));'/.profile 2>&1 echo "$NO_PROXY" echo "$no_proxy"$ echo "$NODE_OPTIONS" echo "$NODE_OPTIONS" | grep -o 'nemoclaw-http-proxy-fix.js'total 20
drwxr-xr-x 1 root root 4096 Jun 25 18:38 .
drwxr-xr-x 1 root root 4096 Jun 26 19:54 ..
-rw-r--r-- 1 root root 122 Jun 25 18:38 nemoclaw-linuxbrew.sh
-r--r--r-- 1 root root 141 Jun 25 18:36 nemoclaw-proxy.sh
sandbox@65dfb7ec49aa:
nemoclaw-http-proxy-fix.js
nemoclaw-http-proxy-fix.js
sandbox@65dfb7ec49aa:
(node:1575) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time.
(Use
node --trace-warnings ...to show where the warning was created)ERR:ERR_PROXY_TUNNEL
sandbox@65dfb7ec49aa:
bash: curl:: command not found
sandbox@65dfb7ec49aa:~$ curl -v --max-time 10 --proxy http://10.200.0.1:3128 https://api.telegram.org/
< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< Content-Length: 89
< Connection: close
<
curl: (56) CONNECT tunnel failed, response 403
sandbox@65dfb7ec49aa:~$
kagura-chen commentedon Jun 26, 2026
Thanks for the thorough retest @PrachiShevate-nv! The logs are very clear.
This confirms the original thesis of the issue — the proxy wiring itself is correct (env vars set,
nemoclaw-http-proxy-fix.jsloaded, CONNECT tunnel established), but the upstream proxy at10.200.0.1:3128denies the CONNECT toapi.telegram.org:443with a 403. Node then surfaces this asERR_PROXY_TUNNEL, which conflates a policy denial with a wiring failure from the user's perspective.The actionable improvement here would be for the proxy test (or its error reporting) to distinguish between:
For validation on an allowed path: @sandl99's earlier test showed
https://api.telegram.org/bot<TOKEN>/getMereturning STATUS:200, so the bot API paths appear to be in the proxy's allowlist — only the bare/root is blocked.fix(e2e): fold proxy-policy denial into the existing skip branch
test(e2e): extract proxy-policy denial classifier into a tested helper
fix(e2e): probe the allowed Telegram bot path instead of skipping on …
test(e2e): log the error code in the messaging reachability probes
fix(e2e): probe the allowed Telegram bot path in the messaging reacha…
fix(e2e): probe the allowed Telegram bot path in the messaging reacha…