`nemoclaw status` reports inference healthy on endpoint reachability, not model invocability — a green status can mask an unreachable model

[Hokonoken]

Summary

nemoclaw status can report healthy inference while the configured model cannot actually be invoked. The inference health probes check that the provider endpoint answers an HTTP request (or that the local daemon lists models), not that the configured model responds to a real completion call. This confirms the observation @mikemason left on #6635 ("At the time OpenClaw couldn't reach the model, nemoclaw status was reporting healthy inference, so whatever it's checking is slightly different than what OpenClaw actually uses to hit the model") — and it reproduces without the corporate-proxy / TLS specifics of that thread.

Mechanism (code)

For every provider except the NVIDIA Kimi K2.6 special case, the health probe is a reachability/listing check:

• Local (ollama-local / vllm-local) — probeLocalProviderHealth(provider, options) (src/lib/inference/local.ts) does not take a model argument at all. It hits /api/tags and returns healthy when the body parses as { "models": [...] } — and an empty array is explicitly accepted ("just means no models pulled yet", isValidOllamaTagsResponseBody). Daemon-up ⇒ healthy, regardless of the configured model.
• Remote (anthropic-prod, openai-api, gemini-api, non-Kimi nvidia-*) — getRemoteProviderHealthEndpoint returns the provider's /v1/models listing URL, and probeRemoteProviderHealth curls it with no credential (["-sS", "--connect-timeout", "3", "--max-time", "5", endpoint]), then sets ok = (curlStatus === 0) with the comment "Even a 401/403 means the endpoint is reachable" (src/lib/inference/health.ts). Because it is unauthenticated, any HTTP answer — including 401 Unauthorized — reads as healthy: the check confirms the host is network-reachable, nothing about credential validity or model invocability.
• Only NVIDIA_MANAGED_PROVIDERS + isKimiK26Model routes to probeNvidiaKimiK26Health, which hits /chat/completions — an actual invocation. That is the one path that reflects model invocability.

The in-sandbox probe (probeSandboxInferenceGatewayHealth, inference-route-health.ts) is likewise a route-reachability check on https://inference.local/v1/models, not a completion call.

Reproduction (by execution, proxy-independent)

Driving the compiled probes on x86_64/WSL2, no proxy dependence:

Local — the probe is model-blind:

$ node -e 'console.log(require("./dist/lib/inference/local.js").probeLocalProviderHealth("ollama-local"))'
{
  ok: true,
  providerLabel: 'Local Ollama',
  endpoint: 'http://127.0.0.1:11434/api/tags',
  detail: 'Local Ollama is reachable on http://127.0.0.1:11434/api/tags.',
  probeLabel: 'ollama backend',
  subprobes: [ { probeLabel: 'auth proxy', ok: false, ... } ]   // auth-proxy hop down in this lab; incidental — the point is the top-level model-blind `ok: true`
}

# but an actual call to a model that is not present fails:
$ curl -s http://127.0.0.1:11434/api/chat -d '{"model":"nonexistent-model:latest","messages":[{"role":"user","content":"hi"}],"stream":false}'
{"error":"model 'nonexistent-model:latest' not found"}

probeLocalProviderHealth takes no model parameter, so a sandbox configured for a model that is not pulled still probes ok: true while a real call errors model not found.

Remote — a 401 reads as healthy:

$ node -e 'console.log(require("./dist/lib/inference/health.js").probeProviderHealth("openai-api"))'
{
  ok: true,
  probed: true,
  providerLabel: 'OpenAI',
  endpoint: 'https://api.openai.com/v1/models',
  detail: 'OpenAI endpoint is reachable at https://api.openai.com/v1/models.'
}

# the endpoint actually answered unauthenticated:
$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' https://api.openai.com/v1/models
HTTP 401

The probe sends no credential, so /v1/models returns 401; curlStatus === 0 (curl received a response) makes it ok: true regardless. A real deployment gets the same unauthenticated 401 here — the probe never authenticates — so this reachability signal is what drives the healthy line, independent of whether the key or model actually works.

Impact

A green status inference line can coexist with a model that cannot be reached: missing/invalid credential, a configured model that is not pulled (local) or not enabled (remote), the model backend down, or — as in #6635 — a broken in-sandbox upstream hop while the local gateway still serves /v1/models. The user sees "healthy" and looks elsewhere.

Scope / honesty

Reproduced by execution on ollama-local and openai-api (host-side probes), x86_64/WSL2, NemoClaw main (3de1de6b1). The anthropic-prod path @mikemason hit uses the same /v1/models reachability dispatch (established from the code, not separately executed). This is a diagnostic-accuracy gap independent of the gateway-supervisor topic #6635 was closed on (#6677); filing separately rather than reopening.

Suggested direction

The repo already has the right primitive: probeNvidiaKimiK26Health exercises /chat/completions. Extending a lightweight, authenticated model-invocation hop (a minimal completion, or at least an authenticated /v1/models call that confirms the configured model is present) to the general provider health path would make status reflect what OpenClaw actually depends on. The current remote probe is deliberately unauthenticated, so the fix is to add authentication + a configured-model/invocation check rather than to reinterpret its HTTP status; for local providers, the probe should likewise confirm the configured model is in /api/tags rather than accepting any well-formed (even empty) list.

Offer

Happy to send a PR generalizing the invocation-style probe (guarded to stay cheap and offline-friendly for local providers) with tests. Confirm the direction — full completion probe vs. auth+configured-model validation — and I'll implement it.

[mikemason]

Yesterday I investigated this a bit more with Claude, looking for potential enhancements I could contribute. The 2xx-4xx = OK unauthenticated-reachability pattern in health.ts/local.ts matches something I found independently while chasing the reboot-recovery bug (#6635), and I've got two things that might be useful additions.

1. The same bucketing bug exists in a second, separate probe. src/lib/actions/sandbox/connect-inference-route-probe.ts — the in-sandbox probe that status/doctor use to check https://inference.local/v1/models from inside the container — has the identical defect, still present verbatim on current main (checked against 20130dd4, today):

'case "$HTTP_CODE" in [2-4][0-9][0-9]) printf \'OK %s\' "$HTTP_CODE" ;; *) printf \'BROKEN %s\' "$HTTP_CODE" ;; esac',

Same consequence as you described for the host-side probes: a 401/403 (bad credential), 429 (quota), or 400/404 (bad request/model) all read OK here too. This is a different code path than the ones in your report, so it's evidence the pattern is systemic rather than confined to health.ts/local.ts.

Worth noting this specific probe's 2xx-4xx = OK scope looks like a deliberate design choice, not an oversight — #6192 (the issue that introduced this exact probe) was about the network path to inference.local being fundamentally intact (DNS/proxy/TLS-chain level), and answering that question correctly doesn't require distinguishing a 200 from a 401. So I'd frame this less as "another instance of the same bug" and more as "the same scope tradeoff, made independently in a second place, with the same accumulating cost you're describing" — which is maybe an argument for addressing it once, centrally, rather than probe-by-probe.

2. A second, structurally distinct gap on top of reachability-vs-invocability. This probe always runs in a freshly spawned sandbox exec shell, and OpenShell injects a correct CA trust bundle into every new exec unconditionally. So the probe's result reflects that fresh exec's environment, not the environment of the actual long-running gateway process already serving real traffic. If that process was started earlier with a different or missing environment (as it was on my system when Claude duct-taped a restart script together to workaround #6635), it can be failing every real call while the probe — a totally separate execution context — keeps reporting healthy.

I have an executed reproduction of exactly this, for anthropic-prod specifically (the path you noted you'd only confirmed by reading code, not running):

• Worked around the #6635 recovery gap by hand-starting the OpenClaw gateway without NODE_EXTRA_CA_CERTS.
• Every real model call failed: causeCode=SELF_SIGNED_CERT_IN_CHAIN.
• nemoclaw <sandbox> status reported both inference legs healthy throughout:

  Inference: healthy (https://inference.local/v1/models)
  Inference (upstream): healthy (https://api.anthropic.com/v1/models)
  

• Confirmed the trust-mismatch mechanism directly: an undici ProxyAgent request through the same proxy fails with SELF_SIGNED_CERT_IN_CHAIN with no CA supplied, and returns HTTP 200 with the CA bundle supplied — same cert, same proxy, different environment.

Worth flagging because it means your proposed fix (an authenticated invocation/configured-model check) fully closes the reachability-vs-invocability gap you're describing, but — at least for the in-sandbox route probe specifically — wouldn't by itself close this gap if the authenticated check still runs via a fresh exec rather than validating the actual serving process's environment.

I decided not to pursue the issue further yesterday because the code is very specific -- "2xx-4xx = OK" is doing something deliberate, and I wasn't sure if messing with it goes against a deliberate design intent for what this reachability check is supposed to do. One thing I did consider is that OpenClaw (and some other providers) themselves expose health check interfaces. OpenClaw does an LLM ping every 5 minutes, so it would notice if the LLM became unreachable, ran out of quota, etc. So one enhancement might be to add pluggable health checks to the existing NemoClaw flow -- for OpenClaw, also check its health, for Hermes, check its health (although Hermes has only very basic internal health checking, I think).

Environment for the above: NemoClaw 0.0.79, OpenShell 0.0.72 (docker driver), Ubuntu 24.04, OpenClaw agent, Anthropic provider.

[wscurran]

✨ Thanks for the detailed report and code pointers. This is a clear bug — the status command should verify model invocability, not just endpoint reachability.

---

Related open issues:

• #6635 [All Platforms][Recovery] Sandbox gateway cannot be recovered after a container restart: every managed recovery path requires the supervisor it is supposed to start

[Hokonoken]

Thanks @wscurran for the confirmation, and @mikemason — both of your findings are real additions, and the second one closes a hole in my own report.

On your finding 1: confirmed on today's main (3cccebf57) — the probe line is still there verbatim (src/lib/actions/sandbox/connect-inference-route-probe.ts:34). I also agree with your framing over my own original one: given #6192, that probe's 2xx-4xx = OK is a deliberate "is the network path intact" check, not a bug — the defect is that status renders its answer as if it meant "inference healthy". Which is exactly the argument for fixing the vocabulary once, centrally, instead of patching probe by probe.

On your finding 2: thank you for executing the anthropic-prod leg — my report had only traced that path in code, so your SELF_SIGNED_CERT_IN_CHAIN-everywhere-while-status-says-healthy run is the first executed proof of it in this thread. And you're right that it's structurally distinct: a probe running in a fresh sandbox exec can only ever attest that exec's environment, never the serving process's. An authenticated invocation check inherits that limit — it shrinks the lie, it doesn't eliminate it.

Given wscurran's confirmation, here's the scope I'd propose for a first PR, splitting along that structural line:

Stage A — what a fresh probe can honestly answer (this issue):

• A shared verdict vocabulary for the inference probes (two distinct facts instead of one bucketed "healthy"): network path reachable vs configured model invocable — the latter via an authenticated invocation of the configured model, so 401/403/429/404 stop reading as healthy.
• health.ts / local.ts verdicts move to "invocable"; the [All Platforms][CLI&UX] nemoclaw status/doctor report inference "healthy" by probing the upstream, not the inference.local route the agent uses #6192 route probe keeps its deliberate reachability semantics but gets rendered as what it is ("network path to inference.local: OK"), never as "inference healthy".
• Where a leg wasn't actually verified, status says so explicitly — same honesty pattern feat(messaging): telegram channels-status health probe #6887 just applied to the Telegram probe (Runtime health: not checked).

Stage B — what only the serving process can answer (follow-up issue, shouldn't block A):

• Your fresh-exec-vs-serving-process gap. The honest fix is process-level self-report rather than a better external probe — your pluggable-health idea (surfacing OpenClaw's own periodic LLM ping) looks like the right shape for it. Since you have the executed repro and the design instinct here, do you want to file/take that one? Happy to co-draft the issue text with your evidence instead, whichever you prefer — I don't want to trample a contribution you were already lining up.

If the maintainers are good with the Stage A scope, I'll take that PR (regression tests included: the 401/403/429 cases asserted against the new verdicts, plus a rendering test pinning that the route probe's answer can no longer surface as "healthy").

[mikemason]

Thanks Hoko! My situation is that I’m still trying to get useful with this project (feels like stepping onto a train moving at 100mph!) and am about to head away for a weekend trip with the kids. So no pending in-flight effort from me at this time - I wanted to see what “the maintainers” made of all this before proceeding. If you think the right next step is to draft an issue / discussion for Stage B, and you have energy around it, please go ahead, mention me, and I’ll weigh in when time allows. Or, if when I check back ~Monday it’s still a gap, I can work towards that myself.

…

On Wed, Jul 15, 2026 at 3:11 AM Hoko ***@***.***> wrote: *Hokonoken* left a comment (NVIDIA/NemoClaw#6846) <#6846 (comment)> Thanks @wscurran <https://github.com/wscurran> for the confirmation, and @mikemason <https://github.com/mikemason> — both of your findings are real additions, and the second one closes a hole in my own report. *On your finding 1:* confirmed on today's main (3cccebf) — the probe line is still there verbatim ( src/lib/actions/sandbox/connect-inference-route-probe.ts:34). I also agree with your framing over my own original one: given #6192 <#6192>, that probe's 2xx-4xx = OK is a deliberate "is the network path intact" check, not a bug — the defect is that status *renders* its answer as if it meant "inference healthy". Which is exactly the argument for fixing the vocabulary once, centrally, instead of patching probe by probe. *On your finding 2:* thank you for executing the anthropic-prod leg — my report had only traced that path in code, so your SELF_SIGNED_CERT_IN_CHAIN-everywhere-while-status-says-healthy run is the first executed proof of it in this thread. And you're right that it's structurally distinct: a probe running in a fresh sandbox exec can only ever attest *that exec's* environment, never the serving process's. An authenticated invocation check inherits that limit — it shrinks the lie, it doesn't eliminate it. Given wscurran's confirmation, here's the scope I'd propose for a first PR, splitting along that structural line: *Stage A — what a fresh probe can honestly answer (this issue):* - A shared verdict vocabulary for the inference probes (two distinct facts instead of one bucketed "healthy"): *network path reachable* vs *configured model invocable* — the latter via an authenticated invocation of the configured model, so 401/403/429/404 stop reading as healthy. - health.ts / local.ts verdicts move to "invocable"; the #6192 <#6192> route probe keeps its deliberate reachability semantics but gets rendered as what it is ("network path to inference.local: OK"), never as "inference healthy". - Where a leg wasn't actually verified, status says so explicitly — same honesty pattern #6887 <#6887> just applied to the Telegram probe (Runtime health: not checked). *Stage B — what only the serving process can answer (follow-up issue, shouldn't block A):* - Your fresh-exec-vs-serving-process gap. The honest fix is process-level self-report rather than a better external probe — your pluggable-health idea (surfacing OpenClaw's own periodic LLM ping) looks like the right shape for it. Since you have the executed repro and the design instinct here, do you want to file/take that one? Happy to co-draft the issue text with your evidence instead, whichever you prefer — I don't want to trample a contribution you were already lining up. If the maintainers are good with the Stage A scope, I'll take that PR (regression tests included: the 401/403/429 cases asserted against the new verdicts, plus a rendering test pinning that the route probe's answer can no longer surface as "healthy"). — Reply to this email directly, view it on GitHub <#6846?email_source=notifications&email_token=AABHOQHP3KMTIAINS3XEPD35E5DDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA4DONZTHA3KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4978877386>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABHOQCCFZ524QJ3VA2ZW5T5E5DDHAVCNFSNUABGKJSXA33TNF2G64TZHMYTCOBSGU2DOMBZGI5US43TOVSTWNBYHAYTSMZVGAYDFILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AABHOQF6MOUQCGC6EZOHVND5E5DDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA4DONZTHA3KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AABHOQAO3PRNNPOUKQWS7HD5E5DDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJXHA4DONZTHA3KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[Hokonoken]

Stage B is now filed as #7003, with an executed reproduction of the fresh-exec-vs-serving-process gap on current main (c1bda8069, ollama-local — so it confirms the class independently of any proxy/TLS-interception specifics): a hand-restarted gateway with a stripped environment serves fine and status shows all three inference legs healthy, while the same call replayed under the serving process's exact environment fails before TLS is even attempted (Could not resolve host: inference.local). It also captures OpenClaw's own [health-monitor] started (interval: 300s, ...) line, i.e. the self-report primitive your pluggable-health idea would surface.

@mikemason the finding and the design instinct are yours, and #7003 says so — weigh in there whenever suits you, on Monday or whenever time allows; if you want to take the implementation, it's yours, and I'm equally happy to support or review. Thanks again for everything you brought to this thread — and have a great weekend away with your kids ;)