Canonical provider credential resolution via separate --provider/-m flags sends mask_secret() output as literal API key (401 Invalid API key)

[mill101]

Investigation Summary

• hermes -z ... --provider opencode-zen -m <model> fails with a generic "no final response was produced"; no further detail reaches the terminal.
• agent.log for this one-shot invocation stops right after plugin discovery — the conversation-turn/API-call entries that normally follow never appear there.
• The real cause was only visible in the auto-generated per-request dump at ~/.hermes/sessions/request_dump_*.json: reason: non_retryable_client_error, error.type: AuthError, 401 Invalid API key.
• The Authorization header actually sent is not the real credential — it is mask_secret()'s display-only masked rendering of the unresolved openshell:resolve:env:OPENCODE_ZEN_API_KEY placeholder string, confirmed by exact reproduction (see Debug Output).
• Workaround confirmed working: invoke with a single -m "<provider>/<model>" argument instead of separate --provider/-m flags — same credential, same target, succeeds and returns a correct response.

Description

Invoking a canonical custom provider (e.g. opencode-zen) via separate --provider and -m CLI flags fails to authenticate. The credential resolution for this code path never substitutes the real secret for the openshell:resolve:env:<VAR> placeholder stored in .env; instead, the raw placeholder string appears to get passed through the display-only mask_secret() helper (normally used by hermes config/status/dump for safe display), and that masked output is sent as the literal Authorization bearer token. The origin naturally rejects it with 401, which surfaces to the user only as an opaque "no final response was produced".

Expected: either the real credential resolves correctly for this flag combination, or resolution failure produces a clear, actionable error instead of silently sending a masked placeholder as if it were a real key.

Reproduction Steps

13:52 2026/07/22

1. Register a canonical provider (e.g. opencode-zen) with a real API key stored via the openshell:resolve:env:<VAR> placeholder in .env (OPENCODE_ZEN_API_KEY=openshell:resolve:env:OPENCODE_ZEN_API_KEY).
2. Run: nemoclaw <sandbox> exec -- hermes -z "test" --provider opencode-zen -m nemotron-3-ultra-free
3. Observe: hermes -z: no final response was produced; treating the run as failed.
4. Inspect ~/.hermes/sessions/request_dump_<timestamp>.json for the matching timestamp — reason is non_retryable_client_error, error.type is AuthError, error.message contains 401 ... Invalid API key.
5. In that same dump, request.headers.Authorization is 22 characters. Locally computing mask_secret("openshell:resolve:env:OPENCODE_ZEN_API_KEY", head=8, tail=4) produces "openshel..._KEY" — concatenated as "Bearer openshel..._KEY", this is an exact match for the value actually sent.
6. Compare: same session, same credential, same model, invoked instead as nemoclaw <sandbox> exec -- hermes -z "test" -m "opencode-zen/nemotron-3-ultra-free" (no --provider flag) — succeeds, returns a correct response.

Environment

• OS: Windows 11 + WSL2 (Ubuntu)
• NemoClaw: v0.0.79 (source), OpenShell 0.0.72
• Docker: sandbox container nemoclaw-sandbox-local:my-hermes-*, 2 days uptime
• hermes 0.17.0, openai_sdk 2.24.0 (from hermes dump)
• Credential registered via standard openshell provider create/update --credential using the openshell:resolve:env: placeholder pattern

Debug Output

nemoclaw debug --quick --sandbox my-hermes (relevant excerpts, hostname redacted):

═══ System ═══
Linux <redacted-hostname> 6.6.114.1-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC x86_64 GNU/Linux

═══ Docker ═══
CONTAINER ID   IMAGE                                            STATUS       NAMES
a01e1f15ba67   nemoclaw-sandbox-local:my-hermes-<sandbox-uuid>   Up 4 hours   openshell-my-hermes-<sandbox-uuid>

═══ OpenShell ═══
Server Status
  Gateway: nemoclaw-8990
  Status: Connected
  Version: 0.0.72
NAME       CREATED              PHASE
my-hermes  2026-07-19 06:48:37  Ready
Sandbox:
  Phase: Ready
  Policy source: sandbox
  Revision: 12

Local reproduction of the mask leak (no real secret involved — input is the literal, non-sensitive placeholder text, not a resolved credential):

>>> from agent.redact import mask_secret
>>> mask_secret("openshell:resolve:env:OPENCODE_ZEN_API_KEY", head=8, tail=4)
'openshel..._KEY'

hermes_cli/config.py's load_env() does a plain line.partition('=') parse of .env with no handling of the openshell:resolve:env: prefix — real substitution is expected to happen before get_env_value() reads it. For this provider's --provider/-m-flags path, that substitution does not happen, so get_env_value() falls through to the raw file parse and returns the literal unresolved placeholder — which then appears to reach mask_secret() and gets used as the literal credential instead of triggering a clean resolution error.

Logs

`~/.hermes/sessions/request_dump_<timestamp>.json` (values illustrative except the Authorization derivation above, which is exact):


{
  "timestamp": "2026-07-22T04:01:05.868029",
  "reason": "non_retryable_client_error",
  "request": {
    "method": "POST",
    "url": "https://opencode.ai/zen/v1/chat/completions",
    "headers": {
      "Authorization": "Bearer openshel..._KEY",
      "Content-Type": "application/json"
    }
  },
  "error": {
    "type": "AuthError",
    "message": "Error code: 401 - {'type': 'error', 'error': {'type': 'AuthError', 'message': 'Invalid API key.'}}",
    "status_code": 401
  }
}


`agent.log` for this invocation shows plugin discovery completing normally, then nothing further — no conversation-turn or API-call entries at all, unlike the long-running gateway process's log. The `request_dump` file was the only place this failure's real cause was visible.

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[Dongni-Yang]

Triage notes — verified against hermes-agent source (github.com/NousResearch/hermes-agent at v2026.6.19 = 0.17.0 as reported, v2026.7.1 = 0.18.0 = current NemoClaw pin, and HEAD 0.19.0), NemoClaw origin/main, and the #6379 history.

Thanks for the unusually deep investigation — the split-flag divergence you found is real, but two links of the causal chain need correcting, and the stack you reproduced on predates the fix for the underlying failure mode.

1. What actually diverges (confirmed in source). An explicit --provider overrides the model.provider: "custom" + proxy base_url + sk-OPENSHELL-PROXY-REWRITE configuration NemoClaw writes for inference (src/lib/actions/inference-set.ts, patchHermesInferenceConfig). With --provider opencode-zen, hermes takes its canonical-provider path (hermes_cli/runtime_provider.py resolve_requested_provider → _resolve_api_key_provider_secret in hermes_cli/auth.py): it reads OPENCODE_ZEN_API_KEY out of ~/.hermes/.env — which inside a sandbox is by design the unresolved openshell:resolve:env:… placeholder — and targets the provider's public registry base_url, leaving the OpenShell-configured route entirely. The combined -m "opencode-zen/nemotron-3-ultra-free" form keeps model.provider = "custom" and stays on the proxy route, which is why it works. (Your load_env() reading is correct — plain partition('='), no placeholder semantics — but substitution is never supposed to happen in-sandbox: placeholders are resolved at the OpenShell L7 egress boundary, see docs/about/ecosystem.mdx and agents/hermes/validate-env-secret-boundary.py, which explicitly allows unresolved openshell:resolve:env: values at runtime.)

2. The mask_secret() claim doesn't hold at source level. There is no mask_secret(..., head=8) call site anywhere in hermes (default is head=4; the log redactor uses head=6). The exact openshel..._KEY 8/4 rendering matches the display-only startup banner print(f"🔑 Using API key: {key_used[:8]}...{key_used[-4:]}") in agent/agent_init.py — and a few lines above it the OpenAI client is constructed with the full raw value (client_kwargs["api_key"]). Hermes also masks Authorization values when persisting logs/dumps, so the 22-character string in request_dump_*.json is a redacted rendering of the stored header, not evidence of the wire content. The wire almost certainly carried the intact 42-char placeholder — i.e. the same literal-placeholder-as-bearer failure mode as #6379, just reached via a different path. (No packet capture was taken, so this is source-flow analysis, not a capture.)

3. Staleness — the reproduced stack predates the fix for that failure mode. OpenShell 0.0.72 is the vulnerable version root-caused in #6379 (NVIDIA/OpenShell#2161). OpenShell 0.0.85 ships the fix (NVIDIA/OpenShell#2162 — fail closed when credential placeholders cannot be rewritten), supported since NemoClaw v0.0.85 (#6726); current release is v0.0.90, and the hermes pin moved to 0.18.0 (#6507). Note that 0.18.0 also changed this exact resolver to prefer .env over process env (get_env_value_prefer_dotenv), so 0.17.0 behavior does not predict 0.18.0 behavior on this path. Could you re-test both invocations on NemoClaw v0.0.90 + OpenShell 0.0.85? Expected outcomes: the split-flag path either works (egress rewrite now applies) or fails closed with an explicit credential error — either would supersede the silent 401 reported here.

4. What remains NemoClaw-actionable if the split-flag path still fails on the current stack: the in-sandbox hermes entrypoint is NemoClaw's argv wrapper (agents/hermes/hermes-wrapper.py, installed over the real binary), which already performs argv translation for one upstream defect (#5254) and already allowlists --provider/-m. Translating --provider X -m Y into the known-good combined -m "X/Y" form (or failing fast with an actionable message when a canonical provider's credential resolves to a placeholder) would keep split-flag invocations on the proxy route. The error-visibility half of this report is upstream: hermes_cli/oneshot.py calls logging.disable(CRITICAL) — which, contrary to its own comment, also silences the agent.log file handler (explaining the log stopping after plugin discovery) — and redirects stdout/stderr to devnull, so an internally-caught 401 can only surface as the generic "no final response was produced". That deserves an upstream issue against NousResearch/hermes-agent regardless of the retest outcome.

If your retest on v0.0.90 / OpenShell 0.0.85 still produces a silent 401, I'd propose the wrapper argv translation above as the NemoClaw-side fix.

[mill101]

@Dongni-Yang Thanks — I retested on the stack you asked for. Summary: upgraded successfully, but OpenCode Zen still fails, and the retest clarified that the real blocker is a catch‑22 rather than the credential-resolution path alone.

Environment (retested)

• nemoclaw v0.0.90, host OpenShell 0.0.85, in-sandbox openshell-sandbox 0.0.85 (confirmed the sandbox itself upgraded, not just the host CLI), Hermes Agent v0.18.0.
• Upgrade done in place via nemoclaw my-hermes rebuild (no destroy needed).

Retest result

Both invocation forms still end in 401 Invalid API key when talking to https://opencode.ai/zen/v1. Digging into why with the network policy in two modes made the underlying problem clear — it's a genuine catch‑22 between credential rewrite and TLS:

opencode.ai egress policy	TLS to opencode.ai	openshell:resolve:env: credential rewrite	Result
protocol: rest / access: full (L7 MITM)	NET:FAIL — Cloudflare still resets the proxy's re-originated handshake on 0.0.85	happens at the L7 boundary	Connection error
access: full + tls: skip (raw L4 passthrough)	reaches opencode.ai fine, no reset	skipped — the proxy never decrypts, so it can't rewrite the Authorization header	401 Invalid API key

So the credential fail-closed hardening in 0.0.85 (#6379) is in place, but it can't help here: to inject the credential the proxy must terminate TLS (L7), and the L7 re-originated handshake is exactly what Cloudflare rejects (the still-open TLS re-origination issue you referenced for OpenShell). tls:skip avoids the reset but by definition also skips the header rewrite. Neither mode yields a working request.

On the "combined form works" point

In this environment the combined -m "opencode-zen/<model>" form does not reach opencode.ai. It stays on model.provider: custom + the inference.local proxy route, and that route forwards to whatever the NemoClaw inference route is set to (here gemini-api) — the OCSF log shows the chat completion going to generativelanguage.googleapis.com, not opencode.ai. The opencode-zen/ model-id prefix doesn't repoint the route. And nemoclaw inference set --provider doesn't accept opencode-zen as a kind, so the managed route can't be pointed at OpenCode Zen either (only compatible-endpoint → opencode.ai, which then hits both the SSRF-guard issue in #7324 and the same L7 TLS reset).

Net

On the current stack the only thing that would make OpenCode Zen (or any Cloudflare-fronted, credential-required endpoint) reachable from inside a sandbox is the OpenShell-side fix that makes access: full a true CONNECT passthrough / stops the origin from rejecting the proxy's re-originated TLS — i.e. the separate OpenShell issue you mentioned. Happy to retest again once that lands. Everything else on the upgraded stack (Gemini via the managed route, DGX over the LAN, Slack) works.

[kagura-agent]

I'd like to work on this. The root cause is that --provider <name> -m <model> (separate flags) bypasses the OpenShell proxy rewrite path and reads the raw openshell:resolve:env: placeholder from .env. The combined -m "provider/model" form works because it routes through the config's sk-OPENSHELL-PROXY-REWRITE sentinel.

Proposed fix: in agents/hermes/hermes-wrapper.py, translate separate --provider + -m flags into the combined -m "provider/model" form, following the existing _translate_resumed_oneshot pattern. This ensures credential resolution always goes through the proxy rewrite path.

[mill101]

@kagura-agent Thanks for offering to pick up the wrapper translation. One important scope clarification so this doesn't get mistaken for a fix for the original OpenCode Zen use case:

The --provider X -m Y → -m "X/Y" argv translation is reasonable credential-routing hygiene, but on its own it does not make OpenCode Zen (or any Cloudflare-fronted, credential-required endpoint) reachable from inside the sandbox. Two independent reasons, both verified on the upgraded stack (nemoclaw v0.0.90, in-sandbox OpenShell 0.0.85):

1. The combined form doesn't reach opencode.ai at all in this setup. Keeping model.provider: custom routes through inference.local, which forwards to whatever the NemoClaw inference route is set to (here gemini-api). OCSF logs show the chat completion going to generativelanguage.googleapis.com, not opencode.ai — the opencode-zen/ model-id prefix doesn't repoint the route. And nemoclaw inference set --provider doesn't accept opencode-zen as a kind, so the managed route can't be pointed at it either.

2. Even reaching opencode.ai, there's a catch‑22 between credential rewrite and TLS (detailed in my previous comment): L7 termination is required to rewrite the openshell:resolve:env: placeholder, but the L7 re-originated TLS is what Cloudflare resets; tls: skip avoids the reset but then skips the rewrite → 401.

So the wrapper translation is fine to land as general hygiene for the split-flag credential path, but the actual OpenCode Zen blocker is the OpenShell-side TLS re-origination behavior (the separate OpenShell issue @Dongni-Yang referenced). I'd suggest not closing this on the translation alone. Happy to retest once the OpenShell TLS fix lands.

[kagura-agent]

@mill101 Thanks for the detailed clarification — that makes complete sense. I'll scope my PR strictly to the split-flag credential-routing hygiene (--provider X -m Y → combined form translation) and keep the issue open for the OpenShell TLS re-origination fix. Agreed that the argv normalization alone doesn't unblock the OpenCode Zen path.

[mill101]

@kagura-agent The PR scope itself (#7434) looks right and matches what we agreed on — just the wrapper translation, no attempt to solve the underlying transport issue.
One thing worth fixing before merge though: the PR body uses "Closes #7361", which will auto-close this issue on merge. That contradicts what we just agreed a few comments up — that this issue should stay open until the actual OpenCode Zen blocker (OpenShell-side TLS re-origination, catch-22 with credential rewrite) is resolved. The flag-merge fix is good hygiene but it doesn't touch that problem at all.
Could you change it to something like "Related to #7361" (or "Contributes to") instead of "Closes", so merging #7434 doesn't close this out from under the still-open transport issue? Happy to reopen it myself if it does get auto-closed, just flagging now to avoid the extra step.