`status` inference health can only attest a fresh exec's environment, never the serving process's — surface process-level (self-reported) health

[Hokonoken]

Problem Statement

This is Stage B of the scope split proposed in #6846 (comment): the structurally distinct gap @mikemason identified there, filed separately so it doesn't block the Stage A probe-vocabulary work.

Every inference health signal status/doctor can produce today is measured from a freshly spawned sandbox exec (buildSandboxInferenceRouteProbeArgs, src/lib/actions/sandbox/connect-inference-route-probe.ts:82, returns ["sandbox", "exec", ...]). The file itself documents why that exec is trustworthy: "OpenShell injects the per-sandbox trust bundle into each exec process" (connect-inference-route-probe.ts:23). That injection is exactly what makes the probe blind to the process it stands in for: the long-running agent gateway serving real traffic has whatever environment it was started with, and no probe running in a fresh exec can attest it.

The environment contract at stake is not just the CA bundle. On a live openclaw sandbox the serving process depends on, among others: NODE_EXTRA_CA_CERTS, SSL_CERT_FILE/CURL_CA_BUNDLE, HTTPS_PROXY/NODE_USE_ENV_PROXY (all egress goes through the in-sandbox proxy), and NODE_OPTIONS preloads. In the reproduction below, a serving process without that contract cannot even resolve inference.local — while every status leg stays green.

@mikemason hit this organically (#6846 discussion): after working around #6635 by hand-starting the gateway, NODE_EXTRA_CA_CERTS was missing, every real model call failed with SELF_SIGNED_CERT_IN_CHAIN, and status reported both inference legs healthy throughout. The reproduction below confirms the same class of failure on current main with a local provider — no corporate proxy or TLS interception involved — by emulating that recovery scenario.

Stage A (#6846: authenticated invocation verdicts) shrinks the reachable-vs-invocable gap but inherits this limit by construction: an authenticated invocation in a fresh exec is still an assertion about the fresh exec.

Reproduction (by execution, main @ c1bda8069, x86_64/WSL2, docker driver, ollama-local)

1. Healthy baseline. Fresh onboard --name stageb-demo --agent openclaw — all green, and the serving process carries the same injected trust/proxy contract as a fresh exec (the only difference in the compared env sets is the order of the NODE_OPTIONS preloads).

2. Enter the #6635 state, then recover by hand. Killed the in-container supervision loop (nemoclaw-start) and the gateway, then restarted the gateway the way a stranded user does after #6635 — by hand, without the injected contract (env -i HOME=... PATH=... TERM=... openclaw gateway run, emulating the environment loss @mikemason hit for real). The gateway comes up and serves:

$ openshell -g nemoclaw-18899 sandbox exec --name stageb-demo -- sh -c 'curl -s --max-time 5 -o /dev/null -w "gateway: HTTP %{http_code}\n" http://127.0.0.1:18791/'
gateway: HTTP 200

Its environment, read from /proc/<pid>/environ, is now three variables (HOME, PATH, TERM) — no CA bundle, no proxy, no NODE_OPTIONS preloads.

3. status reports healthy inference on every leg:

  Sandbox: stageb-demo
    Model:    llama3.2:1b
    Provider: ollama-local
    Inference: healthy (https://inference.local/v1/models)
    Inference (ollama backend): healthy (http://127.0.0.1:11434/api/tags)
    Inference (auth proxy): healthy (http://127.0.0.1:11435/api/tags)

4. The serving process cannot make the call status just vouched for. Same sandbox, same instant, same URL — only the environment differs:

[A] serving-process env:
curl: (6) Could not resolve host: inference.local
HTTP 000
[B] fresh-exec env:
HTTP 200

([A] replays the call under the hand-started process's exact environment; [B] is the same call in the fresh exec's injected environment. inference.local only resolves/routes through the injected egress-proxy config, so the broken serving process fails at name resolution before TLS is even attempted — an even earlier failure than the SELF_SIGNED_CERT_IN_CHAIN variant in #6846.)

The window is not exotic: any path where the serving process's env diverges from the current exec-injection contract can produce it — manual recovery after #6635, an env/CA change after the process started, or any supervisor-less restart. And it is silent by construction: the probes get a fresh, correct environment every time, so they can never see it.

Desired Behavior

• status/doctor gain a leg that reports what only the serving process can know, sourced from the process itself rather than from a fresh probe, e.g.:
  • Serving process (openclaw gateway): last model call OK (2m ago)
  • Serving process (openclaw gateway): failing since 06:54 (name resolution)
• Where no self-report is available, say so explicitly instead of staying silently green — the honesty pattern VDR 5 checklist: DGX Station & DGX Spark golden path #6743 and feat(messaging): telegram channels-status health probe #6887 already applied elsewhere in status (Runtime health: not checked).
• Rendering keeps the two kinds of facts distinguishable: "a fresh exec can invoke the model" (Stage A) vs "the process serving traffic can" (this issue).

Constraints and Non-goals

• Complements Stage A (nemoclaw status reports inference healthy on endpoint reachability, not model invocability — a green status can mask an unreachable model #6846); neither replaces the other. Non-goal: changing probe semantics — that's Stage A's scope.
• The self-report should be agent-pluggable, not OpenClaw-specific. The manifest already declares a per-agent health surface (health_probe, agents/openclaw/manifest.yaml:27), currently used by onboard/recovery as a liveness gate but never surfaced by status as serving-process health. Agents with weaker internals (hermes) degrade to not checked.
• Non-goal: NemoClaw running its own monitoring daemon. The point is to read what the serving process already knows.

Implementation Idea (non-binding)

The primitive already exists on the agent side. The hand-started (broken) gateway in the repro logs, verbatim:

2026-07-16T06:54:17.071+00:00 [health-monitor] started (interval: 300s, startup-grace: 60s, channel-connect-grace: 120s)

i.e. OpenClaw runs an internal health monitor on a 5-minute cycle (plus a heartbeat, intervalMs: 1800000). The 300s interval lines up with the "OpenClaw does an LLM ping every 5 minutes" behavior @mikemason described in #6846 (I observed the monitor running, not what it checks — that part is his report). A first cut could be:

• extend the agent manifest contract (health_probe → e.g. a self_report entry) declaring where/how the serving process exposes its own last health verdict;
• have status/doctor fetch and render that verdict with its age, replacing silence with not checked when the agent declares nothing.

@mikemason mentioned in #6846 he may want to work on this — the design instinct (pluggable health) is his; happy to support, review, or pair on it, and Stage A on my side won't collide with it.

[wscurran]

✨ Thanks for the detailed report with root cause analysis. The 256 MiB maxBuffer limit in backupSandboxState is a clear bottleneck for long-lived sandboxes. This will need a streaming or chunked backup approach to avoid the Node.js spawnSync buffer limit.

---

Related open issues:

• #6188 [macOS][Upgrade] v0.0.72 curl|bash upgrade loops forever — pre-upgrade backup-all aborts when a running sandbox's SSH backup times out

---

Related open issues:

• #6846 nemoclaw status reports inference healthy on endpoint reachability, not model invocability — a green status can mask an unreachable model
• #6635 [All Platforms][Recovery] Sandbox gateway cannot be recovered after a container restart: every managed recovery path requires the supervisor it is supposed to start

[mikemason]

#7007 has implemented the extra Serving process: not checked scaffolding and explicitly deferred the concrete probe contract ("response schema, probe protocol, and failure mapping"). Here's a design for that deferred piece. I'd like to take it if it's open.

The key decision: status should read a verdict the serving process already produced, never compute a fresh one. This is an environment-attribution bug, not a probe-thoroughness bug — the serving process runs with whatever environment it was started with, and any probe NemoClaw runs gets a fresh, always-correct exec environment, so it can never attest the process it stands in for. More thoroughness doesn't fix that (which is why Stage A's authenticated invocation inherits the same blindness). The one thing that can: read what the process already knows. OpenClaw's [health-monitor] (300s interval) is exactly that primitive.

Proposed contract — extend the manifest health_probe (agents/openclaw/manifest.yaml:27) with an optional self_report source declaring where the process exposes its last verdict:

health_probe:
  url: "http://localhost:18789/"
  port: 18789
  timeout_seconds: 30
  self_report:
    source: "http://localhost:18789/internal/health"   # served by the process itself
    fields: { verdict, checked_at, detail }

status/doctor fetch that verdict and render it with age, filling in the { checked: true, ... } state #7007 left open:

Serving process (openclaw gateway): last call OK (2m ago)
Serving process (openclaw gateway): failing since 06:54 (name resolution)

Agents that declare no self_report keep the not checked line #7007 already ships.
Open questions for maintainers:

1. Source of the verdict — the fragment above shows the dedicated-endpoint form, but that's not the only option and probably not the lightest. The least-surface alternative is enriching the existing health_probe response the gateway already serves (NemoClaw already knows that URL and calls it); others are a file the health-monitor writes, or a pinned-format monitor log line. Which fits the intended contract? (Load-bearing; drives the schema.)
2. Staleness — a verdict older than ~2× the interval: render stale distinctly from failing? A dead monitor and a failing call are different facts.

If the shape's roughly right I'll pick it up with regression tests (serving-process-unhealthy-while-probes-green). @Dongni-Yang — you did #7007; am I ok to pick up this implementation after you review the design?

[Dongni-Yang]

@mikemason This is the right frame, and the design is good — go ahead and take it.

You've nailed the actual defect: this is environment attribution, not probe thoroughness. A fresh sandbox exec always gets a correct environment, so it structurally cannot attest the long-running process it stands in for — which is exactly why Stage A's authenticated invocation would inherit the same blindness. Reading a verdict the serving process already produced is the one thing that closes it. #7007 deliberately left the { checked: true, ... } state open for precisely this contract, so this is the piece to build.

On your two open questions:

1. Verdict source — prefer enriching the existing health_probe response (your option b), but treat the OpenClaw side as a hard prerequisite, not an assumption. It's the least surface: NemoClaw already knows and calls that URL, it's one hop, and the verdict is authoritatively the process's own. A dedicated /internal/health endpoint adds a surface OpenClaw has to serve and version; a file re-imports the who-wrote-it/which-environment ambiguity we're trying to escape; a parsed log line is fragile. The load-bearing question is whether OpenClaw actually exposes the [health-monitor]'s last verdict on the response NemoClaw already fetches — today health_probe.url is the gateway root (:18789/), and I don't think it carries the monitor verdict yet. Let's confirm that on the OpenClaw side before locking the self_report schema. If the root can't carry it, the dedicated endpoint becomes the justified fallback — but I'd rather pin the contract to a capability we've verified exists.

2. Staleness — yes, render stale distinctly from failing. A dead monitor and a failing call are different facts and should read differently. Three terminal states off the fetched verdict:

• fresh + OK → last call OK (2m ago)
• fresh + failing → failing since <time> (<detail>)
• older than ~2× the monitor interval (~600s) → stale — monitor not reporting (last verdict 15m ago)

Keep self_report optional in the manifest so agents that don't declare it (Hermes, dcode, and anything with only basic internal health) keep the not checked line #7007 already ships — no regression for them.

serving-process-unhealthy-while-probes-green is exactly the regression that proves it. Please open it against #7003.

[mikemason]

@Dongni-Yang — checked, and the answer flipped my design comment above. Short version: OpenClaw already produces and exposes a last-verdict, but not from the subsystem I named. Verified against the shipped OpenClaw 2026.6.10 bundle.

Correction to my earlier comment

I attributed the verdict to OpenClaw's [health-monitor] on a 300s interval. That was two real things welded together:

• src/gateway/channel-health-monitor.ts is real and is 300s — but it is channel-scoped (Telegram/Slack/Discord connection liveness + restart), computes its verdict as a loop-local const, drops it on the healthy path, and only surfaces it as a log line on the restart path. It says nothing about inference. NemoClaw also ships healthMonitor: { enabled: false } in all five channel manifests, so it does not even run on a stock sandbox.
• The agent heartbeat is the inference liveness check. It is a full agent run: injects HEARTBEAT.md, calls the configured model, and expects a HEARTBEAT_OK token back. This is genuinely end-to-end.

So the premise holds — there is a process-produced verdict to read — but it comes from the heartbeat, not the health monitor.

What already exists

emitHeartbeatEvent retains the last event in module state (state.lastHeartbeat), timestamped, and getLastHeartbeatEvent() reads it back. It is exposed on the gateway as an RPC method literally named last-heartbeat (src/gateway/server-methods/system.ts):

"last-heartbeat": ({ respond }) => {
  respond(true, getLastHeartbeatEvent(), void 0);
},

Event shape is { ts, status, reason?, durationMs?, channel?, accountId? }. Statuses observed: ok-token, ok-empty, ran, failed (with reason carrying the error message), and skipped with a specific reason.

This means no upstream OpenClaw change is required. My earlier proposal to add a self_report endpoint is unnecessary for OpenClaw — the primitive ships today. That is the main benefit and it is a real one: status/doctor can fill in the { checked: true, ... } state #7007 left open by reading an existing RPC.

Drawbacks — and there are more than I expected

1. In-memory only. state.lastHeartbeat does not survive a gateway restart; it returns null until the first beat completes. So there is a genuine no-verdict-yet state, not just fresh/stale.
2. 30m default cadence. resolveHeartbeatIntervalMs falls back to "30m". Worst case a failure is up to 30m stale before it shows up, and the post-restart blind window is the same length.
3. skipped is not a health signal, and skips chain. Heartbeats skip for quiet-hours, requests-in-flight, cron-in-progress, and lanes-busy. The last event can therefore be a skipped for a long stretch while no conclusive verdict has been produced at all. Reading "the last event" is wrong; we need the last conclusive verdict, with skips tracked separately as a reason the verdict is aging.
4. Heartbeats can be turned off — per-agent config, or at runtime via the set-heartbeats RPC. Then no verdict is ever produced, and disabled must render distinctly from unknown.
5. RPC, not HTTP. last-heartbeat is a gateway method, not a curl-able GET like the existing health_probe.url. NemoClaw needs to speak the gateway protocol including auth, which is a heavier client than the current probe path.
6. The signal only exists because the agent is paying for it. Heartbeat is on by default for the default agent (isHeartbeatEnabledForAgent falls through to enabled when unconfigured), at 30m, making real model calls. Reading the verdict is free; producing it is not. Worth being explicit that this design consumes a side effect of a billed background activity rather than introducing one.

So the render states are at least four, not two: ok (age), failing since T (reason), stale / no conclusive verdict in N, and disabled.

On generalizing beyond OpenClaw

Being straight about this: the transport does not generalize. For OpenClaw it is a gateway RPC; for another agent it would be a file, a log line, or a different endpoint. Every agent needs its own adapter.

I do not think that sinks it, because it is what manifest.yaml already does — install_method, version_command, health_probe are all per-agent specifics behind a shared contract. The split would be:

• Per agent: a small adapter that answers "where does this agent keep its last verdict, and how do I read it."
• Shared: the normalized verdict schema, the staleness/age logic, and the four-state rendering above.
• Agents that declare nothing: keep the Serving process: not checked line feat(sandbox): add serving process health leg to status and doctor #7007 already ships.

That is one adapter per agent and one renderer for all of them. But it is worth deciding explicitly whether that trade is wanted before I build it, since it is a standing per-agent maintenance cost rather than a one-off.

Questions

1. Is the adapter-per-agent split above the shape you want, or would you rather this stay OpenClaw-specific until a second agent actually needs it?
2. For the verdict source — read last-heartbeat over gateway RPC, or would you prefer NemoClaw not take a gateway-protocol dependency in status/doctor and instead push for the verdict to be mirrored somewhere cheaper to read?
3. Given the added complexity here -- 4 result states, indeterminate outcomes, needing to speak gateway RPC, needing an adapter-per-agent -- is this still worth pursuing, or is it landing in "too hard, too compromised" territory and we should abandon it?

Happy to implement if you still think this is worthwhile, and can settle questions 1 and 2.

[Dongni-Yang]

@mikemason — great correction, and thanks for being straight about the drawbacks. I checked the shipped sandbox bundle and the answers change two of your three questions, both in favor of proceeding.

Your drawback 5 (RPC = heavier client) dissolves: OpenClaw's own CLI already reads the verdict. The shipped sandbox image carries both:

• openclaw system heartbeat last — "Show the last heartbeat event" (exactly the getLastHeartbeatEvent() / last-heartbeat RPC you found)
• openclaw health --json --timeout <ms> — "Fetch detailed health from the running gateway"

(Verified on the openshell/sandbox-from image with OpenClaw 2026.5.22; you verified the RPC on 2026.6.10, so it spans both.) So status/doctor never speak the gateway protocol: run the agent's CLI via sandbox exec, the same way version_command already runs today (terminal-version-drift.ts:93 execs the manifest-declared command via sandbox exec -- sh -lc). The CLI carries its own gateway auth.

That transport is also attribution-clean, and the distinction is worth pinning in the design doc: the environment bug taints producing a verdict in a fresh exec, never reading one. The exec is a courier; the process answering the RPC — and the process that produced the heartbeat by making a real model call in its own environment — is the serving process. In your repro, the hand-started gateway would answer last-heartbeat itself with failed (name resolution) while every probe leg stayed green: exactly the missing line.

Q1 (adapter-per-agent vs OpenClaw-only): neither per-agent code adapters nor OpenClaw hardcoding — make the per-agent piece data, like the manifest already does for version_command/health_probe:

​yaml health_probe: ... self_report: command: "openclaw system heartbeat last" # run via sandbox exec, output normalized kind: heartbeat-event ​

Shared side: the normalized verdict schema, staleness logic, and rendering. Agents that declare nothing keep #7007's not checked. That's one YAML stanza per agent, not a standing per-agent code cost — the trade you flagged mostly disappears.

Q2 (verdict source): prefer openclaw health --json if its payload includes the last heartbeat (one call, versioned JSON — please verify its output shape against a live gateway first); otherwise system heartbeat last. If heartbeat last turns out text-only, the parse format is the one fragility left — check for a --json flag, and if missing that's a small upstream ask (a flag, not a new endpoint). My earlier "enrich the health_probe HTTP response" suggestion is superseded — no upstream change is needed at all. If the CLI can't reach the gateway, that maps to the existing gateway-down rendering, not to this leg.

Q3 (still worth it): yes. The four states aren't incidental complexity — they're the honest rendering of four genuinely different facts, the same honesty pattern #6743/#6887 established. The thing that made this heavy was the gateway-protocol client, and it's gone. On the drawbacks:

• in-memory / restart (1): null → no verdict yet, rendered distinctly from stale — honest about the post-restart blind window.
• 30m cadence (2): always render age; stale threshold ≈ 2× the configured heartbeat interval read from agent config, not hardcoded — my earlier ~600s figure was derived from the misattributed 300s monitor, supersede it.
• skip chains (3): agreed — normalize to the last conclusive verdict; skips only age it and can surface as the aging reason.
• disabled (4): distinct disabled render; and we only ever read — NemoClaw never enables heartbeats itself, which also answers (6): this consumes an existing billed signal, it doesn't add one.

So: take it, with serving-process-unhealthy-while-probes-green as the pinned regression (your repro is the perfect fixture) plus one test per render state.

@apurvvkumaria @ericksoa — this converged on a design that adds a new status/doctor leg (serving-process self-reported health via a manifest-declared self_report command). Since the issue is needs: design and @mikemason is ready to implement, could one of you sanity-check the contract above before he starts? The two decisions to ratify: manifest-declared read command (vs per-agent code adapters), and the four-state rendering (ok (age) / failing since T / no conclusive verdict in N / disabled).

[apurvvkumaria]

The direction makes sense, but before resolving needs: design, can we pin the following contract details?

1. Which exact OpenClaw command and machine-readable schema will NemoClaw consume? Please provide representative output for success, failure, skipped, disabled, and no-verdict-after-restart across supported OpenClaw versions.
2. Does the source retain the last conclusive verdict, or only the latest event? If a skipped event overwrites the prior success/failure, how can NemoClaw truthfully report the last conclusive result without an upstream change?
3. How will we distinguish disabled, never run, skipped, stale, and failed?
4. Where will the effective heartbeat interval come from for staleness calculation, including runtime configuration changes?
5. Which states affect status/doctor exit codes?
6. What redaction and size limits apply to heartbeat failure reasons in text and JSON?
7. Please confirm that status only reads an existing verdict and never enables or triggers a billed heartbeat.