[Hermes][Sandbox] managed gateway restart can health-timeout after the supervised gateway exits

[ericksoa]

Problem

With NemoClaw 0.0.92, a managed Hermes gateway restart can stop the existing gateway cleanly but never produce a healthy replacement:

nemohermes fbroggini-hermes gateway restart
Failure layer: health timeout - gateway restart failed for fbroggini-hermes. GATEWAY_HEALTH_TIMEOUT

The gateway log shows the old process receiving SIGTERM, draining immediately, stopping the API server, and exiting with code 1 so a supervisor can revive it. Its parent is the expected managed supervisor:

parent_name=bash parent_cmdline="bash /usr/local/bin/nemoclaw-start"
Gateway stopped by an unexpected signal — persisting gateway_state=running
Exiting with code 1 (signal-initiated shutdown without restart request) so systemd Restart=on-failure can revive the gateway.

No replacement becomes healthy before the host-side restart controller times out.

Scope and current evidence

Confirmed:

• /usr/local/bin/nemoclaw-start is present as the parent, so this is not the no-supervisor SUPERVISOR_UNAVAILABLE case in [WSL2][Hermes][Sandbox] nemoclaw-start supervisor never invoked on custom-gateway-port sandbox — mcp add/recover/gateway restart fail with SUPERVISOR_UNAVAILABLE despite a healthy, actively-serving gateway process #7142.
• The old supervised Hermes process receives SIGTERM and completes teardown cleanly.
• The controller reaches GATEWAY_HEALTH_TIMEOUT before a replacement gateway and its required surfaces become healthy.

There is also a latent supervisor defect worth fixing and regression-testing: refresh_hermes_supervised_child_pids() does not explicitly return success. Under set -e, an absent optional dashboard log-tail process can make mark_hermes_gateway_stopped() terminate the supervisor before replacement launch. This is a plausible failure path, but the available log does not prove it caused this particular incident; preparation, launch, gateway health, auxiliary readiness, or crash quarantine could also fail while producing the same outer timeout.

Acceptance criteria

• Optional gateway/dashboard log bookkeeping explicitly succeeds when an optional tail is absent and restores missing log tails when appropriate.
• The same managed supervisor remains alive while the old gateway PID is replaced by a new healthy PID.
• Restart failure evidence distinguishes runtime preparation, launch failure, supervisor loss, gateway-health failure, auxiliary-health failure, and crash quarantine.
• Failure/debug output includes supervisor and gateway PIDs plus a bounded, redacted excerpt from /tmp/nemoclaw-start.log.
• Fault tests cover a missing dashboard tail, failed gateway launch, listener-health failure, and crash quarantine.
• An exact-head live Hermes restart E2E succeeds without rebuilding the sandbox or falling back to a detached/manual gateway launch, with API, dashboard, and forwards healthy afterward.

Related work

• Impossible to restart hermes gateway if it ever stops #2426 / fix(sandbox): add host-mediated gateway restart #5874 introduced the supported host-mediated restart command.
• [All Platforms][CLI&UX] config set --restart hangs on the gateway restart — config written but never applied to the running agent #6868 / fix(gateway): respawn openclaw gateway after a managed restart (#6868) #6910 fixed the distinct OpenClaw clean-exit respawn path.
• [WSL2][Hermes][Sandbox] nemoclaw-start supervisor never invoked on custom-gateway-port sandbox — mcp add/recover/gateway restart fail with SUPERVISOR_UNAVAILABLE despite a healthy, actively-serving gateway process #7142 is the distinct Hermes no-supervisor/custom-port case.
• nemohermes sandbox-name rebuild fails when the base image was built with NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD=1 #7477 / fix(cli): rebuild sandboxes from locally built base images #7481 track the independent local-base rebuild failure reported alongside this incident.

[ericksoa]

Investigation update against current main (f2e81b8a3):

We confirmed a deterministic latent supervisor-exit path that matches the outer failure boundary, although the supplied logs do not prove that the reporter hit this exact condition.

• refresh_hermes_supervised_child_pids() ends with the optional dashboard-log identity check and has no explicit successful return. If that optional tail is absent or no longer current, the function returns 1.
• mark_hermes_gateway_stopped() calls it bare under the script-wide set -euo pipefail contract.
• The managed restart loop calls mark_hermes_gateway_stopped() immediately after waiting for the old gateway, before classifying the authorized exit and relaunching it (lines 2931-2945). A missing dashboard tail can therefore terminate PID 1 after the clean shutdown and before replacement launch.

The implementation lane is adding a production-real-shell regression that makes the optional dashboard tail absent and proves the supervisor reaches the post-stop relaunch path. We are also tracing the remaining preparation, launch, health, auxiliary-readiness, and quarantine paths so the final diagnostics do not collapse every failure into GATEWAY_HEALTH_TIMEOUT.

[ericksoa]

Diagnostic refinement after tracing the managed controller:

The missing-dashboard-tail set -e path is a real adjacent supervisor bug, but it is not the leading explanation for the reported GATEWAY_HEALTH_TIMEOUT. During await-replacement, the controller re-proves the pinned supervisor on every candidate scan (_gateway_candidates, called by _wait_for_healthy_gateway). If PID 1 exits, that proof fails as SUPERVISOR_UNAVAILABLE rather than exhausting the health budget.

For the exact reported code, the stronger interpretation is: the pinned supervisor remained alive for the recovery budget, but no replacement candidate satisfied both Hermes gateway health and required auxiliary health. Current leading paths are runtime preparation retries, launch/unproven-child quarantine, repeated replacement health failure/crash quarantine, or API relay repair failure. The in-sandbox supervisor records those distinctions in /tmp/nemoclaw-start.log; the host restart failure currently does not collect that evidence.

The PR will keep the explicit-return regression as a bounded adjacent fix, but the incident-facing work is now centered on safely surfacing that supervisor evidence and preserving stage-specific classification.

[ericksoa]

Implementation is active on a current-main branch and does not depend on an upstream Hermes change. Local regression coverage now proves the set -e recovery fix, auxiliary log-stream repair, exact managed-control error contracts, bounded/redacted diagnostics, and host-side failure classification; the initial focused suites pass 47 integration tests and 18 CLI tests.

An independent sensitive-path review caught one installed-topology mismatch before PR: the root helper expected /tmp/nemoclaw-start.log to be root-owned, while the managed nemoclaw-start supervisor creates it under the sandbox UID. The patch is being tightened so the diagnostic reader accepts only the UID of the exact re-proven supervisor, fails closed on supervisor/file churn, and distinguishes gateway-health from auxiliary-health events. Installed-topology and adversarial filesystem/injection tests are being added before the PR is opened.

[ericksoa]

Implementation is now published at exact commit baffafbcfb918d9ebef3c56149fa0b71f2ace0e1 (GitHub Verified) and is ready for a PR. The final independent nine-category sensitive-path review is GO with no open findings: the diagnostic log owner is bound to the exact re-proven supervisor UID; file/process churn, symlink/hardlink/FIFO, wrong owner/mode, mutation, arbitrary text, controls, suffixes, secrets, and oversized lines all fail closed; gateway-health and auxiliary-health events remain distinct fixed full matches. Final local evidence is 53/53 integration tests, 18/18 CLI tests, ShellCheck, Python compile, CLI typecheck, and npm run check:diff.