Hermes MCP rebuild loses persisted managed intent on OpenShell v0.0.101

[apurvvkumaria]

Agent Diagnostic

• Loaded nemoclaw-skills-guide and the GitHub issue workflow.
• Monitored the canonical exact-main E2E run after PR #8660 merged the OpenShell v0.0.101 upgrade.
• Inspected the completed Hermes MCP job log directly.
• Confirmed the failure occurs in the product test during Hermes MCP rebuild/restore; a later artifact-upload timeout is secondary.
• Made no code or configuration changes.

Description

The canonical main E2E fails the Hermes MCP bridge scenario on the exact main candidate after the OpenShell v0.0.101 upgrade.

The failing stage is the test's rebuild without MCP host secret flow. The rebuild installs OpenShell 0.0.101, rebuilds the Hermes 0.19.0 image, restores state, and then refuses the gateway restart because the restored Hermes MCP configuration does not match the persisted managed intent.

Expected: rebuilding without the host MCP secret restores the registered managed MCP configuration, passes post-restore verification, and leaves the Hermes gateway healthy.

Actual: MCP reconciliation refuses the gateway restart, Hermes post-restore verification fails, and mutation containment prevents subsequent cleanup.

This is narrower than the otherwise-green Hermes lanes in the same run: Hermes E2E, Hermes rebuild, stale-base rebuild, and Hermes security posture all passed. The regression is specific to the Hermes MCP rebuild/restore path.

Reproduction Steps

1. Use NemoClaw main at ea39cc408227cdbffc12c5a625d14a7511867984.
   • This commit directly descends from the feat(openshell): upgrade managed runtime to v0.0.101 #8660 merge commit 6ad7ea6c2a54f1d41cf666d84a1bebf1ab6de70e.
2. Run the canonical E2E workflow with the mcp-bridge (hermes) scenario.
3. Allow the scenario to create and validate the Hermes MCP bridge.
4. Exercise the scenario's rebuildWithoutMcpHostSecret phase.
5. Observe the rebuild complete its image work and policy submissions, then fail the gateway restart/post-restore verification with managed-intent drift.
6. Observe cleanup rejected by the mutation-containment marker created after the failed reconciliation.

Evidence:

• Main E2E run 31346513232
• Failed Hermes MCP job 93334669243
• Candidate SHA: ea39cc408227cdbffc12c5a625d14a7511867984
• feat(openshell): upgrade managed runtime to v0.0.101 #8660 merge SHA: 6ad7ea6c2a54f1d41cf666d84a1bebf1ab6de70e

Environment

• Execution: canonical GitHub Actions main E2E
• Runner OS/architecture: Linux x86_64
• Node.js: pinned Node 22 workflow toolchain
• Container runtime: Docker
• NemoClaw candidate: ea39cc408227cdbffc12c5a625d14a7511867984
• OpenShell CLI: 0.0.101
• Hermes Agent: 0.19.0
• E2E target/shard: mcp-bridge / hermes
• Supervisor image: ghcr.io/nvidia/openshell/supervisor@sha256:b58be5e40c788977ffa0e8305a8cad9c656efdf1a3fe182582a00ca870bb0edb

Debug Output

mcp-bridge-hermes
Failure stage: rebuild without MCP host secret
OpenShell CLI: openshell 0.0.101
Hermes Agent: v0.19.0
Candidate SHA: ea39cc408227cdbffc12c5a625d14a7511867984

Logs

Error: hermes rebuild without MCP host secret failed: Rebuild sandbox 'e2e-mcp-hermes'
    Current:  Hermes Agent v0.19.0
    Target:   Hermes Agent v0.19.0

✓ openshell CLI: openshell 0.0.101
✓ Base image built
✓ Policy version 2 submitted
✓ Policy version 2 loaded

Failure layer: MCP reconciliation refusal - gateway restart failed for 'e2e-mcp-hermes'.
Hermes MCP config does not match persisted managed intent
Run `nemoclaw e2e-mcp-hermes mcp restart` to restore the managed MCP configuration, then retry.
If the sandbox has an old helper or missing runtime metadata, run `nemoclaw e2e-mcp-hermes rebuild --yes` instead.

✓ Policy version 3 submitted
✓ Policy version 3 loaded
Hermes post-restore verification failed for 'e2e-mcp-hermes'.

Cleanup then fails because sandbox mutation containment is active for the same sandbox generation. Separately, uploading the 158-file E2E artifact timed out with Failed to CreateArtifact: ETIMEDOUT; that upload failure happened after the product assertion and is not the primary cause.

Impact and Criticality

This prevents treating OpenShell v0.0.101 as fully qualified for Hermes MCP rebuild/restore. It does not invalidate the green core Hermes E2E, rebuild, stale-base rebuild, or Hermes security-posture results.

Related: #8649 reports older LKG/OpenShell 0.0.85 Hermes config-integrity quarantine and rebuild divergence. This issue records a new exact-main v0.0.101 automated reproducer with a managed MCP bridge and host-secret-removal rebuild path.

Checklist

• I searched existing issues; [DGX Spark][Hermes] current LKG: gateway restart quarantines agent; rebuild can leave registry/live OpenShell divergent #8649 is related but does not cover this exact v0.0.101 E2E path.
• Isolated rerun is pending to establish whether the failure is consistently reproducible outside the full concurrent E2E matrix.