Skip to content

fix(mcp): surface rebuild failures and allow forced recovery from incomplete destroy state - #6451

Closed
sauravdev wants to merge 1 commit into
NVIDIA:mainfrom
sauravdev:fix/6376-mcp-destroy-recovery
Closed

sauravdev wants to merge 1 commit into
NVIDIA:mainfrom
sauravdev:fix/6376-mcp-destroy-recovery

Conversation

@sauravdev

@sauravdev sauravdev commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes the two operator-facing gaps reported in #6376: nemoclaw <name> rebuild exited 1 with no message when an incomplete MCP destroy transaction was present, and the documented recovery (mcp remove <server> --force) was itself blocked by the same guard, leaving full sandbox destruction as the only way out.

Related Issue

Fixes #6376

Changes

  • src/lib/actions/sandbox/rebuild-preflight-confirmation.ts: the CLI-mode bail() previously discarded its message and called process.exit(code) bare. It now prints the redacted failure reason to stderr (Rebuild failed: …) before exiting, so the destroy-marker refusal — and every other rebuild bail — is diagnosable.
  • src/lib/actions/sandbox/mcp-bridge-state.ts: assertMcpDestroyNotPending now names the forced-remove recovery command (with a real server name when one is registered); added hasMcpDestroyMarkers and clearMcpDestroyMarkersIfNoBridges helpers.
  • src/lib/actions/sandbox/mcp-bridge-remove.ts: mcp remove <server> --force now runs under destroy markers, adopting the incomplete transaction's cleanup manifest for that server (announced on stdout). The durable markers are cleared once no bridge entries remain, including the forced-remove paths for a prepared add and for a server that is no longer registered.
  • Tests: 4 new cases in test/mcp-destroy-lifecycle.test.ts (plain remove still refused with the recovery hint; forced remove of the last entry clears markers; markers persist while other entries remain; bare markers with no entries are cleared), plus 2 new cases in rebuild-preflight-confirmation.test.ts pinning the bail message-printing and throwing contracts.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: error-message and recovery-path fix; the improved guard message itself documents the recovery command. Happy to add a troubleshooting docs entry as a follow-up if maintainers prefer.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: requesting maintainer sensitive-path review on this PR (sandbox lifecycle / MCP state).
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Verification detail: npm run build:cli clean; test/mcp-destroy-lifecycle.test.ts 24/24, rebuild-preflight-confirmation.test.ts 11/11, plus rebuild-flow, rebuild-destroy-phase, mcp-provider-ownership, mcp-add-crash-consistency, mcp-bridge-status-removal, and hermes-mcp-shields-order all green (140 tests across the touched surface).


Signed-off-by: sauravdev saurava@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved MCP bridge removal so forced cleanup can recover from incomplete destroy state and finish removing bridges more reliably.
    • Clearer error messages now explain when a removal is blocked and suggest the next step to resolve it.
    • Rebuild failures now show a clearer terminal message before exiting, instead of failing silently.
  • Tests
    • Added coverage for MCP destroy-state cleanup and rebuild failure reporting.

…omplete destroy state

Fixes NVIDIA#6376.

Two gaps let an incomplete MCP destroy transaction strand a sandbox with
no diagnosis and no non-destructive recovery:

1. The rebuild CLI path's bail() discarded its message and exited bare
   (exit 1 with no output). It now prints the redacted failure reason to
   stderr before exiting, so the destroy-marker refusal (and every other
   bail) is visible.

2. assertMcpDestroyNotPending fired before the --force handling in
   mcp remove, so the documented forced cleanup was itself blocked and
   the only way out was full sandbox destruction. A forced remove now
   adopts the incomplete destroy transaction's cleanup manifest for the
   named server, and the durable destroy markers are cleared once no
   bridge entries remain. The guard's error message now names the
   forced-remove recovery command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: sauravdev <saurava@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds detection and clearing of incomplete MCP destroy-transaction markers, wires forced mcp remove to adopt cleanup of these markers, updates the error thrown when destroy is pending to suggest a forced remove command, and changes rebuild's bail callback to print failure reasons to stderr before exiting. Includes new tests for both flows.

Changes

MCP destroy marker recovery

Layer / File(s) Summary
Destroy marker detection and clearing helpers
src/lib/actions/sandbox/mcp-bridge-state.ts
Adds hasMcpDestroyMarkers and clearMcpDestroyMarkersIfNoBridges; rewrites assertMcpDestroyNotPending to throw an actionable error suggesting a forced mcp remove ... --force command.
Forced removal flow adopts destroy recovery
src/lib/actions/sandbox/mcp-bridge-remove.ts
removeMcpBridgeUnlocked skips the pending assertion when forced and destroy markers exist, logs adoption of cleanup, and clears markers once no bridges remain across missing-entry, add-cancellation, and post-removal paths.
Destroy lifecycle test coverage
test/mcp-destroy-lifecycle.test.ts
Adds tests for non-forced rejection under destroy markers, forced removal clearing markers for last/multiple bridges, and marker clearing when no matching entries exist.

Rebuild bail failure reporting

Layer / File(s) Summary
Bail callback prints failure reason
src/lib/actions/sandbox/rebuild-preflight-confirmation.ts
Adds RD styling import; non-throwOnError bail branch now prints a redacted failure message to stderr before exiting with the provided code.
Bail behavior tests
src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts
Adds tests confirming stderr output and process.exit(1) in CLI mode, and that throwOnError preserves throwing behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

Suggested labels: bug-fix

Suggested reviewers: ericksoa, brandonpelfrey

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main MCP recovery and rebuild-failure changes.
Linked Issues check ✅ Passed The changes address #6376 by surfacing rebuild errors and enabling forced MCP recovery from incomplete destroy state.
Out of Scope Changes check ✅ Passed The added code and tests stay focused on rebuild bailout messaging and MCP destroy-state recovery.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts (1)

61-84: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Move issue reference to individual test titles.

The (#6376) reference is on the describe block rather than each it title. As per coding guidelines, **/*.test.ts files should "Write behavior-oriented test titles, and put local issue references in a final (#1234) suffix" — i.e., on each test title.

✏️ Proposed fix
-describe("rebuild bail reporting (`#6376`)", () => {
-  it("prints the failure reason on stderr before exiting in CLI mode", () => {
+describe("rebuild bail reporting", () => {
+  it("prints the failure reason on stderr before exiting in CLI mode (`#6376`)", () => {
     const errors: string[] = [];
     const errorSpy = vi.spyOn(console, "error").mockImplementation((message: unknown) => {
       errors.push(String(message));
     });
     const exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
       throw new Error(`exit:${code}`);
     }) as never);

     const { bail } = createRebuildCommandContext({}, {});
     expect(() => bail("Failed to preserve MCP bridges before rebuild: destroy pending")).toThrow(
       "exit:1",
     );
     expect(errors.join("\n")).toContain("Failed to preserve MCP bridges before rebuild");
     expect(exitSpy).toHaveBeenCalledWith(1);
     errorSpy.mockRestore();
   });

-  it("keeps the throwing bail contract when throwOnError is set", () => {
+  it("keeps the throwing bail contract when throwOnError is set (`#6376`)", () => {
     const { bail } = createRebuildCommandContext({}, { throwOnError: true });
     expect(() => bail("boom")).toThrow("boom");
   });
 });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts` around lines
61 - 84, The issue is that the issue reference is attached to the describe block
instead of the individual test names. Update the rebuild bail reporting tests in
rebuild-preflight-confirmation.test.ts so each it(...) title includes the local
issue suffix as a final "(`#6376`)" while keeping the describe block
behavior-oriented and free of the reference; use the existing test titles around
createRebuildCommandContext and bail to apply the change consistently.

Source: Coding guidelines

test/mcp-destroy-lifecycle.test.ts (1)

741-743: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Key the policy-state mock per server.

After the first removal, this mock reports every policy as "absent", so the second slack cleanup can pass without exercising its own policy cleanup path. Make the mock depend on the policy name instead.

Suggested test tightening
-    testState.getPresetContentGatewayState.mockImplementation(() =>
-      removedPolicies.size > 0 ? "absent" : "match",
-    );
+    testState.getPresetContentGatewayState.mockImplementation(
+      (_sandbox: string, policyName: string) =>
+        removedPolicies.has(policyName) ? "absent" : "match",
+    );

As per path instructions, “Review tests for behavioral confidence rather than implementation lock-in” and flag “broad mocks that bypass the behavior under test.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/mcp-destroy-lifecycle.test.ts` around lines 741 - 743, The policy-state
mock in the destroy lifecycle test is too broad because it keys only off
removedPolicies, so the second slack cleanup can skip its own policy cleanup
path. Update the mock in the test setup around getPresetContentGatewayState to
vary by server/policy identity instead of global removal count, using the
existing server-specific identifiers in the test so each cleanup path is
exercised independently.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts`:
- Around line 61-84: The issue is that the issue reference is attached to the
describe block instead of the individual test names. Update the rebuild bail
reporting tests in rebuild-preflight-confirmation.test.ts so each it(...) title
includes the local issue suffix as a final "(`#6376`)" while keeping the describe
block behavior-oriented and free of the reference; use the existing test titles
around createRebuildCommandContext and bail to apply the change consistently.

In `@test/mcp-destroy-lifecycle.test.ts`:
- Around line 741-743: The policy-state mock in the destroy lifecycle test is
too broad because it keys only off removedPolicies, so the second slack cleanup
can skip its own policy cleanup path. Update the mock in the test setup around
getPresetContentGatewayState to vary by server/policy identity instead of global
removal count, using the existing server-specific identifiers in the test so
each cleanup path is exercised independently.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 758521ed-39b9-446b-92db-d13af91aaf5b

📥 Commits

Reviewing files that changed from the base of the PR and between be3e7cc and ab78a48.

📒 Files selected for processing (5)
  • src/lib/actions/sandbox/mcp-bridge-remove.ts
  • src/lib/actions/sandbox/mcp-bridge-state.ts
  • src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts
  • src/lib/actions/sandbox/rebuild-preflight-confirmation.ts
  • test/mcp-destroy-lifecycle.test.ts

@apurvvkumaria apurvvkumaria self-assigned this Jul 8, 2026
@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Closing as fully superseded by #6377, merged as 05ff005acfbf9878f31f9c9a57ca306f3c1ba634.

Exact-head/current-main comparison:

Thank you @sauravdev for the focused alternative implementation, verification, and test coverage. No code from this branch was copied into #6377 or a replacement commit: the safe behavior had already landed independently, while the only unique behavior is the pending-state widening described above. Accordingly, there is no replacement commit requiring a co-author trailer.

@sauravdev
sauravdev deleted the fix/6376-mcp-destroy-recovery branch July 9, 2026 05:00
@wscurran wscurran added area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression platform: dgx-spark Affects DGX Spark hardware or workflows labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression platform: dgx-spark Affects DGX Spark hardware or workflows

Projects

None yet

3 participants