bug: MCP tool identity not preserved for verification #2668

Summary

Gym currently conflates a harness's model-facing MCP tool alias with the canonical tool identity expected by a resources-server verifier. MCP defines tool names within a server connection, but agent harnesses project (server, tool) into their model-facing tool namespace differently.

Gym should carry explicit MCP call provenance through rollout verification instead of reconstructing identity from client-specific strings.

Current behavior

Gym's MCP auto-exposure advertises bare resources-server tool names such as:

email_reply_email

Harnesses may present that tool to the model under different names:

• Claude Code: mcp__<server>__<tool>
• Gym's currently pinned Hermes runtime: mcp_<sanitized-server>_<sanitized-tool>
• Current OpenClaw: <provider-safe-server>__<provider-safe-tool>

These aliases are harness implementation details, not an MCP protocol naming standard.

nemo_gym/base_resources_server.py::normalize_tool_name() currently recognizes only the Claude Code form. The verify wrapper deep-copies a trajectory, infers a bare name from the emitted name, verifies the copy, and restores the emitted name in the returned artifact.

That string inference is not a general solution. It cannot reliably represent multiple MCP servers, client sanitization or truncation, alias collisions, or future harness naming rules.

Built-in Gym agents may also load user-configured MCP servers that are not Gym resources servers. For example, an agent can use a rollout-specific Gym MCP server for scored environment actions while also using an external GitHub or search MCP server. One trajectory can therefore contain calls from both Gym-owned and external servers, including servers that expose the same bare tool name. Verification must canonicalize only calls belonging to the resources server performing verification; external MCP calls must retain their distinct identity unless that verifier explicitly models their effects.

Evidence and current validation status

• Claude Code's mcp__<server>__<tool> form is supported by the existing implementation and tests.
• Gym's pinned Hermes commit uses a different flattened form. The reproduction in this issue demonstrates that the same Workplace action scores 1 with its bare name and 0 with that Hermes alias.
• Current upstream Hermes has since adopted the Claude-style form, so the pinned-Hermes case is legacy compatibility rather than evidence that a Hermes-specific prefix should be added.
• Current OpenClaw source constructs <provider-safe-server>__<provider-safe-tool>. A complete Gym/OpenClaw MCP rollout depends on the dynamic MCP handoff tracked in bug(openclaw_agent): consume resources-server MCP metadata from seed_session #2670.

The general defect is that Gym has no structured MCP call identity at the verification boundary.

Proposed contract

Preserve two distinct representations:

1. The model-facing response.output[].name remains exactly what the harness emitted. This preserves Responses compatibility, replay behavior, observability, and training-token fidelity.
2. Gym-owned verification metadata records the raw MCP server_name and raw MCP tool_name, correlated to the function call by call_id.

For example, the standard Responses object remains unchanged:

{
  "type": "function_call",
  "call_id": "call_1",
  "name": "workplace_assistant__email_reply_email",
  "arguments": "{}"
}

The Gym verification request or rollout metadata carries the semantic identity separately:

{
  "mcp_tool_call_provenance": {
    "call_1": {
      "server_name": "workplace_assistant",
      "tool_name": "email_reply_email"
    }
  }
}

An MCP-capable agent adapter must populate this provenance from trusted harness execution metadata or from the exact tool mapping it constructed. It must not accept model-generated provenance as authoritative.

During verification, Gym uses tool_name only when the provenance server_name exactly matches the resources server performing verification. The original response remains unchanged. Calls belonging to other MCP servers retain their model-facing names in that verifier's copy.

Backward compatibility

Existing datasets and verifiers use resources-server-local bare tool names and must continue to work. Direct HTTP tool calls therefore require no provenance and remain unchanged.

Persisted trajectories that predate structured provenance must remain reverifiable. The current exact Claude Code normalization can remain as a narrowly scoped legacy fallback when provenance is absent; new client formats should not add more prefix-guessing rules.

Acceptance criteria

• Define an optional, typed Gym MCP call-provenance schema containing raw server_name and raw tool_name, correlated by call_id.
• Keep the standard Responses function_call shape and its emitted name unchanged; provenance is carried in Gym-owned metadata outside the Responses object.
• Populate provenance from adapter-observed execution data or an adapter-owned exact alias map, not from model-authored fields.
• Preserve provenance in saved rollout artifacts so normal verification and gym eval reverify resolve the same canonical identity and reward.
• During verification, canonicalize only a deep copy and only when provenance identifies the exact resources server performing verification.
• Leave bare HTTP calls and calls belonging to other MCP servers unchanged.
• Treat missing, malformed, ambiguous, or mismatched provenance as non-authoritative: do not guess a new client alias format or strip a generic prefix.
• Preserve compatibility for historical provenance-free Claude Code trajectories through the existing exact-server legacy fallback.
• Test a rollout containing both the Gym resources-server MCP entry and a user-configured external, non-Gym MCP server, including the same bare tool name on both servers, server names that sanitize or collide in a client namespace, tool names containing delimiter-like text, and repeated calls correlated by distinct call_id values. Only the matching Gym resources-server calls may be canonicalized for that verifier.
• Demonstrate equal verifier behavior and reward for the same resources-server action over bare HTTP and structured MCP provenance while preserving each harness's original emitted alias in the stored trajectory.
• Add focused adapter tests for the client behaviors already identified (Claude Code, the pinned Hermes legacy behavior, and OpenClaw when bug(openclaw_agent): consume resources-server MCP metadata from seed_session #2670 lands) without making those naming formats part of the canonical verifier contract.

Related work

• bug(hermes_agent): consume resources-server MCP metadata from seed_session #2667 tracks dynamic Gym MCP handoff for hermes_agent.
• bug(openclaw_agent): consume resources-server MCP metadata from seed_session #2670 tracks dynamic Gym MCP handoff for openclaw_agent.