fix: preserve structured MCP tool identity for verification

fern/versions/latest/pages/environment-tutorials/mcp-resources-server.mdx

@@ -160,7 +160,7 @@ At startup (`run_webserver`), Gym:
 
 - turns each plain `POST` route into an MCP tool **named after its path** (`POST /get_weather` → tool `get_weather`), with the input schema derived from the route's Pydantic body model and the description from the handler's docstring. `/seed_session`, `/verify`, `/aggregate_metrics`, and `/mcp` are never tools.
 - mounts a Streamable-HTTP `/mcp` endpoint on the same app.
-- wraps `/seed_session` so its response also carries the `mcp` metadata (server name, `/mcp` URL path, and a signed per-rollout `X-NeMo-Gym-Session-Token`) — which is how an MCP-native agent discovers the endpoint and its rollout-scoped credentials.
+- wraps `/seed_session` so its response also carries the `mcp` metadata (server name, `/mcp` URL path, transport, exposed tool names, and a signed per-rollout `X-NeMo-Gym-Session-Token`) — which is how an MCP-native agent discovers the endpoint, builds an exact tool-identity map, and receives its rollout-scoped credentials.
 
 An MCP `tools/call` invokes the route's own handler directly, with the rollout's session materialized on the `Request` — handlers keep their `request.session` reads exactly as written. The route stays callable over plain HTTP too: the same tool serves both transports, and `/verify` scores both identically. Tool errors (`HTTPException`, validation failures, crashes) surface to the MCP client as tool errors with the same status and text the plain HTTP route would have returned.
 
@@ -295,7 +295,33 @@ gym eval run --no-serve \
     --output results/workplace_claude_rollouts.jsonl
 ```
 
-A correct rollout shows Claude Code calling `mcp__workplace_assistant__*` tools (e.g. `mcp__workplace_assistant__email_reply_email`) and a `reward` of `1.0`. `/verify` scores MCP and HTTP trajectories identically: when the flag is on, the verify endpoint is wrapped at startup to normalize MCP-namespaced tool-call names (`mcp__workplace_assistant__email_reply_email` → `email_reply_email`) for scoring only, while the persisted rollout keeps the names the model actually emitted.
+A correct rollout shows Claude Code calling `mcp__workplace_assistant__*` tools (for example, `mcp__workplace_assistant__email_reply_email`) and a `reward` of `1.0`. The stored rollout keeps that client-facing name and records its canonical MCP identity separately under the same `call_id`:
+
+```json
+{
+  "response": {
+    "output": [
+      {
+        "type": "function_call",
+        "call_id": "call_1",
+        "name": "mcp__workplace_assistant__email_reply_email"
+      }
+    ]
+  },
+  "mcp_tool_call_provenance": {
+    "call_1": {
+      "server_name": "workplace_assistant",
+      "tool_name": "email_reply_email"
+    }
+  }
+}
+```
+
+`/verify` uses this sidecar on a copy of the trajectory: calls attributed to the Resources Server being verified use their canonical `tool_name` for scoring, while built-in calls, calls from other MCP servers, and calls absent from the sidecar keep their emitted names. The persisted rollout and verify response retain the original client-facing names.
+
+<Note>
+For historical Claude Code trajectories that do not have `mcp_tool_call_provenance`, Gym retains a narrow compatibility fallback that recognizes `mcp__<server>__<tool>`. New trajectories use structured provenance rather than parsing a client-specific naming convention.
+</Note>
 
 ### The weather server runs the same way
 
@@ -337,6 +363,7 @@ Things to know about this flow:
 
 - **No cookie/session entanglement.** Gym's session cookie flows only between the agent server and the Resources Server (`/seed_session` ↔ `/verify`). The agent-to-external-MCP connection is a separate channel with its own auth (whatever `headers` you put in the static config). They don't interfere.
 - **Verify off the trajectory.** Gym can't observe the external server's calls, so `verify()` must score the `function_call` / `function_call_output` items in the agent's Responses-API output — not server-side session state.
+- **Identity remains source-specific.** When an agent adapter records structured MCP provenance for an external call, its external `server_name` is preserved. A Gym Resources Server only substitutes canonical names for sidecar entries attributed to that Resources Server; it does not reinterpret calls from other MCP servers.
 - **Static + per-rollout compose.** When both are present, the agent merges your static `mcp_config` with the per-rollout Gym-owned entry, so a single rollout can use external tools *and* Gym-owned MCP tools at once. If a static server happens to share the **same name** as the Gym resources server, the per-rollout Gym entry takes precedence and overwrites it.
 
 ---

fern/versions/latest/pages/evaluation-tutorials/reverify-rollouts.mdx

@@ -25,11 +25,15 @@ You need the two artifact files that `gym eval run` writes alongside the main ro
 
 | File | Contents |
 | --- | --- |
-| `rollouts.jsonl` | One row per rollout — includes the model `response` (the expensive artifact) |
+| `rollouts.jsonl` | One row per rollout — includes the model `response` (the expensive artifact) and any rollout-level metadata needed to verify it |
 | `rollouts_materialized_inputs.jsonl` | One row per rollout — includes per-task verifier inputs (for example, `options` and `expected_answer` in the `mcqa` resources server) that are not echoed back into the rollout row |
 
 Both files are required. `rollouts.jsonl` alone is not sufficient because many resources servers drop verifier-specific fields from the verify response; those fields are preserved in `*_materialized_inputs.jsonl`.
 
+<Note>
+For MCP-backed rollouts, preserve the top-level `mcp_tool_call_provenance` mapping when copying or transforming `rollouts.jsonl`. Reverification forwards this mapping with the stored response so the verifier can identify each MCP call by its canonical `server_name` and `tool_name` without rewriting the client-facing function name. Historical Claude Code rollouts without this field can still use the legacy `mcp__<server>__<tool>` fallback.
+</Note>
+
 ## Step 1 — Run reverification
 
 `gym eval reverify` starts the resources server automatically from the config you provide. It does **not** need a model server or an agent — pass a resources-server-only config to avoid starting inference infrastructure you don't need.

nemo_gym/base_resources_server.py

@@ -17,7 +17,7 @@
 from typing import TYPE_CHECKING, Any, ClassVar, Optional
 
 from fastapi import FastAPI
-from pydantic import BaseModel
+from pydantic import BaseModel, Field
 
 
 if TYPE_CHECKING:
@@ -38,18 +38,19 @@
 
 NEMO_GYM_MCP_SESSION_TOKEN_HEADER = "X-NeMo-Gym-Session-Token"
 NEMO_GYM_MCP_METADATA_KEY = "mcp"
+NEMO_GYM_MCP_TOOL_CALL_PROVENANCE_KEY = "mcp_tool_call_provenance"
 # Salt namespacing the signed MCP session token, so it can't be confused with another signer
 # that happens to share the same session-middleware secret.
 _MCP_TOKEN_SALT = "nemo-gym-mcp-session-token"
 
 
 def normalize_tool_name(name: str, server_name: Optional[str] = None) -> str:
-    """Map a trajectory tool-call name to the server's bare tool name.
+    """Legacy fallback for provenance-free Claude Code MCP tool names.
 
     HTTP-driven agents record bare tool names ("email_reply_email"); MCP-native agents (e.g.
     Claude Code) record them namespaced per server ("mcp__workplace_assistant__email_reply_email").
-    Verifiers compare trajectory names against dataset/ground-truth vocabulary, so names are
-    normalized before verify sees them and rollouts score identically on both transports.
+    New trajectories should carry ``MCPToolCallProvenance`` instead of relying on this client-specific
+    string convention. This fallback keeps existing saved trajectories reverifiable.
     Non-namespaced names pass through unchanged. When ``server_name`` is given, only that server's
     prefix is stripped (robust to tool names that themselves contain double underscores).
     This runs only for servers exposed over MCP and mirrors how MCP clients namespace tool names,
@@ -89,8 +90,24 @@ class BaseRunRequest(BaseModel):
     responses_create_params: NeMoGymResponseCreateParamsNonStreaming
 
 
+class MCPToolCallProvenance(BaseModel):
+    """Canonical identity of one adapter-observed MCP call."""
+
+    server_name: str = Field(min_length=1, pattern=r"\S")
+    tool_name: str = Field(min_length=1, pattern=r"\S")
+    # Resources servers stamp this after validating the identity against an actual MCP execution.
+    # It lets a stored trajectory prove the same call identity during later reverification.
+    execution_token: Optional[str] = Field(default=None, exclude_if=lambda value: value is None)
+
+
 class BaseVerifyRequest(BaseRunRequest):
     response: NeMoGymResponse
+    # None means a historical/unknown trajectory, so the narrow Claude Code fallback may run.
+    # A mapping (including an empty one) is authoritative and complete: absent call IDs are non-MCP.
+    mcp_tool_call_provenance: Optional[dict[str, MCPToolCallProvenance]] = Field(
+        default=None,
+        exclude_if=lambda value: value is None,
+    )
 
 
 class BaseVerifyResponse(BaseVerifyRequest):
@@ -128,6 +145,7 @@ class MCPServerMetadata(BaseModel):
     url_path: str = "/mcp"
     transport: str = "http"
     headers: dict[str, str]
+    tool_names: list[str]
 
 
 class SimpleResourcesServer(BaseResourcesServer, AggregateMetricsMixin, SimpleServer):

nemo_gym/mcp.py

@@ -0,0 +1,177 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared client-side helpers for rollout-scoped Gym MCP servers."""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable, Mapping
+from dataclasses import dataclass
+from typing import Any, Optional
+
+from nemo_gym.base_resources_server import (
+    NEMO_GYM_MCP_METADATA_KEY,
+    NEMO_GYM_MCP_TOOL_CALL_PROVENANCE_KEY,
+    MCPToolCallProvenance,
+)
+from nemo_gym.global_config import get_first_server_config_dict
+from nemo_gym.openai_utils import NeMoGymResponse
+
+
+@dataclass(frozen=True, slots=True)
+class RolloutMCPServer:
+    """Validated Gym MCP metadata resolved to an absolute resources-server URL."""
+
+    server_name: str
+    url: str
+    transport: str
+    headers: dict[str, str]
+    tool_names: Optional[tuple[str, ...]]
+
+
+@dataclass(frozen=True, slots=True)
+class AgentExecutionResult:
+    """An agent response and its authoritative structured MCP provenance."""
+
+    response: NeMoGymResponse
+    mcp_tool_call_provenance: Optional[dict[str, MCPToolCallProvenance]] = None
+
+
+def resources_server_base_url(server_client: Any, resources_server_name: str) -> str:
+    """Resolve a configured resources-server instance to its base URL."""
+    config = get_first_server_config_dict(server_client.global_config_dict, resources_server_name)
+    return server_client._build_server_base_url(config)
+
+
+def parse_rollout_mcp_server(
+    seed_response_json: Mapping[str, Any],
+    *,
+    resources_server_name: str,
+    resources_server_base_url: str | Callable[[], str],
+    logger: logging.Logger | None = None,
+) -> RolloutMCPServer | None:
+    """Parse Gym ``/seed_session`` MCP metadata without assuming a harness config shape."""
+    if NEMO_GYM_MCP_METADATA_KEY not in seed_response_json:
+        return None
+
+    metadata = seed_response_json[NEMO_GYM_MCP_METADATA_KEY]
+    if not isinstance(metadata, Mapping):
+        if logger is not None:
+            logger.warning("Ignoring non-object MCP seed metadata")
+        return None
+
+    raw_server_name = metadata.get("server_name")
+    server_name = resources_server_name if raw_server_name is None else raw_server_name
+    if not isinstance(server_name, str) or not server_name:
+        raise ValueError("MCP seed metadata server_name must be a non-empty string")
+
+    raw_url_path = metadata.get("url_path")
+    url_path = "/mcp" if raw_url_path is None or raw_url_path == "" else raw_url_path
+    if not isinstance(url_path, str):
+        raise ValueError("MCP seed metadata url_path must be a string")
+
+    raw_transport = metadata.get("transport")
+    transport = "http" if raw_transport is None or raw_transport == "" else raw_transport
+    if not isinstance(transport, str):
+        raise ValueError("MCP seed metadata transport must be a string")
+    transport = transport.replace("_", "-")
+
+    raw_headers = metadata.get("headers")
+    if raw_headers is not None and not isinstance(raw_headers, Mapping):
+        raise ValueError("MCP seed metadata headers must be an object")
+    headers: dict[str, str] = {}
+    if isinstance(raw_headers, Mapping):
+        for key, value in raw_headers.items():
+            if not isinstance(key, str) or not isinstance(value, (str, int, float, bool)):
+                raise ValueError("MCP seed metadata headers must contain scalar values")
+            headers[key] = str(value)
+    if not headers and logger is not None:
+        logger.warning(
+            "MCP seed metadata for %r has no headers; the tool endpoint will be called without a "
+            "session token and may reject the calls.",
+            server_name,
+        )
+
+    raw_tool_names = metadata.get("tool_names")
+    tool_names: Optional[tuple[str, ...]]
+    if raw_tool_names is None:
+        tool_names = None
+    elif not isinstance(raw_tool_names, list):
+        raise ValueError("MCP seed metadata tool_names must be an array")
+    else:
+        if not all(isinstance(name, str) and name for name in raw_tool_names):
+            raise ValueError("MCP seed metadata tool_names must contain non-empty strings")
+        tool_names = tuple(raw_tool_names)
+
+    base_url = resources_server_base_url() if callable(resources_server_base_url) else resources_server_base_url
+    return RolloutMCPServer(
+        server_name=server_name,
+        url=f"{base_url.rstrip('/')}/{url_path.lstrip('/')}",
+        transport=transport,
+        headers=headers,
+        tool_names=tool_names,
+    )
+
+
+def build_mcp_tool_aliases(
+    server: RolloutMCPServer,
+    *,
+    wire_name: Callable[[str, str], str],
+) -> Optional[dict[str, MCPToolCallProvenance]]:
+    """Map harness wire names to canonical identities, omitting ambiguous aliases."""
+    if server.tool_names is None:
+        return None
+
+    aliases: dict[str, MCPToolCallProvenance] = {}
+    ambiguous_aliases: set[str] = set()
+    for tool_name in server.tool_names:
+        alias = wire_name(server.server_name, tool_name)
+        identity = MCPToolCallProvenance(server_name=server.server_name, tool_name=tool_name)
+        previous = aliases.get(alias)
+        if previous is not None and previous != identity:
+            aliases.pop(alias)
+            ambiguous_aliases.add(alias)
+        elif alias not in ambiguous_aliases:
+            aliases[alias] = identity
+    return aliases or None
+
+
+def provenance_from_response_aliases(
+    response: NeMoGymResponse,
+    aliases: Mapping[str, MCPToolCallProvenance],
+) -> Optional[dict[str, MCPToolCallProvenance]]:
+    """Join emitted Responses function calls to canonical identities by wire name."""
+    if not aliases:
+        return None
+
+    provenance: dict[str, MCPToolCallProvenance] = {}
+    for item in response.output:
+        if getattr(item, "type", None) != "function_call":
+            continue
+        call_id = getattr(item, "call_id", None)
+        name = getattr(item, "name", None)
+        if not isinstance(call_id, str) or not call_id or not isinstance(name, str):
+            continue
+        identity = aliases.get(name)
+        if identity is not None:
+            provenance[call_id] = identity
+        elif name.startswith("mcp__"):
+            return None
+    return provenance
+
+
+def build_mcp_verify_payload(
+    body: Any,
+    response: NeMoGymResponse,
+    provenance: Optional[Mapping[str, MCPToolCallProvenance]],
+) -> dict[str, Any]:
+    """Build a verify payload without accepting caller-supplied provenance."""
+    payload = body.model_dump()
+    payload.pop(NEMO_GYM_MCP_TOOL_CALL_PROVENANCE_KEY, None)
+    payload["response"] = response.model_dump(mode="json")
+    if provenance is not None:
+        payload[NEMO_GYM_MCP_TOOL_CALL_PROVENANCE_KEY] = {
+            call_id: identity.model_dump(mode="json") for call_id, identity in provenance.items()
+        }
+    return payload