[cwing-nvidia]

Summary

This PR makes MCP-backed rollouts score consistently across agent harnesses while preventing an untrusted verification request from forging a canonical MCP tool identity.

Details

• add an mcp_tool_call_provenance sidecar keyed by Responses API call_id, while preserving each harness's emitted tool name in the stored trajectory
• validate unsigned provenance against the resources server's per-session MCP execution ledger, including canonical tool arguments
• stamp accepted identities with a signed execution token bound to the call ID, emitted name, arguments, server, and tool so stored artifacts can be safely reverified by a fresh server
• reject forged, argument-mismatched, tampered, or duplicate-ID claims and retain the narrow Claude mcp__<server>__<tool> fallback for historical trajectories
• distinguish unknown provenance (None) from authoritative empty provenance ({}), reject malformed seeded tool metadata, and strip caller-provided sidecars before agent adapters build verification requests
• centralize rollout-scoped MCP metadata validation, URL resolution, collision-safe alias construction, verification payload construction, and response-to-provenance joins in nemo_gym/mcp.py
• populate Codex provenance from structured execution events and Claude provenance by joining completed responses to aliases advertised by /seed_session

Closes #2668

Test plan

• pytest tests/unit_tests/test_mcp.py tests/unit_tests/test_mcp_auto_exposure.py responses_api_agents/claude_code_agent/tests/test_app.py responses_api_agents/codex_agent/tests/test_app.py -q — 172 passed
• scoped pre-commit run --files ...
• forgery regressions: no-execution identity claim, mismatched arguments, tampered signed artifact, duplicate call IDs, and fresh-server reverification
• fresh Workplace MCP rollout with Codex 0.144.4: execution-backed sidecar stamped, reward 1.0, fresh-server reverification 1.0
• fresh Workplace MCP rollout with Claude Code 2.1.178: execution-backed sidecar stamped, reward 1.0, fresh-server reverification 1.0

Scope

Codex and Claude Code are validated end to end here. OpenClaw and Hermes consume the shared contract in stacked PRs #2733 and #2778; their native handoff and event parsing remain adapter-specific.

[cwing-nvidia]

Real rollout and reverification evidence

What was tested

I used one deterministic Workplace Assistant email-reply task with nvidia/nvidia/nemotron-3.5-lightning.

The simple HTTP path establishes the expected verifier result without MCP. Codex and Claude Code exercise the new structured-provenance path through Gym's rollout-specific MCP server.

Results

• Stored function name is the exact function_call.name retained in the rollout; verification does not rewrite the persisted trajectory.
• Sidecar identity is the canonical (server_name, tool_name) recorded in mcp_tool_call_provenance under the same call_id.
• Rollout reward comes from the original /run request.
• Reverification reward comes from the saved materialized input and rollout via gym eval reverify.

Execution path	Stored function name	Sidecar identity	Rollout reward	Reverification reward
simple_agent HTTP baseline	email_reply_email	absent by design	1	not run
Codex 0.144.4 MCP	email_reply_email	(workplace_assistant, email_reply_email)	1	1
Claude Code 2.1.178 MCP	mcp__workplace_assistant__email_reply_email	(workplace_assistant, email_reply_email)	1	1

Claude Code still emits a flattened mcp__<server>__<tool> name. For new rollouts, /seed_session advertises the raw tools exposed by the rollout-specific Gym MCP server, and the Claude adapter builds an exact alias-to-identity map before execution. The legacy string parser is therefore needed only for saved Claude trajectories that predate this metadata.

Stored artifact checks

Selected fields from the Codex rollout:

{
  "response": {
    "output": [
      {
        "type": "function_call",
        "call_id": "item_3",
        "name": "email_reply_email"
      }
    ]
  },
  "mcp_tool_call_provenance": {
    "item_3": {
      "server_name": "workplace_assistant",
      "tool_name": "email_reply_email"
    }
  },
  "reward": 1.0
}

Selected fields from the Claude Code rollout:

{
  "response": {
    "output": [
      {
        "type": "function_call",
        "call_id": "chatcmpl-tool-949109ad5d825174",
        "name": "mcp__workplace_assistant__email_reply_email"
      }
    ]
  },
  "mcp_tool_call_provenance": {
    "chatcmpl-tool-949109ad5d825174": {
      "server_name": "workplace_assistant",
      "tool_name": "email_reply_email"
    }
  },
  "reward": 1.0
}

Controlled reverification

To isolate the sidecar from every supported naming convention, I copied the stored rollouts and changed only function_call.name to a test-only alias. These aliases were not emitted by either harness.

Artifact mutation	Sidecar	Reverification result
Codex name changed to synthetic_codex_alias	present	1
Same Codex artifact	removed	0 (unknown tool synthetic_codex_alias)
Claude name changed to synthetic_claude_alias	present	1

synthetic_claude_alias cannot match the historical Claude mcp__<server>__<tool> fallback. Its reward of 1 therefore demonstrates that reverification used the structured identity rather than parsing the stored name.

Scope and observations

• Workplace reports reverify_mode=UNSUPPORTED or UNKNOWN; --force therefore prefixes reverified outputs with unsafe_. The same warning applied to both sides of each controlled comparison.
• Codex reported a stream-completion parse error (ResponseCompleted: invalid type: null, expected i64) after the successful tool action. The call, result, provenance, and reward had already been stored; this is separate from the provenance behavior under test.
• OpenClaw and Hermes were not run because their dynamic Gym MCP handoffs remain tracked by #2670 and #2667.

Reproduction files

Save the following four files at the repository root. env.yaml must define policy_base_url and policy_api_key for the selected model.

issue_2668_workplace_smoke.jsonl

{"id": 0, "responses_create_params": {"input": [{"role": "system", "content": "Use the available workplace tools to complete the requested action."}, {"role": "user", "content": "Reply to email ID 00000057 with exactly: Thanks for the update - I will get back to you tomorrow."}], "tools": [{"type": "function", "name": "email_reply_email", "description": "Replies to an email by its ID.", "parameters": {"type": "object", "properties": {"email_id": {"type": "string"}, "body": {"type": "string"}}, "required": ["email_id", "body"], "additionalProperties": false}, "strict": false}], "parallel_tool_calls": false, "temperature": 0.0}, "ground_truth": [{"name": "email_reply_email", "arguments": "{\"email_id\": \"00000057\", \"body\": \"Thanks for the update - I will get back to you tomorrow.\"}"}], "category": "workplace_assistant_email", "environment_name": "workplace_assistant"}

issue_2668_simple_http_smoke.yaml

workplace_assistant:
  resources_servers:
    workplace_assistant:
      entrypoint: app.py
      domain: agent
      verified: false
      expose_tools_over_mcp: true
      description: Temporary HTTP baseline smoke test

workplace_assistant_simple_http:
  responses_api_agents:
    simple_agent:
      entrypoint: app.py
      max_steps: 6
      resources_server:
        type: resources_servers
        name: workplace_assistant
      model_server:
        type: responses_api_models
        name: policy_model
      datasets:
        - name: smoke
          type: validation
          jsonl_fpath: issue_2668_workplace_smoke.jsonl
          license: Apache 2.0

issue_2668_codex_mcp_smoke.yaml

workplace_assistant:
  resources_servers:
    workplace_assistant:
      entrypoint: app.py
      domain: agent
      verified: false
      expose_tools_over_mcp: true
      description: Temporary Codex MCP provenance smoke test

workplace_assistant_codex_mcp:
  responses_api_agents:
    codex_agent:
      entrypoint: app.py
      resources_server:
        type: resources_servers
        name: workplace_assistant
      model_server:
        type: responses_api_models
        name: policy_model
      concurrency: 1
      model: ${policy_model_name}
      openai_api_key: EMPTY
      openai_base_url: null
      sandbox_mode: danger-full-access
      timeout: 600
      system_prompt: null
      reasoning_effort: null
      codex_version: 0.144.4
      cwd: null
      stream_idle_timeout_ms: null
      extra_config: {}
      datasets:
        - name: smoke
          type: validation
          jsonl_fpath: issue_2668_workplace_smoke.jsonl
          license: Apache 2.0

issue_2668_claude_mcp_smoke.yaml

workplace_assistant:
  resources_servers:
    workplace_assistant:
      entrypoint: app.py
      domain: agent
      verified: false
      expose_tools_over_mcp: true
      description: Temporary Claude Code MCP provenance smoke test

workplace_assistant_claude_mcp:
  responses_api_agents:
    claude_code_agent:
      entrypoint: app.py
      resources_server:
        type: resources_servers
        name: workplace_assistant
      model_server:
        type: responses_api_models
        name: policy_model
      concurrency: 1
      model: ${policy_model_name}
      anthropic_api_key: ${policy_api_key}
      anthropic_base_url: null
      max_turns: 10
      timeout: 600
      system_prompt: null
      allowed_tools: null
      disallowed_tools: null
      claude_code_version: null
      thinking: null
      max_thinking_tokens: null
      bare: true
      mcp_config: null
      settings: null
      datasets:
        - name: smoke
          type: validation
          jsonl_fpath: issue_2668_workplace_smoke.jsonl
          license: Apache 2.0

Rollout commands

Run .venv/bin/gym directly; wrapping the command in uv run can activate Ray's uv runtime hook for child component directories.

MODEL=nvidia/nvidia/nemotron-3.5-lightning

.venv/bin/gym eval run \
  --config issue_2668_simple_http_smoke.yaml \
  --model-type vllm_model \
  --agent workplace_assistant_simple_http \
  --output outputs/issue-2668/simple_http.jsonl \
  --limit 1 --concurrency 1 --split validation \
  ++policy_model_name="$MODEL"

.venv/bin/gym eval run \
  --config issue_2668_codex_mcp_smoke.yaml \
  --model-type vllm_model \
  --agent workplace_assistant_codex_mcp \
  --output outputs/issue-2668/codex_mcp.jsonl \
  --limit 1 --concurrency 1 --split validation \
  ++policy_model_name="$MODEL"

.venv/bin/gym eval run \
  --config issue_2668_claude_mcp_smoke.yaml \
  --model-type vllm_model \
  --agent workplace_assistant_claude_mcp \
  --output outputs/issue-2668/claude_mcp.jsonl \
  --limit 1 --concurrency 1 --split validation \
  ++policy_model_name="$MODEL"

Reverification commands

.venv/bin/gym eval reverify \
  --config issue_2668_codex_mcp_smoke.yaml \
  --model-type vllm_model \
  --inputs outputs/issue-2668/codex_mcp_materialized_inputs.jsonl \
  --rollouts outputs/issue-2668/codex_mcp.jsonl \
  --output outputs/issue-2668/codex_mcp_reverified.jsonl \
  --concurrency 1 --limit 1 --force

.venv/bin/gym eval reverify \
  --config issue_2668_claude_mcp_smoke.yaml \
  --model-type vllm_model \
  --inputs outputs/issue-2668/claude_mcp_materialized_inputs.jsonl \
  --rollouts outputs/issue-2668/claude_mcp.jsonl \
  --output outputs/issue-2668/claude_mcp_reverified.jsonl \
  --concurrency 1 --limit 1 --force

Post-refactor validation (shared handoff infrastructure)

After centralizing metadata parsing and response/provenance joins in nemo_gym/mcp.py, I reran the real Codex and Claude Workplace rollouts and reverified the new stored artifacts.

• Codex 0.144.4: emitted call_id=item_3, stored name email_reply_email, sidecar (workplace_assistant, email_reply_email), rollout reward 1.0, reverification reward 1.0.
• Claude Code 2.1.178: emitted call_id=chatcmpl-tool-a4f3be1f3a36644b, stored name mcp__workplace_assistant__email_reply_email, sidecar (workplace_assistant, email_reply_email), rollout reward 1.0, reverification reward 1.0.

The Claude result specifically exercises the refactored behavior: provenance is now joined from the completed NeMoGymResponse rather than collected while parsing stream events. Both sidecar entries use the exact emitted function_call.call_id, and both fresh artifacts reverify successfully without rewriting the stored function name.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-fix-structured-mcp-tool-provenance.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/environment-tutorials/mcp-resources-server
• nemo/gym/tutorials/evaluation-tutorials/reverify-rollouts

[cwing-nvidia]

/claude review

[claude]

SHIP — structured MCP provenance is a net correctness improvement over string-parsing, and the scoring path is carefully guarded and well-tested.

Traced the paths that can silently corrupt reward:

• Verify wrapper (nemo_gym/mcp_auto_exposure.py:663-695): "present mapping is authoritative → fallback disabled; absent call_ids are non-MCP" is implemented consistently. Duplicate call_ids fail closed (len(emitted[...]) != 1 → continue), and response-name restoration is per-call_id indexed so multi-call ids restore correctly. Canonicalization runs on a deep copy and only for identity.server_name == resources_server_name, so other MCP sources and built-ins are left untouched. Covered by the new test_verify_prefers_structured_mcp_provenance_*, test_present_empty_provenance_disables_legacy_prefix_guessing, and test_duplicate_call_ids_* tests.
• Codex (parse_exec_jsonl): provenance is read from authoritative server/tool event fields, and metadata[provenance_key] is always initialized, so the usage[...] subscript in _create_response can't KeyError.
• Reverification (rollout_reverification.py:_build_verify_payload): forwards the sidecar only when present — historical rollouts without it still take the legacy fallback.
• Backward compat: missing tool_names → aliases None → provenance None → legacy normalize_tool_name fallback. The new required MCPServerMetadata.tool_names field is constructed only internally; seed parsing tolerates its absence.

NOTE (author's call): The Claude path reconstructs provenance by exact-matching emitted names against f\"mcp__{server_name}__{tool_name}\" built from raw names (_claude_mcp_wire_name, build_mcp_tool_aliases). This is correct as long as the model emits names Claude Code did not sanitize (routable server names per #2671; tool names derived from route path segments). build_mcp_tool_aliases correctly drops sanitization collisions, and a plain join miss is fail-safe (keeps the emitted name — same outcome as the old fallback for that case). But a miss is silent: the call simply gets no canonicalization and no log line. If Claude Code's sanitization ever diverges from the raw reconstruction for a valid tool name, reward would drop with no signal. A debug log when an exposed tool_name produces no matching function_call would make that observable. Codex is unaffected (structured fields).

No blockers.

[cmunley1]

shared some comments offline

[ffrujeri]

Could you clarify whether mcp_tool_call_provenance needs to become part of the shared BaseVerifyRequest / BaseVerifyResponse contract? This is an additive optional field, so it does not look immediately breaking, but it changes the base verification schema used by every resources server and was not called out explicitly as an API-contract change in the PR summary.

Could this metadata instead remain in a Gym-owned verification envelope or another layer outside the common verifier models, while still being persisted and forwarded by gym eval reverify? If changing the base models is necessary so FastAPI validates and carries the sidecar through arbitrary verifier implementations, could we document that rationale and the compatibility implications explicitly?

My understanding is that there are two opposite transformations:

• MCP auto_exposure advertises: get_weather
• Claude Code Binary flattens: mcp__example_mcp_weather__get_weather
• Gym normalizes back in the mcp_auto_exposure.py middleware: get_weather

The flattened tool name enters NeMoGymResponse, because Claude Code exposes MCP tools to its model using:
mcp____, and then then it emits that name in a tool_use event. NeMo Gym’s Claude adapter copies it unchanged:

pending_calls[call_id] = {
    "name": block.get("name", ""),
    # ...
}

NeMoGymResponseFunctionToolCall(
    call_id=tool_id,
    name=call_info["name"],
    # ...
)

The place where NeMo-Gym normalizes it back is in the

POST /verify
 → verify_normalized()
 → server.normalize_tool_name()
 → strip mcp__example_mcp_weather__
 → original verify() receives get_weather

server.normalize_tool_name() eventually runs:

if not name.startswith("mcp__"):    <--------- We can potentially could create a factory pattern here to denormalize @adil-a 
    return name
if server_name is not None:
    prefix = f"mcp__{server_name}__"
    return name[len(prefix) :] if name.startswith(prefix) else name

[ffrujeri]

Thinking out loud, since this is somehow a contract change.

Could _create_response() remain responsible only for producing the standard NeMoGymResponse, with run() deriving and forwarding mcp_tool_call_provenance afterward?

That seems closer to the separation in simple_agent: responses() produces the Responses API object, while run() owns rollout orchestration (/seed_session, verification, and Gym-specific metadata). I understand why Claude's run() cannot simply self-call the public responses() endpoint—the temporary rollout-specific MCP config is an internal parameter—but the provenance join can still happen in run() after _create_response() returns. That would avoid changing this execution helper's return contract solely for verification metadata and keep responses() from immediately discarding part of AgentExecutionResult.

[ffrujeri]

The core problem as I understand it is that an MCP tool call currently has two representations: the harness/model-facing name (for example, Claude Code's mcp__example_mcp_weather__get_weather) and the canonical MCP identity (server_name=example_mcp_weather, tool_name=get_weather). The emitted name is useful for faithful replay, observability, and training, while the canonical identity is what the environment understands.

A Resources Server's verify() should remain generic and agent-agnostic: it should consume its own canonical tool vocabulary without needing to know Claude, Codex, Hermes, or OpenClaw naming and sanitization conventions. The agent adapter is the component that understands how its harness projected (server, tool) into the model-facing namespace, so that mapping needs to be established before the environment verifier runs.

The PR's structured-provenance direction proposes one way to addresses that boundary. I think it is worth understanding design-wise where is this issue coming from and the data contracts involved. @ananthsub, @marta-sd, @adil-a

A Resources Server’s verify() should understand its own domain vocabulary:

get_weather
email_reply_email
append

It should not need to understand every agent harness’s presentation rules:

Claude:    mcp__example_mcp_weather__get_weather
Hermes:    mcp_example_mcp_weather_get_weather
OpenClaw:  example_mcp_weather__get_weather
Codex:     get_weather

Otherwise every verifier becomes coupled to:

• Which agent produced the rollout
• That agent’s version
• Its sanitization and truncation rules
• Future naming changes
• External MCP server configurations

The desired boundary by this PR is something like

Agent adapter
    understands harness-specific names
             ↓
Gym verification boundary
    maps them to canonical MCP identity
             ↓
verify()
    sees only environment-level tool names

Notice that it is an architectural choice here to

Keep the standard Responses function_call shape and its emitted name unchanged.

The rationale is that NeMoGymResponse represents what the model actually produced, which matters for:

• Exact replay
• Training-token fidelity
• Debugging and observability
• Distinguishing calls as presented by different harnesses

I think the friction here is that fundamentally Gym cannot preserve the raw alias and later recover an ambiguous MCP identity without storing extra information somewhere. The real choice is:

• Canonicalize NeMoGymResponse and store raw evidence elsewhere, or
• Keep NeMoGymResponse raw and introduce a first-class canonical trajectory.

Alternatively, maybe something worth considering @cwing-nvidia is if the verify method really needs to depend on the tool_call name (and replay of the trajectory) or if we could enforce it to depend on state of the environment.

[cmunley1]

agentexecutionresult feels overly general for mcp only path

[cmunley1]

• The in-memory execution_ledger has no eviction. Sessions that call MCP tools but never reach /verify leave entries forever.
• Provenance validation compares claimed and executed calls as ordered lists. Parallel MCP calls may arrive in a different order than the trajectory, causing valid provenance to be rejected. It needs an order-independent comparison or an explicit stable call identifier.