[DanialTaheri]

What does this PR do?

Adds the Agentic-MME benchmark: multi-turn visual reasoning where the policy answers image questions using atomic image tools and, optionally, web retrieval. It is onboarded as a manifest-backed benchmark (experimental: true). No existing environment covers this task, so there is no linked issue; this PR is the proposal.

• resources_servers/agentic_mme: task-only verification. The terminal answer is scored by an LLM judge, with the released evaluator's exact / contains / numeric string match reported alongside as string_match_reward. Verifier metadata never enters the model context. Exports a VERIFIER_FIXTURE.
• responses_api_agents/agentic_mme_agent: native multi-turn agent with append-only history, so every model call's prompt extends the previous one (compatible with token capture for RL). It is fronted by a legacy_agent environment server.
  • 14 atomic image tools (crop, rotate, flip, resize, enhance, grayscale, autocontrast, blur, sharpen, denoise, edge_detect, invert, equalize, threshold) over a per-episode image workspace with pixel budgets. Each successful operation appends an image the model sees next turn.
  • Optional retrieval: web search (Serper / SerpAPI / Tavily / Exa), Google Lens (Serper / SerpAPI), page fetch (Jina / Exa), with a benchmark-contamination guard that scans the whole fetched page.
  • Replay mode that feeds recorded retrieval results back in order, for reproducible evaluation.
  • Shared, persistent retrieval cache (content-addressed flat files, safe across processes and nodes). Exact query match by default; fuzzy matching (FTS5 + rapidfuzz) is opt-in and shows the model which cached query a fuzzy hit answers.
  • Failed policy calls (HTTP errors, timeouts, connection loss, unparseable responses) end the episode and score 0, with the error recorded in policy_error, so no rollout is silently dropped.
• benchmarks/agentic_mme: manifest.yaml, config.yaml (judge binding, no round or tool-call limit), prepare.py, retrieval overlays, and a README with the measurement contract.

Tasks and licensing. 418 tasks: the full release of Crystal1047/Agentic-MME at pinned revision b9ea9d3f, unfiltered. The release's only split is named train but is an evaluation set. Domains include charts, architecture, news, music and anime, at levels L1 to L3. Dataset license: Apache-2.0 (dataset card). The reference harness (ChoS3nE11ven/Agentic-MME) declares no code license, so nothing is vendored: the tools are an independent implementation on Pillow (HPND), NumPy (BSD) and OpenCV (Apache-2.0). The five checked-in example tasks are templated colour-identification questions on generated solid-colour images; no model was used to generate task data.

Validation

Run against the final diff (branch merged with current main):

• pre-commit run --all-files: all hooks pass (ruff, ruff-format, import ordering, verified flag, README table, import-linter).
• gym env validate agentic_mme: manifest, components and 418 prepared rows OK; inferred profile custom-gym-agent-loop matches the declared one.
• gym env test agentic_mme: verifier fixture, 3 cases pass. gym env publish agentic_mme: publication checks pass (local only).
• gym env test +entrypoint=resources_servers/agentic_mme +should_validate_data=true: 32 passed, data validated. gym env test +entrypoint=responses_api_agents/agentic_mme_agent: 122 passed.
• scripts/add_legacy_agent_environment_servers.py --check: clean repo-wide.
• Component tests also pass with -W error on nemo_gym deprecation warnings (imports use the current module paths).

Rollouts

Smoke rollouts on the final code, policy Kimi K3 through an OpenAI-compatible endpoint, judge gpt-4o-mini, retrieval disabled, servers started with gym env start --config benchmarks/agentic_mme/config.yaml and collected with gym eval run --no-serve --agent agentic_mme_agent:

• Example data (resources_servers/agentic_mme/data/example_rollouts.jsonl): 5 of 5 correct. Each episode calls crop, receives the cropped image in the next turn, and answers in <answer> tags; judge and string match agree.

• First 5 benchmark tasks (--limit 5): 3 of 5 correct, no failed rollouts, no policy_error.

  Task	Gold	Answer	Tools (all succeeded)	Reward
  L1 Charts	Yes	Yes	none	1
  L2 Architecture	Conrad Wahn	Conrad Wahn	none	1
  L2 News	44.6 million	$44.6 million	none	1
  L1 Music	Rachel's	The Velvet Underground	crop ×8, enhance ×1	0
  L2 Anime	Kodansha USA	Seven Seas Entertainment	crop ×2	0

  The judge accepted the unit difference ($44.6 million) and rejected both wrong entities, matching string match on all five.

A full evaluation and reward profile are not included. No published result has been reproduced yet, so the manifest stays experimental: true.

Compatibility and benchmark impact

None for existing environments: all changes are new packages plus one new README table row. Retrieval is opt-in through the benchmark overlays and needs provider API keys only when enabled.

Proposed labels: feature, area:environment, benchmark-onboarding.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test b9b90d0

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5523-b9b90d0f842c.docs.buildwithfern.com/nemo/gym

[yfw]

resources_servers/agentic_mme/app.py:15

These imports name nemo_gym submodules, so a core unit test fails at this head.

AGENTS.md asks code outside nemo_gym/ to import public names from each package's public API. test_code_outside_the_library_imports_exported_names_from_the_public_api enforces that rule, and it lists 26 names in 13 import statements of this PR. The Core tests job runs this test for server-only changes too (unit-tests.yml), so that job fails once CI runs.

Suggested fix: import each name from its package. With the same change in the 3 files listed below, the test passes, and the 154 component tests still pass.

Suggested change

	from nemo_gym.config.config_types import ModelServerRef
	from nemo_gym.core.responses_api_types import NeMoGymChatCompletion
	from nemo_gym.server_runtime.base_resources_server import (
	BaseResourcesServerConfig,
	BaseVerifyRequest,
	BaseVerifyResponse,
	ReverifyMode,
	SimpleResourcesServer,
	)
	from nemo_gym.verification import VerifierFixture
	from nemo_gym.verification.judge import JudgeError, call_judge
	from nemo_gym.config import ModelServerRef
	from nemo_gym.core import NeMoGymChatCompletion
	from nemo_gym.server_runtime import (
	BaseResourcesServerConfig,
	BaseVerifyRequest,
	BaseVerifyResponse,
	ReverifyMode,
	SimpleResourcesServer,
	)
	from nemo_gym.verification import JudgeError, VerifierFixture, call_judge

The same change in the agent app.py, retrieval.py and the resources tests

responses_api_agents/agentic_mme_agent/app.py L16-L26:

from nemo_gym.agents import BaseResponsesAPIAgentConfig, Body, SimpleResponsesAPIAgent
from nemo_gym.client import get_response_json, raise_for_status
from nemo_gym.config import ModelServerRef, ResourcesServerRef
from nemo_gym.core import (
    NeMoGymEasyInputMessage,
    NeMoGymFunctionCallOutput,
    NeMoGymResponse,
    NeMoGymResponseCreateParamsNonStreaming,
    accumulate_response_usage,
)
from nemo_gym.server_runtime import BaseRunRequest

responses_api_agents/agentic_mme_agent/retrieval.py L43: from nemo_gym.client import request

resources_servers/agentic_mme/tests/test_app.py: L8 from nemo_gym.core import NeMoGymResponse, L106 from nemo_gym.core import NeMoGymChatCompletion and L184 from nemo_gym.verification import JudgeError.

scripts/migrate_imports.py does not make this change, because these are current paths, not moved paths.

[DanialTaheri]

Done in 4d1cd0b5e. Every nemo_gym name is now imported from its package, and test_public_api_facades passes.

[yfw]

resources_servers/agentic_mme/app.py:29

This server has no task_data.py, so two core data tests fail at this head.

A resources server ships a task_data.py module next to its app.py exporting a single
symbol TaskData

That is the contract in nemo_gym/data/task_data.py. The agent names agentic_mme as its resources_server (configs/agentic_mme.yaml), so this server owns the schema of the example rows and the benchmark rows. test_every_resources_server_ships_a_schema fails with "resources servers without a task_data.py schema: ['agentic_mme']". test_all_committed_rows_validate fails with "agentic_mme: no loadable schema".

Today nothing validates a row's golden_answer before a run. A bad gold value becomes reward 0.0 with failure_reason="invalid_golden_answer" (app.py:195-199), and that row counts in the mean.

Suggested fix: add this resources_servers/agentic_mme/task_data.py. It owns GoldenAnswer, so the schema and the verifier share one class. It adds a file, so it cannot be a suggestion block.

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Agentic-MME row fields: grading data inside verifier_metadata, and the agent's optional retrieval replay."""

import math
from typing import Any, Literal

from pydantic import BaseModel, ConfigDict, Field, field_validator


class GoldenAnswer(BaseModel):
    value: str | int | float
    match_type: Literal["exact", "contains", "numeric"] = "contains"
    tolerance: float = Field(default=0.0, ge=0, allow_inf_nan=False)

    @field_validator("value", mode="before")
    @classmethod
    def valid_target(cls, value: Any) -> str | int | float:
        if isinstance(value, bool) or not isinstance(value, (str, int, float)) or not str(value).strip():
            raise ValueError("golden_answer.value must be a nonempty string or finite number")
        if isinstance(value, float) and not math.isfinite(value):
            raise ValueError("golden_answer.value must be finite")
        return value


class TaskData(BaseModel):
    model_config = ConfigDict(extra="allow")

    golden_answer: GoldenAnswer = Field(json_schema_extra={"legacy_location": "verifier_metadata"})
    # The agent reads process_evaluation.efficiency.reference_tool_calls for the overthink metric.
    process_evaluation: dict[str, Any] | None = Field(
        default=None, json_schema_extra={"legacy_location": "verifier_metadata"}
    )
    task_id: str | None = Field(default=None, json_schema_extra={"legacy_location": "verifier_metadata"})
    level: int | None = Field(default=None, json_schema_extra={"legacy_location": "verifier_metadata"})
    domain: str | None = Field(default=None, json_schema_extra={"legacy_location": "verifier_metadata"})
    synthetic: bool = Field(default=False, json_schema_extra={"legacy_location": "verifier_metadata"})
    # Read by the agent's run request at the row's top level when retrieval.mode is replay.
    retrieval_replay: list[dict[str, Any]] = Field(default_factory=list)

In app.py, please delete GoldenAnswer (L29-L41) and import it with from resources_servers.agentic_mme.task_data import GoldenAnswer. Then please drop the unused Literal, BaseModel and field_validator imports.

Why `level` is an int and why the schema declares `synthetic`

The pinned release stores meta.level as 1, 2 or 3 in all 418 tasks. With level: str, all 418 prepared rows fail the schema. With int | None, all 418 pass. The drift test also rejects keys that the schema does not declare, so the five example rows need synthetic. With this file, both core tests pass, and a row with {"value": ""} or match_type: "exact_match" fails at collate time instead of scoring 0.

[DanialTaheri]

Done in 387379d0a. task_data.py exports TaskData, which validates golden_answer (value, match type, tolerance) and the other verifier_metadata fields. A malformed gold answer now fails data validation instead of scoring 0. All 418 prepared rows and the committed example rows validate, and both core data tests pass.

[yfw]

resources_servers/agentic_mme/app.py:215

At the PR head, a failed judge call scores the answer 0 here, so a judge outage counts against the model.

Gym wraps every verify in judge_failsafe. It turns a JudgeError into a masked judge_failed row, because "A failed judge call is a distinct outcome, not a wrong answer" (judge.py). This block catches the error first. With the judge unreachable, <answer>44.6 million</answer> for gold 44.6 million returns reward: 0.0, string_match_reward: 1.0 and mask_sample: false. The row then counts in mean/reward. If OPENAI_API_KEY is unset, the judge answers HTTP 401, and every answered task scores 0.

The add-benchmark skill at this PR's merge base already asks for this (SKILL.md):

For an environment failure (sandbox down, judge unreachable), do not score 0.0. Return mask_sample, failure_kind (as <server>:<kind>) and failure_reason so the row leaves the aggregate. Add one test case that masks a row

On current main, call_judge raises a judge_failed event instead of JudgeError (#5072, judge.py on main), so after a merge with main this block catches nothing.

Suggested fix: let the error reach judge_failsafe.

Suggested change

	try:
	judged = await self.judge(
	question_text(body.responses_create_params), str(golden.value), result.extracted_answer, final_text
	)
	except JudgeError as exc:
	result.reward = 0.0
	result.failure_reason = "judge_error"
	result.judge_reason = str(exc)[:500]
	return result
	# A failed judge call propagates: judge_failsafe masks the row instead of scoring the answer 0.
	judged = await self.judge(
	question_text(body.responses_create_params), str(golden.value), result.extracted_answer, final_text
	)

Please also drop JudgeError from the nemo_gym.verification import (line 25). Please also replace test_judge_error_scores_zero with this test through judge_failsafe. It needs import json at the top of the file.

@pytest.mark.asyncio
async def test_judge_error_is_masked_through_the_verify_route(judged_server, monkeypatch) -> None:
    from nemo_gym.verification import JudgeError, judge_failsafe

    async def failing_call_judge(*args, **kwargs):
        raise JudgeError("judge down")

    monkeypatch.setattr("resources_servers.agentic_mme.app.call_judge", failing_call_judge)
    # /verify serves judge_failsafe(self.verify).
    result = await judge_failsafe(judged_server.verify)(body=judged_request("<answer>44.6 million</answer>"))
    data = json.loads(result.body)
    assert data["reward"] == 0.0 and data["mask_sample"] is True and data["failure_kind"] == "judge_failed"

Steps from a failed judge call to an unmasked zero, and the analogs on main

1. AgenticMMEServer.verify calls self.judge (app.py L214-L218).
2. At this head, call_judge wraps any failure as JudgeError (judge.py L68-L107).
3. The handler sets reward 0.0 and returns (app.py L219-L223), so judge_failsafe (base_resources_server.py L568) never sees the error.
4. mask_sample stays false, so select_measured keeps the row in the quality metrics (reward_profile.py L888).
5. NeMo RL drops a row from the loss only through mask_sample (rollouts.py), so it trains on this row as a wrong answer.

On main, simpleqa raises JudgeError and lets judge_failsafe handle it (simpleqa/app.py L284-L286). opencode_search retries and then raises "rather than recording a fabricated zero" (opencode_search/app.py L400-L408).

[DanialTaheri]

Done in 971d64983. The except JudgeError block is gone, so judge_failsafe turns a failed judge call into a masked judge_failed row. test_judge_error_goes_to_failures_sidecar covers it.

[yfw]

benchmarks/agentic_mme/prepare.py:83

The match type comes from the last checkpoint's answer_check, but the released Track B scorer reads it from golden_answer.

The PR's own matcher states the contract (matches_answer):

Match the released evaluator's exact/contains/numeric task semantics.

The paper says normalized exact match, but the released dataset/evaluator
default to contains.

The released evaluator scores final answers in check_final_answer_only, which reads golden.get("match_type", "contains"). In the pinned release, golden_answer has only value in all 418 tasks, so the reference uses contains for every task.

This code writes exact for 202 tasks and numeric for 132. Task 0004 (gold Yes) gets string_match_reward 0.0 for yes. Task 0009 (gold 202) gets 0.0 for 202 people. The reference marks both correct. Without a judge, this value is the reward.

Suggested fix: read the match type from golden_answer, as the reference does. MATCH_TYPES on line 38 is then unused, so please delete it and its comment.

Suggested change

	answer_checks = [
	c["answer_check"] for c in task["process_evaluation"].get("checkpoints", []) if "answer_check" in c
	]
	match_type = MATCH_TYPES.get((answer_checks[-1].get("match_type") if answer_checks else None) or "", "contains")
	return {
	"responses_create_params": {
	"input": [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": content}]
	},
	"verifier_metadata": {
	"task_id": task["task_id"],
	"level": task["meta"].get("level"),
	"domain": task["meta"].get("domain"),
	"golden_answer": {"value": task["golden_answer"]["value"], "match_type": match_type},
	golden = task["golden_answer"]
	# Track B accuracy (the reference check_final_answer_only) reads golden_answer's own match_type and
	# tolerance, and scores any other label as contains.
	match_type = golden.get("match_type") if golden.get("match_type") in ("exact", "numeric") else "contains"
	return {
	"responses_create_params": {
	"input": [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": content}]
	},
	"verifier_metadata": {
	"task_id": task["task_id"],
	"level": task["meta"].get("level"),
	"domain": task["meta"].get("domain"),
	"golden_answer": {
	"value": golden["value"],
	"match_type": match_type,
	"tolerance": float(golden.get("tolerance", 0.0)),
	},

Where the reference scorer and the READMEs say this

• Track B "Final Answer Accuracy" is check_final_answer_only (eval_runs_search.py L428-L504), counted in aggregate_shards.py L74. It scores any label other than exact, contains and numeric as contains.
• The resources-server README shows match_type inside golden_answer as "the released shape" (README L36-L45).
• The benchmark README presents string_match_reward as the released evaluator's string match (README L40-L42).
• On the pinned release, this fix writes contains for all 418 rows, which matches the reference. It also keeps a golden tolerance if a later release adds one.

[DanialTaheri]

Agreed. Done in 0c29c6783. golden_answer is now the release's value only, so string_match_reward uses "contains" for every task, as check_final_answer_only does. MATCH_TYPES is removed. The author's numeric tolerance from the final answer check moves to verifier_metadata.answer_tolerance (29 tasks) and is shown only to the judge.

[yfw]

responses_api_agents/agentic_mme_agent/app.py:136

A failed policy call on the serving side scores 0 like a wrong answer, so outages lower the score and give RL a false zero.

HTTP 408, 425, 429 and 5xx replies end the episode with reward: 0.0, mask_sample: false and no failure_kind. Lost connections, timeouts and unreadable bodies end it the same way. Gym's own vLLM model server answers 503 when vLLM aborts a generation, with "vLLM cancelled the generation; retry the rollout." (vllm_model/app.py). Here that 503 becomes a scored zero. A wrong policy API key gives HTTP 401 on every call, and the benchmark then reads 0.0 instead of failing. NeMo RL reads only mask_sample (rollouts.py), so it trains on these rows as wrong answers.

Gym's default agent outcome policy grades only 400, 413 and 422 replies, retries serving failures and fails the rest (agent_outcome.py). Commit 4831dc6a made the failures without an HTTP status score 0, because rollout collection left a raised /run out of the mean. A raised /run still lands in the failures sidecar, and health counts it. To count failed rollouts in the mean, rollout collection already has count_failure_classes_as_zero (rollout_collection.py).

Suggested fix: keep the scored 0 only for the statuses that Gym's default policy grades, and add failure_kind. Let every other failure fail /run, so collection and trainers retry or report it. The change also touches L283 and needs imports and a module constant, so it is a plain block. Its import lines already include the public-API import change for this file. The ten test_failed_model_call_without_status_scores_zero cases then need to expect the error.

from nemo_gym.agents import (
    AgentOutcomePolicyConfig,
    BaseResponsesAPIAgentConfig,
    Body,
    OutcomeAction,
    SimpleResponsesAPIAgent,
)
from nemo_gym.client import get_response_json, raise_for_status
from nemo_gym.config import ModelServerRef, ResourcesServerRef
from nemo_gym.core import (
    MODEL_REQUEST_ERROR,
    NeMoGymEasyInputMessage,
    NeMoGymFunctionCallOutput,
    NeMoGymResponse,
    NeMoGymResponseCreateParamsNonStreaming,
    accumulate_response_usage,
)
...
# Gym's default outcome policy grades a refused policy request with these statuses (400, 413, 422).
_OUTCOME_POLICY = AgentOutcomePolicyConfig()


class AgenticMMEAgentConfig(BaseResponsesAPIAgentConfig):
...
            response = await self.server_client.post(
                server_name=self.config.model_server.name,
                # Under token capture this is /ng-rollout/<id>/training-token-capture/v1/responses, so the
                # policy model server records every call of this rollout; otherwise plain /v1/responses.
                url_path=model_url_path,
                json=new_params,
                cookies=cookies,
            )
            if not response.ok and _OUTCOME_POLICY.by_status_code.get(response.status) == OutcomeAction.VERIFY:
                # The policy's own request was refused (a full context, or too many images for the endpoint).
                # The episode ends unanswered and scores 0.
                error_body = (await response.read()).decode(errors="replace")
                policy_error = f"HTTP {response.status}: {error_body[:1000]}"
                break
            # Any other failure (408, 429, 5xx, 401, a lost connection, a body that is not a Response) is not the
            # policy's answer. It fails /run, so rollout collection and NeMo RL retry it or report it.
            await raise_for_status(response)
            reply = NeMoGymResponse.model_validate(await get_response_json(response))
...
                **({"failure_reason": "policy_error", "failure_kind": MODEL_REQUEST_ERROR} if policy_error else {}),

What each policy-call failure becomes with this change, and why it raises instead of masking

Failure	At this head	With this change
400 (for example a full context), 413 request too large, 422	0, not masked, no failure_kind	0, failure_kind: model_request_error
401 bad key, 403, 404 wrong model	0 on every row, so the run reads 0.0	/run raises, so the run fails loudly
408, 425, 429, 5xx (for example a vLLM abort)	0, not masked	/run raises, and collection and NeMo RL retry it
lost connection, timeout, a body that is not a Response	0, not masked	/run raises, and collection and NeMo RL retry it

A 400 for too many images still scores 0. The max_images_per_request change at app.py:41 removes that cause when you set it to the server's image limit.

Under NeMo RL's default masked_reward_policy: include (grpo.py), a masked row still votes in the group baseline (grpo.py). A raised /run reaches NeMo RL as an infrastructure failure, and the SingleController retries the prompt under its infrastructure retry limit (rollout_manager.py). The sync path retries the whole batch instead (rollouts.py).

On main, simple_agent raises out of /run (simple_agent/app.py), and vcqa_agent masks model-server errors (vcqa_agent/app.py). Gym's harness guide leaves a provider or runtime failure to the Environment Server, which decides whether to grade it or retry it (add-or-update-a-harness.mdx). Behind legacy_agent, the agent has to make that decision itself. The verify response contract sets mask_sample "when the environment or its infrastructure failed rather than the evaluated system" (base_resources_server.py).

[DanialTaheri]

Done in 971d64983. A serving-side failure raises PolicyRequestError, so the row goes to the failures sidecar and is retried on resume. That covers 401, 408, 425, 429, 5xx (including vLLM's 503), timeouts, lost connections, unreadable bodies, and any 400 that isn't a context limit. Only a context or request-size refusal still ends the episode and scores 0.

[yfw]

responses_api_agents/agentic_mme_agent/retrieval.py:459

The ImgBB Lens path encodes the PNG on the event loop, which pauses every other rollout on this server.

The SerpAPI branch above already moves its encode to a thread (retrieval.py:450). With a 10 ms ticker on the loop, one google_lens_search stalled it for 0.45 s on a 2048x1536 image. A 4000x4000 image stalled it for 2.1 s. The SerpAPI path stalled it for 0.02 s on both.

Suggested fix:

Suggested change

	data_url = image_data_url(image)
	data_url = await asyncio.to_thread(image_data_url, image)

[DanialTaheri]

Resolved by removing the ImgBB path in 6306b56fd. Google Lens runs only through SerpAPI's Image API, with the JPEG encoding in a thread.

[yfw]

responses_api_agents/agentic_mme_agent/app.py:64

Replay of a run recorded with Tavily or Exa offers google_lens_search, so the replayed prompt differs from the recorded one.

With retrieval_tavily.yaml and mode: live, the model sees 16 tools. With the same overlay and mode: replay, it sees 17, because this line removes Lens only in live mode. A replay is then not the same experiment, and a Lens call ends in "retrieval replay mismatch". The README presents replay as the way to repeat a run (README). With the fix, a Serper replay still offers Lens.

Suggested fix:

Suggested change

	if self.config.retrieval.mode == "live" and not self.config.retrieval.lens_available():
	if not self.config.retrieval.lens_available():

[DanialTaheri]

Done in 126df0122. Google Lens is now always offered with text search, in live and replay mode alike, so a replay offers exactly the recorded run's tools. Live retrieval therefore requires the SerpAPI key and upload consent. The Lens-less Tavily and Exa overlays are removed.

[yfw]

benchmarks/agentic_mme/retrieval_cache.yaml:16

If AGENTIC_MME_SEARCH_CACHE is not set, this overlay turns the cache off and gives no error.

An empty cache_root means "no cache" (retrieval.py L100, L296-L300). So a run that lists this overlay but misses the variable sends every query to the paid provider. The only sign is that tool outputs have no "cache" key.

Suggested fix: drop the empty default, so configuration resolution names the missing variable. OmegaConf 2.3.0 raises when the variable is missing and there is no default (resolvers). Without the variable, this overlay does nothing, so this blocks no working run.

Suggested change

	cache_root: ${oc.env:AGENTIC_MME_SEARCH_CACHE,""}
	cache_root: ${oc.env:AGENTIC_MME_SEARCH_CACHE}

[DanialTaheri]

Done in 3a5049c62. The empty default is gone, so a missing AGENTIC_MME_SEARCH_CACHE fails at configuration time.

[yfw]

responses_api_agents/agentic_mme_agent/sqlite_fuzzy_index.py:37

This file looks adapted from another code base, and its docstrings cite files and runs that are not in Gym.

The header says 2025 and carries the long Apache text (L1-L14), unlike the other new files. The docstrings cite bench/parity_recall.py (L37, L70), "leg 6562317" (L20-L25) and "run 6677570" (L144). None of these exist in Gym. The PR body says that nothing from the Agentic-MME reference is vendored, but it does not mention this file.

AGENTS.md "Code provenance" asks for the source URL, the commit and the license in the PR description and in the file header.

Suggested fix: please name the source repository and commit in the file header and the PR description. Please also drop the references to bench/parity_recall.py and the leg and run IDs. This is a docstring change, so it is not a suggestion block.

[DanialTaheri]

The file is adapted from an internal NVIDIA training codebase (same copyright holder, Apache-2.0), not from the Agentic-MME reference. 3c41afae6 switches it to the standard two-line SPDX header.

[yfw]

resources_servers/agentic_mme/tests/test_app.py:202

No component test runs VERIFIER_FIXTURE, so CI never runs its three cases.

VERIFIER_FIXTURE (app.py:232-236) runs only from gym env test agentic_mme, which needs the gitignored 418-row file. So CI and a fresh checkout never run it. On main, 11 of the 12 servers that export a fixture also run it in their component tests, for example chembench. The scaffold template generates that test too (scaffold.py). For example, a change from <= to < in the numeric match at app.py:69 scores 2225 against gold 2225 as 0. All 32 resources-server tests still pass with that change, and only the fixture's full_reward case catches it.

Suggested fix: add the fixture test at the end of this file.

Suggested change

	assert result.reward == 0.0 and result.failure_reason == "missing_answer"
	assert result.reward == 0.0 and result.failure_reason == "missing_answer"
	@pytest.mark.asyncio
	async def test_verifier_fixture() -> None:
	from nemo_gym.verification import exercise_verifier_fixture
	from resources_servers.agentic_mme.app import VERIFIER_FIXTURE
	results = await exercise_verifier_fixture(
	VERIFIER_FIXTURE, reward_range=(0.0, 1.0), higher_is_better=True, determinism="unknown"
	)
	assert [(r.kind, r.observed_rewards) for r in results] == [
	("full_reward", (1.0,)),
	("zero_reward", (0.0,)),
	("malformed", ()),
	]

[DanialTaheri]

Done in 3a5049c62. The component tests now run exercise_verifier_fixture(VERIFIER_FIXTURE, ...), so CI covers all three cases.

[DanialTaheri]

Thanks for the thorough review. All 17 comments are addressed at 3c41afae6; each thread names its commit. Main is merged in (410dee600), and both components now ship pyproject.toml + uv.lock. Other changes since your review:

• Tool images sent to the model are capped at 2048×2048 (5ab8ee478).
• Search results reach the model as the reference harness's compact text, without the answer box (d3db4a6fc).
• The leak guard is invisible to the model (a7a6a88c9).
• Google Lens is always offered with text search (126df0122).

Could you trigger CI on the new head?