feat(core): report infrastructure failure from verify, not just a reward by waple0820 · Pull Request #2611 · NVIDIA-NeMo/Gym

waple0820 · 2026-08-19T09:13:00Z

BaseVerifyResponse carries a single scalar for the outcome, so an environment that knows it failed — lost session, unavailable judge, OOM-killed container, reset timeout — can only return 0.0, which is indistinguishable from a policy that genuinely scored zero.

The concept is not new to this repo. It exists in four private locations:

mask_sample: bool = False on SWEBenchWrapperInstanceConfig (responses_api_agents/swe_agents/app.py:272), also anyterminal_agent/app.py:108 and anyswe_agent/app.py:180, documented at swe_agents/README.md:670.
conversational_tool_use/simulation/app.py:112 keeps its own {"mask_sample": ...} dict.
nemo_gym/token_id_capture/delivery.py:49 already defines MASK_SAMPLE_KEY = "mask_sample" as a top-level rollout field, described there as what "a consumer reads to exclude a rollout from the loss".
NeMo-RL consumes only the first one, through full_result["instance_config"]["mask_sample"] (nemo_rl/experience/rollouts.py:229-241), which zeroes loss_multiplier in nemo_rl/algorithms/grpo.py:2104.

So core already agrees what the key is called and what it means; it is reachable only if your environment happens to be SWE-shaped or if token capture produced it. verl gets nothing.

This is the same objection raised on #2384 — that a fix "should be applied uniformly, to other benchmarks as well" rather than to one harness. Putting the field on the contract is what makes it uniform.

What this changes

1. The contract (nemo_gym/base_resources_server.py)

class BaseVerifyResponse(BaseVerifyRequest):
    reward: float
    mask_sample: bool = False              # is this reward a valid measurement
    failure_kind: Optional[str] = None     # which kind of failure, from #3179's vocabulary
    failure_reason: Optional[str] = None   # what happened, for a person

The three are orthogonal. mask_sample decides usability, failure_kind groups in logs and metrics, failure_reason explains one occurrence. A sample that degraded but was still measured validly names a kind and stays unmasked.

failure_kind validates against nemo_gym.failure_kinds and warns rather than rejects — an unregistered name is a migration signal, and failing the response over it would replace a visible wrong label with an invisible lost failure. An environment that needs something the shared set should not grow uses <server>:<kind>.

Gym states the fact and stops there. Masking, resampling, excluding from group statistics or dropping the group stays the training framework's decision — the three frameworks we checked genuinely differ.

swe_agents and anyswe_agent mirror their instance-config flag onto the contract field and keep the old location for one release. anyterminal_agent already surfaces it through TerminalBenchMetrics.

2. Something that reads it (nemo_gym/reward_profile.py, nemo_gym/rollout_collection.py)

A declaration nothing consumes is not worth much, and the consumer that matters is the one that publishes the score.

compute_aggregate_metrics() computes quality from the scored subset: the profiler input, a custom compute_metrics(), and key-metric selection all skip masked samples. Masked rows stay in the input and are reported as coverage instead — coverage/measured_rollouts|masked_rollouts and coverage/measured_tasks|fully_masked_tasks. A run where every sample is masked publishes coverage rather than inventing a zero, and a fully masked task is counted without contributing a group. mask_sample is stripped before profiling, so it is no longer averaged as though the flag were a measurement.

perf_summary deliberately keeps using every response, masked included: a masked sample still executed and still reported its latency, so observability coverage should not shrink because a rollout was excluded from scoring.

Progress telemetry gets the same split while a run is still going, over the results that reach the main rollout output:

progress/<agent>/reward is unchanged — it still averages over every rollout, so an existing dashboard reads the number it read before.
progress/<agent>/reward_unmasked is that same average with the masked rollouts removed.
progress/<agent>/masked_pct is how many were removed.
progress/<agent>/failed and /omitted count the attempts that never reach the main output at all, so a sidecar failure is never folded into a quality average as a reward of zero.

The gap between the two reward series is the score a run lost to its own infrastructure rather than to the policy. On the run that motivated this, 1172 of 1280 rollouts across steps 61–80 failed in session reset and entered GRPO as legitimate zeros; nothing in the metrics said so.

Every new key appears only once something is actually masked, failed or omitted, so a healthy run publishes exactly what it published before.

Scope

Run-level reconciliation is no longer in this PR. #3302 implements it for #2135, using the materialized inventory and attempt journal to account for omitted and unknown work offline — neither of which this contract can see. What remains here is the per-sample half.

Relationship to the other failure-handling work

feat: declare failure_reason on BaseVerifyResponse #2552 (merged) landed failure_reason and its comment already points here: "Machine-readable handling belongs to mask_sample/failure_kind." This PR is that half.
feat: add shared failure-kind registry #2728 owns failure_kind and the shared registry, and already reserves session_lost with source="#2611". No overlap — this PR does not touch failure_kind.
Make rollout failures bounded, visible, and recoverable #2750 lists this under Failure fields and artifact integrity. The design document's rule — "the producer returns the real result with mask_sample=True, a stable failure_kind, and a human-readable failure_reason" — is exactly what the three fields together express once this lands.

Compatibility

Defaults are inert; an unmodified environment behaves identically, and a run that masks nothing publishes exactly the keys it published before.

One inheritance detail worth naming: AnySweVerifyResponse inherits mask_sample from both SWEBenchMetrics and BaseVerifyResponse, and the metrics model wins the MRO, so the contract field resolved to Optional[bool] = None with no description on that response. It is restated on the subclass so the schema keeps the contract's type, default and description.

Tests

tests/unit_tests/test_base_resources_server.py — defaults, round trip, that a plain reward=0.0 is not implicitly masked, and that a degraded-but-scored rollout stays unmasked.
tests/unit_tests/test_aggregate_metrics.py — a masked zero does not halve the score, the flag is not published as a quality metric, a run that masks nothing is byte-identical, a fully masked run reports coverage instead of a zero, a fully masked task is counted but not scored, custom compute_metrics() sees only scored samples, and a fully masked run still reports observability coverage.
tests/unit_tests/test_rollout_collection.py — the disposition account closes for every combination and the residual never goes negative; a healthy run adds no keys; a partially masked agent reports its share and the score without it; a fully masked agent publishes no average; failed and omitted attempts are counted, never averaged.
responses_api_agents/anyswe_agent/tests/test_app.py — an end-to-end /run that constructs and serializes the response. Reverting the fix makes it fail with TypeError: got multiple values for keyword argument 'mask_sample'.

uv run --extra dev pytest tests/unit_tests -q → 3383 passed. Two failures on this branch, test_e2b_provider and test_enroot_provider, reproduce on a clean upstream/main and are unrelated. pre-commit clean.

copy-pr-bot · 2026-08-19T09:13:04Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

waple0820 · 2026-08-27T03:42:26Z

@ananthsub rebased onto main and reshaped around the three fields as you described them here: #2552 (comment)

This PR is mask_sample only. failure_reason landed in feat: declare failure_reason on BaseVerifyResponse #2552, failure_kind belongs to feat: add shared failure-kind registry #2728 — it touches neither, and it does not re-declare what main already has.
The contract comment now states the orthogonality, including the case you named: a rollout that degraded but was still scored legitimately keeps mask_sample=False while naming a failure_kind/failure_reason.
Added the half that reads it. The flag was a declaration nothing consumed, so rollout_collection now reports progress/<agent>/masked_pct and progress/<agent>/reward_unmasked next to the existing progress/<agent>/reward, which is unchanged. The gap between the two reward series is the score a run lost to its own infrastructure rather than to the policy. Both keys appear only once something is actually masked, so a run that reports no masking exports exactly what it exported before.

Two things worth checking against your design:

nemo_gym/token_id_capture/delivery.py:49 already defines MASK_SAMPLE_KEY = "mask_sample" as a top-level rollout field meaning "exclude this rollout from the loss", and token capture already keeps a masked count with a max_mask_fraction abort. So core agrees on the name and the meaning; an environment that reports the same fact through verify is simply invisible to that accounting today. This PR routes it into the metrics but deliberately leaves the abort guardrail alone — coupling env-reported masking to max_mask_fraction looks like your call, not mine.

In #2728 session_lost is reserved with source="#2611". This PR does not produce that kind — it produces mask_sample. The producer for session_lost is the browser environment in #1865, and it will set both once the registry exists.

Ordering is yours: happy to land after #2728 and drop anything that overlaps, or before it if that is easier.

ananthsub · 2026-08-27T21:49:03Z

 class BaseVerifyResponse(BaseVerifyRequest):
    reward: float

+    # True when ``reward`` does not reflect policy quality because the environment or its


This description is too specific to RL training. Gym evaluation and other library callers can consume the same field, while loss masking, resampling, and group handling are only some possible policies. It is also an ordinary comment, so it will not appear in Pydantic's generated schema.

instead, lets add this with Field(description=...) that defines the result semantics: whether this completed sample should be excluded from evaluation scores and other downstream quality calculations because its reward is not a valid measurement of the evaluated system

Done

One prerequisite this needed: AnySweVerifyResponse(SWEBenchMetrics, BaseVerifyResponse) inherits mask_sample from both, and SWEBenchMetrics wins the MRO, so on that response the field resolved to Optional[bool] = None with no description — the contract field was invisible in its schema whatever the base declared. Restated on the subclass so the type, default and description survive.

ananthsub · 2026-08-27T21:49:03Z


+
+def _masking_step_metrics(agent_name: str, total: int, unmasked: Counter) -> Dict[str, float]:
+    """Progress metrics for the rollouts an environment reported as masked.


This helper changes progress telemetry only; it does not make the final evaluation result mask-aware. Later, every persisted result is sent to /aggregate_metrics, where the generic profiler still averages masked rewards and treats mask_sample itself as a numeric metric. For example, a valid reward of 1 plus a masked reward of 0 still publishes mean/reward=0.5.

Please apply masking centrally in compute_aggregate_metrics(), including the input to custom compute_metrics() and key-metric selection. Masked rows should remain persisted for coverage, and fully masked tasks should publish coverage without inventing a zero quality score.

ananthsub · 2026-08-27T21:49:04Z

            metrics = agent_name_to_metrics[agent_name]
            metrics.update({k: v for k, v in result.items() if isinstance(v, (int, float)) and not k.startswith("_")})
            agent_name_to_counts[agent_name] += 1
+            if not result.get(MASK_SAMPLE_KEY):


reward_unmasked currently includes failure-sidecar and non-persisted results because these counters are updated for every returned dictionary. Please update quality metrics only for results eligible for the main rollout output. Track failed and omitted results separately.

ananthsub · 2026-08-27T21:49:04Z

                reward=1.0 if metrics.resolved else 0.0,
+                # Report it on the contract as well; `instance_config.mask_sample` stays
+                # for one release so existing consumers keep working.
+                mask_sample=bool(instance_config.mask_sample),


SWEBenchMetrics.model_dump() already contains mask_sample, so this explicit keyword followed by **metrics.model_dump() raises TypeError: got multiple values for keyword argument 'mask_sample' on every result. The value here also comes from instance_config, which remains at its default, while the computed decision is written to metrics.mask_sample.

Please resolve the value once from metrics.mask_sample, copy it into the existing compatibility instance_config field, exclude the key from the expanded metrics mapping, and pass it once at the top level.

An end-to-end /run test should construct and serialize the response so this path is covered.

Fixed.
We added the end-to-end /run test. Reverting the fix makes it fail with exactly the TypeError you predicted.

… telemetry Addresses review on NVIDIA-NeMo#2611. `mask_sample` moves from a comment to `Field(description=...)` so it reaches the generated schema, and the description states the result semantics rather than one consumer's policy: whether a completed sample should be excluded from evaluation scores and other downstream quality calculations because its reward is not a valid measurement of the evaluated system. `compute_aggregate_metrics()` now applies that centrally. Quality metrics, the input to a custom `compute_metrics()`, and key-metric selection all come from the scored subset; masked rows stay in the input and are reported as coverage. A run where every sample is masked publishes coverage instead of inventing a zero, and `mask_sample` is no longer coerced into a numeric metric of its own. The keys appear only once something is masked, so a run that masks nothing is unchanged. Progress accounting is restricted to results that reach the main rollout output. A sidecar failure and a kill-shaped omission are counted as such rather than averaged in as rewards of zero. `anyswe_agent` raised `TypeError: got multiple values for keyword argument 'mask_sample'` on every result, because `SWEBenchMetrics` already carries the key that was also passed explicitly. It also read `instance_config.mask_sample`, which keeps its default, while `_should_mask_sample()` writes the decision to `metrics.mask_sample`. The value is now resolved once from the metrics, mirrored onto the compatibility field, and passed once. `SWEBenchMetrics.mask_sample` is declared `Optional[bool] = None` and wins the MRO on `AnySweVerifyResponse`, so the contract field is restated there to keep its type, default and description. An end-to-end `/run` test constructs and serializes the response, and fails with the original `TypeError` without the fix. Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com>

…coring it zero `verify` returned `reward=0.0` whenever the browser session was missing, which is the conflation this environment exists to demonstrate: a rollout whose session was lost is not a policy that solved nothing, and downstream the two are identical. It now reports `failure_reason` on that path, using the field NVIDIA-NeMo#2552 added to `BaseVerifyResponse`. A browser that died mid-episode was worse than mis-scored. `_score` reads the live page, so a dead browser raised out of `verify`; only `JudgeError` is converted to a routed row, so that exception ended the whole collection run rather than the one rollout it belonged to. Browser reads are now wrapped and reported the same way. An unsupported scoring key still raises, because a dataset typo is a configuration error rather than an infrastructure failure, and failing on the first rollout is the intended behaviour. Building the response by spreading the request and also passing `reward` as a keyword was a latent `TypeError: got multiple values`: `BrowserVerifyRequest` allows extra fields, so a caller putting `reward` or `failure_reason` in the body crashed verify. Response-owned fields are now dropped from the spread. The same shape was found and fixed in `anyswe_agent` on NVIDIA-NeMo#2611. Closing the previous browser on a re-seeded session swallowed every exception. A browser we could not close is a resource the run still holds, so it is logged. Tests build and serialize the real response, since the collision only appears at construction time. Reverting any one of the three fixes fails a test. Signed-off-by: waple0820 <feng.wang@lexmount.com>

… telemetry Addresses review on NVIDIA-NeMo#2611. `mask_sample` moves from a comment to `Field(description=...)` so it reaches the generated schema, and the description states the result semantics rather than one consumer's policy: whether a completed sample should be excluded from evaluation scores and other downstream quality calculations because its reward is not a valid measurement of the evaluated system. `compute_aggregate_metrics()` now applies that centrally. Quality metrics, the input to a custom `compute_metrics()`, and key-metric selection all come from the scored subset; masked rows stay in the input and are reported as coverage. A run where every sample is masked publishes coverage instead of inventing a zero, and `mask_sample` is no longer coerced into a numeric metric of its own. The keys appear only once something is masked, so a run that masks nothing is unchanged. Progress accounting is restricted to results that reach the main rollout output. A sidecar failure and a kill-shaped omission are counted as such rather than averaged in as rewards of zero. `anyswe_agent` raised `TypeError: got multiple values for keyword argument 'mask_sample'` on every result, because `SWEBenchMetrics` already carries the key that was also passed explicitly. It also read `instance_config.mask_sample`, which keeps its default, while `_should_mask_sample()` writes the decision to `metrics.mask_sample`. The value is now resolved once from the metrics, mirrored onto the compatibility field, and passed once. `SWEBenchMetrics.mask_sample` is declared `Optional[bool] = None` and wins the MRO on `AnySweVerifyResponse`, so the contract field is restated there to keep its type, default and description. An end-to-end `/run` test constructs and serializes the response, and fails with the original `TypeError` without the fix. Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com>

…coring it zero `verify` returned `reward=0.0` whenever the browser session was missing, which is the conflation this environment exists to demonstrate: a rollout whose session was lost is not a policy that solved nothing, and downstream the two are identical. It now reports `failure_reason` on that path, using the field NVIDIA-NeMo#2552 added to `BaseVerifyResponse`. A browser that died mid-episode was worse than mis-scored. `_score` reads the live page, so a dead browser raised out of `verify`; only `JudgeError` is converted to a routed row, so that exception ended the whole collection run rather than the one rollout it belonged to. Browser reads are now wrapped and reported the same way. An unsupported scoring key still raises, because a dataset typo is a configuration error rather than an infrastructure failure, and failing on the first rollout is the intended behaviour. Building the response by spreading the request and also passing `reward` as a keyword was a latent `TypeError: got multiple values`: `BrowserVerifyRequest` allows extra fields, so a caller putting `reward` or `failure_reason` in the body crashed verify. Response-owned fields are now dropped from the spread. The same shape was found and fixed in `anyswe_agent` on NVIDIA-NeMo#2611. Closing the previous browser on a re-seeded session swallowed every exception. A browser we could not close is a resource the run still holds, so it is logged. Tests build and serialize the real response, since the collision only appears at construction time. Reverting any one of the three fixes fails a test. Signed-off-by: waple0820 <feng.wang@lexmount.com>

… telemetry Addresses review on NVIDIA-NeMo#2611. `mask_sample` moves from a comment to `Field(description=...)` so it reaches the generated schema, and the description states the result semantics rather than one consumer's policy: whether a completed sample should be excluded from evaluation scores and other downstream quality calculations because its reward is not a valid measurement of the evaluated system. `compute_aggregate_metrics()` now applies that centrally. Quality metrics, the input to a custom `compute_metrics()`, and key-metric selection all come from the scored subset; masked rows stay in the input and are reported as coverage. A run where every sample is masked publishes coverage instead of inventing a zero, and `mask_sample` is no longer coerced into a numeric metric of its own. The keys appear only once something is masked, so a run that masks nothing is unchanged. Progress accounting is restricted to results that reach the main rollout output. A sidecar failure and a kill-shaped omission are counted as such rather than averaged in as rewards of zero. `anyswe_agent` raised `TypeError: got multiple values for keyword argument 'mask_sample'` on every result, because `SWEBenchMetrics` already carries the key that was also passed explicitly. It also read `instance_config.mask_sample`, which keeps its default, while `_should_mask_sample()` writes the decision to `metrics.mask_sample`. The value is now resolved once from the metrics, mirrored onto the compatibility field, and passed once. `SWEBenchMetrics.mask_sample` is declared `Optional[bool] = None` and wins the MRO on `AnySweVerifyResponse`, so the contract field is restated there to keep its type, default and description. An end-to-end `/run` test constructs and serializes the response, and fails with the original `TypeError` without the fix. Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com>

cafzal · 2026-09-05T22:28:53Z

Independent evaluation evidence supports the core distinction in this PR: one rollout with an agent/infrastructure failure and its fresh recovery with a valid-but-wrong result both produced scalar reward 0. Treating both as ordinary zeros would hide invalid-execution denominator drift; treating both as failures would hide a real policy miss. A top-level, producer-owned masking fact is general across SQL, coding, browser, sandbox, tool-use, and judge-backed environments and avoids coupling consumers to a SWE-specific private config path.

The strongest validation properties are already reflected in this PR's plan: valid zero versus masked zero, inert defaults for unmodified environments, fully masked groups producing coverage rather than an invented score, custom metrics receiving only scored samples, and observability retaining all attempts. I would additionally preserve an exact reconciliation invariant across scored, masked, failed, omitted, and unknown dispositions as #2750 evolves.

Given that #2611 already owns the implementation, I will not open a duplicate PR for #2608.

…coring it zero `verify` returned `reward=0.0` whenever the browser session was missing, which is the conflation this environment exists to demonstrate: a rollout whose session was lost is not a policy that solved nothing, and downstream the two are identical. It now reports `failure_reason` on that path, using the field NVIDIA-NeMo#2552 added to `BaseVerifyResponse`. A browser that died mid-episode was worse than mis-scored. `_score` reads the live page, so a dead browser raised out of `verify`; only `JudgeError` is converted to a routed row, so that exception ended the whole collection run rather than the one rollout it belonged to. Browser reads are now wrapped and reported the same way. An unsupported scoring key still raises, because a dataset typo is a configuration error rather than an infrastructure failure, and failing on the first rollout is the intended behaviour. Building the response by spreading the request and also passing `reward` as a keyword was a latent `TypeError: got multiple values`: `BrowserVerifyRequest` allows extra fields, so a caller putting `reward` or `failure_reason` in the body crashed verify. Response-owned fields are now dropped from the spread. The same shape was found and fixed in `anyswe_agent` on NVIDIA-NeMo#2611. Closing the previous browser on a re-seeded session swallowed every exception. A browser we could not close is a resource the run still holds, so it is logged. Tests build and serialize the real response, since the collision only appears at construction time. Reverting any one of the three fixes fails a test. Signed-off-by: waple0820 <feng.wang@lexmount.com>

yuhezhang-ai · 2026-09-15T14:11:22Z

/ok to test ca37a95

yuhezhang-ai

Looks good to me. Thanks for addressing the review feedback and updating the server-test expectations! I reviewed the latest head (ca37a95a), and CI is now passing.

@ananthsub, could you please take a final look and make the merge decision?

github-actions · 2026-09-18T21:54:04Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35399030034

…coring it zero `verify` returned `reward=0.0` whenever the browser session was missing, which is the conflation this environment exists to demonstrate: a rollout whose session was lost is not a policy that solved nothing, and downstream the two are identical. It now reports `failure_reason` on that path, using the field NVIDIA-NeMo#2552 added to `BaseVerifyResponse`. A browser that died mid-episode was worse than mis-scored. `_score` reads the live page, so a dead browser raised out of `verify`; only `JudgeError` is converted to a routed row, so that exception ended the whole collection run rather than the one rollout it belonged to. Browser reads are now wrapped and reported the same way. An unsupported scoring key still raises, because a dataset typo is a configuration error rather than an infrastructure failure, and failing on the first rollout is the intended behaviour. Building the response by spreading the request and also passing `reward` as a keyword was a latent `TypeError: got multiple values`: `BrowserVerifyRequest` allows extra fields, so a caller putting `reward` or `failure_reason` in the body crashed verify. Response-owned fields are now dropped from the spread. The same shape was found and fixed in `anyswe_agent` on NVIDIA-NeMo#2611. Closing the previous browser on a re-seeded session swallowed every exception. A browser we could not close is a resource the run still holds, so it is logged. Tests build and serialize the real response, since the collision only appears at construction time. Reverting any one of the three fixes fails a test. Signed-off-by: waple0820 <feng.wang@lexmount.com>

@yaoyu-33

… scoring them zero Adopt the mask_sample/failure_kind contract from NVIDIA-NeMo#2611: an unparseable judge response is an infrastructure/judge failure, not a policy one, so it's now reported via mask_sample=True + failure_kind=JUDGE_UNPARSEABLE and excluded from quality metrics as lost coverage. An empty model generation stays an unmasked, genuine zero, per review from @yaoyu-33 on NVIDIA-NeMo#3348. Signed-off-by: Tanish Kushalkar <38599503+tanishku@users.noreply.github.com>

Closes #3179 Gym describes failures with free text and component-local labels, so the same failure reaches a collector, a log line and a metric under three different names and cannot be grouped. ## What a name is, and what it is not A name says what kind of thing went wrong and nothing else. Retryability, replay safety, whether a completed result should be masked, and what to tell a person are properties of the occurrence rather than of the name — they belong to the failure record, the verify response, and `failure_reason`. Keeping them out is the reason the registry carries no per-entry metadata: anything attached to a name goes stale the moment one caller wants to retry what another does not. That is also what the acceptance criterion "no retry-policy metadata that can become stale" asks for. Names are `<domain>_<condition>`, so a reader can tell an unreachable model server from an unreachable sandbox without a second field, and they are low cardinality, so they are safe as a metric label or a span dimension. `failure_reason` never is. ## Unknown names warn rather than raise `validate_failure_kind` returns the name unchanged and logs once. Rejecting an unregistered value would trade a visible wrong label for an invisible dropped failure, which is worse than the problem the vocabulary exists to fix, and it would break components mid-migration. An environment that needs something the shared set should not grow can use `<server>:<kind>`. ## The vocabulary describes the repo, not an intention `agent_run_error`, `agent_request_failed` and `judge_failed` are labels `main` emits today. Tests import those constants from `rollout_collection` and `rollout_reverification` and assert they are registered, so the set cannot drift from the code that produces it. ## Scope This adds the vocabulary only. `failure_kind` reaching `BaseVerifyResponse` is #2608's follow-on and lands with #2611, which rebases onto whatever merges here. Typed rollout outcomes referencing the same names are #2135. ## Relationship to #2728 This replaces rather than rebases #2728. That PR is on a fork branch we cannot push to, and #3179's last acceptance criterion is "PR #2728 is rebased or replaced under this issue". The vocabulary here is drawn from what `main` produces now, which has moved since #2728 was drafted — #2017 and #2883 introduced the `agent_request_failed` / `agent_run_error` split. The `status`/`source` fields are dropped for the reason above. ## Tests 16 cases: every name matches one convention and no two differ only by case or separator; every name states the layer that observed the failure; registered, unregistered and namespaced lookup; validation staying silent on known names and warning without raising on unknown ones; the set staying small enough to be a metric label and carrying no occurrence detail; the labels `main` already produces staying registered; and the module importing nothing from the server stack so data tooling can read it without installing a server's requirements. `uv run --extra dev pytest tests/unit_tests -q` → 4348 passed. `test_e2b_provider` and `test_enroot_provider` fail on a clean `upstream/main` too. --------- Signed-off-by: waple0820 <232305951+waple0820@users.noreply.github.com> Co-authored-by: waple0820 <232305951+waple0820@users.noreply.github.com> (cherry picked from commit 4402098) Signed-off-by: Peter Jin <pjin@nvidia.com>

nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Aug 19, 2026

This was referenced Aug 19, 2026

feat(observability): return the environment session id and allow correlation without full capture #2613

Closed

[trainer] feat: pass non_tensor_batch to advantage estimators that can accept it verl-project/verl#7473

Open

waple0820 force-pushed the feat/verify-response-mask-sample branch from 1d8d648 to d626da4 Compare August 25, 2026 03:49

ananthsub mentioned this pull request Aug 25, 2026

Make rollout failures bounded, visible, and recoverable #2750

Open

8 tasks

waple0820 force-pushed the feat/verify-response-mask-sample branch from d626da4 to f55c8a0 Compare August 27, 2026 03:41

waple0820 marked this pull request as ready for review August 27, 2026 03:42

ananthsub self-requested a review August 27, 2026 19:58

ananthsub requested changes Aug 27, 2026

View reviewed changes

svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Aug 27, 2026

waple0820 force-pushed the feat/verify-response-mask-sample branch from f95c306 to ceb910e Compare August 28, 2026 03:49

svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Aug 28, 2026

svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Aug 30, 2026

waple0820 mentioned this pull request Aug 31, 2026

feat(resources_servers): add interactive browser RL environment (local Playwright + remote CDP backends) #1865

Merged

8 tasks done

waple0820 force-pushed the feat/verify-response-mask-sample branch from ceb910e to df1d43f Compare August 31, 2026 06:15

waple0820 force-pushed the feat/verify-response-mask-sample branch from df1d43f to b05103d Compare August 31, 2026 06:15

waple0820 force-pushed the feat/verify-response-mask-sample branch from b05103d to 2bbc5a3 Compare September 1, 2026 06:59

cafzal mentioned this pull request Sep 5, 2026

Add NeMo Gym as an optional evaluation execution backend NVIDIA/auto-ontology-eval#16

Closed

svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Sep 6, 2026

waple0820 force-pushed the feat/verify-response-mask-sample branch from 2bbc5a3 to 80a4e17 Compare September 6, 2026 07:24

waple0820 force-pushed the feat/verify-response-mask-sample branch from 4add419 to ca37a95 Compare September 15, 2026 05:45

copy-pr-bot Bot temporarily deployed to public September 15, 2026 14:12 Inactive

copy-pr-bot Bot temporarily deployed to public September 15, 2026 14:13 Inactive

copy-pr-bot Bot temporarily deployed to public September 15, 2026 14:14 Inactive

yuhezhang-ai approved these changes Sep 15, 2026

View reviewed changes

svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 17, 2026

yaoyu-33 mentioned this pull request Sep 18, 2026

fix(grpo): keep masked Gym failures out of prompt-group statistics NVIDIA-NeMo/RL#4061

Open

6 tasks

yuhezhang-ai enabled auto-merge September 18, 2026 21:52

ananthsub approved these changes Sep 18, 2026

View reviewed changes

yuhezhang-ai added this pull request to the merge queue Sep 18, 2026

Merged via the queue into NVIDIA-NeMo:main with commit cba9a98 Sep 18, 2026
39 checks passed

svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Sep 19, 2026

yaoyu-33 mentioned this pull request Sep 19, 2026

feat(translation_with_judge): add Indic-focused LLM-judge translation… #3348

Open

5 tasks done

ananthsub mentioned this pull request Sep 21, 2026

fix(genrm): bound cohort deadlines and failure cleanup #3351

Merged

waple0820 mentioned this pull request Sep 22, 2026

Gym has no browser environment after four attempts — propose landing the browser-supply layer first #2921

Closed

wdykas mentioned this pull request Oct 4, 2026

fix(swe): mask evaluator launch failures #4002

Merged

7 tasks done

qiaochuz-nv mentioned this pull request Oct 13, 2026

fix(conversational_tool_use): start the nested simulation stack, constrain judge JSON, report mask_sample #4568

Merged

This was referenced Oct 21, 2026

fix: complete failure metadata on existing masked results #4882

Open

refactor(core): replace reward-zero failure sentinels with typed outcomes #3180

Open

tracking: audit reward-zero and fake-success failure producers #4883

Open



		def _masking_step_metrics(agent_name: str, total: int, unmasked: Counter) -> Dict[str, float]:
		"""Progress metrics for the rollouts an environment reported as masked.

Conversation

waple0820 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Scope

Relationship to the other failure-handling work

Compatibility

Tests

Uh oh!

copy-pr-bot Bot commented Aug 19, 2026

Uh oh!

waple0820 commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cafzal commented Sep 5, 2026

Uh oh!

yuhezhang-ai commented Sep 15, 2026

Uh oh!

yuhezhang-ai left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

waple0820 commented Aug 19, 2026 •

edited

Loading

waple0820 commented Aug 27, 2026 •

edited

Loading