tracking: audit reward-zero and fake-success failure producers #4883

Activity

1. yuhezhang-ai

   self-assigned this

   on Oct 21, 2026
2. yuhezhang-ai

   added a parent issue on Oct 21, 2026

   • refactor(core): replace reward-zero failure sentinels with typed outcomes #3180
3. github-project-automation

   added this to NeMo Gym 0.8.0 - Nov 4and NeMo Gym 0.7.0 - Oct 7on Oct 21, 2026
4. github-project-automation

   moved this to Dev Todo in NeMo Gym 0.7.0 - Oct 7on Oct 21, 2026
5. github-project-automation

   moved this to Dev Todo in NeMo Gym 0.8.0 - Nov 4on Oct 21, 2026
6. anwithk

   added

   Rollout

   on Oct 29, 2026
7. yuhezhang-ai commented on Oct 30, 2026

   

   yuhezhang-ai

   on Oct 30, 2026

   Last edited by yuhezhang-ai

   ContributorAuthor

   More actions

   Inventory: failures scored as results, at main 837ce27ea (Oct 28)

   This is the audit this issue tracks: every place in Gym where a failure (sandbox, judge, verifier, provider, bad task data, harness crash) can be scored as a model result. The fixes fall under #3180; the plan for them is under "Next for #3180" in the #2750 status refresh.

   Full tables: https://gist.github.com/yuhezhang-ai/bec5538e3982f932c8c50848fbdef8e5 (open-findings.csv: the 421 open findings with file and line; components.csv: one line per component). Search them for your component.

   How main treats a result

   • A row with _ng_failure_class goes to the failures file. It isn't scored and can be retried.
   • On the legacy_agent relay, an exception from an agent's /run ends the collection run, unless route_failures_to_sidecar=true turns it into a failures-file row. On an environment server (single_agent_turn, single_agent_turn_legacy), a failed /verify becomes a failure row instead, retried when the error is transient.
   • mask_sample=true keeps the row but leaves it out of scores and the training loss. failure_kind says why.
   • Everything else is scored.

   Results

   All 237 components were reviewed (169 resources servers, 62 agents, 6 environment servers), plus 21 shared nemo_gym/ modules. Each component is counted under its most serious finding:

   Most serious finding	Components
   A failure scored as a model result	119
   A failure turned into a successful-looking response	12
   Unclear whose failure it is; needs a decision	40
   Masked without a failure_kind (#4882)	3
   Deliberate, documented zero or mask	14
   Failures handled correctly	10
   Zeros are real wrong answers	33
   No scoring path	6

   So 171 of 237 components have at least one open finding. 12 components mask some failures without a failure_kind; #5223, #5315 and #5342 cover 7 of them.

   Where it matters most

   Of the 122 resources servers with an open finding:

   • 38 ship a training split in their Gym config, so their false zeros reach training rewards. 13 are coding (SWE) environments (r2e_gym, swe_gym, swe_rebench, scale_swe, swe_next, the swemer servers, swerl_gen, swe_pivot, denovo_swe, swe_internal_v1). 25 are judge- or rule-based, such as math_with_judge, instruction_following, mcqa and workplace_assistant. Together they have 97 open failure paths, 59 of which need no decision.
   • 56 are used only by benchmarks.
   • 28 are in neither.

   A training split in Gym's config is a lower bound: any environment can be trained on with other data.

   What goes wrong

   Failure	Example	Components
   Broken task data is scored 0	mcqa: a row with no expected answer scores 0	41
   A failed agent run reaches the grader as if it were a normal answer	codex_agent sends an empty "completed" message after a failed run	36
   Sandbox or grading failures in coding benchmarks are graded as failing tests	r2e_gym: the grading command reports a sandbox error without raising, so it is scored wrong and not retried	25
   The scorer itself crashes	bigcodebench: the test runner is killed by the out-of-memory killer and scored wrong	22
   Session state is lost before grading	newton_bench: session data is missing at grading time	13
   The judge replied, but its verdict is empty or unreadable	beam: an empty judge reply scores that rubric item 0	12
   A judge error is caught inside the server and scored 0	text_to_sql: an error in judge bookkeeping returns reward 0	8
   A tool or provider outage is shown to the model as a tool error	enterpriseops_gym: tool endpoint server errors reach the model	6
   Failure details are dropped on the way to the output	ns_tools, aviary, tau2	3
   A timeout that could be the model's or the machine's	2 clear cases, plus 18 that need a decision	2

   A component can appear in several rows.

   Fixed since a first pass on Oct 23

   • arena: judge outages go to the failures file (fix(arena): propagate judge outages and cancel sibling calls #5073).
   • interactive_browser: a missing session at verify is masked as session_lost (fix(resources): move swebench_pro and interactive_browser onto the session contract #4692).
   • swebench: staged and committed fixes are graded (fix(swebench): grade staged and committed fixes, not just unstaged changes #4775).
   • harbor: a failed agent response with a structured outcome follows the outcome policy (feat(harbor): multi-turn tasks, Harbor [[steps]] as turns (M34) #4819).
   • A shared helper for lost sessions, lost_session_verification (fix(resources): tighten the session contract for close, refusals and lost sessions #4679), already used by gdpval, nemo_user_sim, swebench_pro and partner_if.
   • After this snapshot, swe_atlas_qa masks a run whose judge scored no rubric (fix(swe-atlas): mask inconclusive grading and preserve cleanup retries #5105).

   How to fix a failure path

   The proposed fix column in open-findings.csv follows one rule, applied to every place a component returns 0 or masks:

   When the grader sees	It should
   The model's own failure: a wrong answer, no answer, or the model ran out of its own budget	Score it. 0 is the right result.
   A failure the model can't cause, where a retry may help: a sandbox error, a judge outage, a dropped connection	Raise a failure with a failure_kind, so Gym records it and can retry it.
   A failure the model can't cause, where a retry won't help: broken task data, judging turned off, a session that no longer exists	Mask it with a failure_kind.
   A failure that could be either	Follow Q1 below.
   A benchmark that must match an official score	Still record the failure, and opt in to counting it as 0 at scoring time (Q6).

   Most fixes are a few lines. Two shared pieces are still in review: #5072 carries a raised failure's kind and reason back to the collector, and #3302 saves failures and retries them on resume.

   Questions to settle

   The table refers to these as Q1–Q6.

   1. If a run fails after the model has started working, whose failure is it? For example, the sandbox dies mid-episode, or the tests hang and the patch may be the cause. Today benchmarks differ: fix(terminal_bench_4): score a verifier that times out or leaves no reward as 0, as Terminal-Bench does #5038 and fix(eval): make opted-in policy timeouts final scored zeros #5220 score some of these as 0, while swebench_pro masks them. Proposal: the environment states the fault when it knows; when it can't tell, the failure is marked unknown and doesn't count against the model by default. For failed agent runs, that default lives in agent_outcome_policy, which today sends a failed response with no outcome to the grader. feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts RL#4968 takes the same default for training: a masked row counts in GRPO advantages only if its failure_kind is a policy failure such as agent_timeout, agent_limit or model_context_limit. About 60 components.
   2. When a judge replies but its verdict can't be read, is that a judge failure? fix(frontierscience): use paper rubric prompt and reject failed grades #4891 and Fixes to partner_if benchmark, required to pass Frontier Eval quality checks #5007 treat it as one; most judges score it 0. Proposal: retry once, then mask with judge_unparseable. 12 components.
   3. Which failures are worth retrying, and which are final? Today masked rows are never retried, and raised failures are retried by HTTP status, not by cause. Proposal: retry what a new attempt can fix (sandbox errors, judge outages); mask with a reason what it can't (bad task data, judging turned off).
   4. Do all consumers read mask_sample and failure_kind together? A caller that ignores mask_sample sees every masked failure as a real 0. See fix(grpo): keep masked Gym failures out of prompt-group statistics RL#4061, and feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts RL#4968, which reads failure_kind to keep masked infrastructure failures out of advantages.
   5. When a tool the model uses has an outage mid-episode, should the model see it? For example, a search API returns a server error, and the model sees it as a tool error it can't fix. Proposal: retry inside the tool, then end the rollout as an infrastructure failure.
   6. Benchmarks whose official scoring counts failures as 0. Only 13 of the 252 scored-failure paths look intentional; most are failures the model can't cause, on infrastructure the official harness never ran on, so counting them as 0 makes the score depend on cluster health. Proposal: still record the failure, and let the benchmark opt in to counting it as 0 at scoring time with count_failure_classes_as_zero, as benchmarks/hle_diamond does, noted in its README, and report the failure rate next to the score. That option covers failures-file rows only, so masked rows need the same option keyed by failure_kind. It matters for masked policy failures too: Gym's eval scores leave every masked row out, so a benchmark run through swe_agents doesn't count a run that hit its iteration limit as 0.
8. giuliolovisotto commented on Oct 30, 2026

   

   giuliolovisotto

   on Oct 30, 2026

   Contributor

   More actions

   @yuhezhang-ai @AdamRajfer I clarified the failure contracts: the direction is configurable OutcomePolicy at each boundary, extending #4834 and the shared OutcomePolicyConfig / choose_action now in #5072. Each boundary supplies documented defaults and supported actions; users/benchmarks can override them through Gym config, with partial overrides preserving the remaining defaults.

   For Yuhe’s questions, I suggest:

   • Q1: retain unknown attribution as unknown; configure the handling action separately, without treating model activity alone as proof of model fault.
   • Q3: configure retry/stop behavior, with bounded retries enforced by the executing component. Stopping retries must not implicitly score zero.
   • Q6: make benchmark-specific zeros an explicit policy choice, preserving the failure evidence.

   We still need to agree on those defaults, but they should be defaults of this shared mechanism rather than independent hard-coded producer rules. Supported actions and context can differ by boundary; accounting/recovery planning still enforces budgets and execution constraints. Q4 remains a consumer-contract requirement, and reporting must retain failed attempts even when retries succeed.