[AdamRajfer]

When either Arena judge request fails, propagate the outage through existing judge-failure handling and cancel and await its sibling request. Check HTTP status before parsing the response so an HTTP error retains its actual status. Preserve the saved answer, native verdict scoring, parsing and retry limits.

flowchart LR
    A[Saved answers] --> B[Two judge orderings]
    B --> C[Both succeed: native score]
    B --> D[Either fails: cancel and await sibling]
    D --> E[Existing masked judge failure]

Loading

The LMArena v2/v3 configs set the shared non-final failure limit to 1%, and Vision Arena to 5%. The README distinguishes judge outages in the failures sidecar from missing or unparseable judgments counted by Arena's aggregation guard. Shared health marks an above-limit run unhealthy but still publishes its score; withholding that score is a separate follow-up. The existing aggregation guard is unchanged.

This PR stands on main and changes only Arena and its benchmark configs. Shared run state, latest-attempt selection and saved-answer reverification belong to #3302 and #3879.

Validation: 183 Arena tests and 50 health-limit tests pass. Actual benchmark config resolution reaches the collector with limits of 1%, 1% and 5%. Changed-path hooks pass. No model benchmark score is claimed.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5073-63f974f87bab.docs.buildwithfern.com/nemo/gym

[yuhezhang-ai]

Thanks Adam, this matches the split we discussed: Arena only, with the shared recovery left to #3302 and #3879. The cancel-and-await logic looks right, and the 182 Arena tests pass for me locally. Two things before merging:

[yuhezhang-ai]

Thanks Adam, both points are addressed. The judge's HTTP status is checked before parsing, and the Arena configs set the health limits, with the README saying what is and isn't counted. The 183 Arena tests pass for me on 69b605f.

One note for me, not for this PR: with #3302, an outage that has used up its retries counts as a final failure, so the not-final limit would stop catching it. I'll add the matching final limit to these three configs in #3302.

Generated by Claude Code

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40471986253