[willtong-aq]

Closes #3342. This PR implements the proposal in that issue.

What this adds

resources_servers/critpt_custom_grader: a resources server that grades a candidate response for a CritPt-style problem. It executes the reference preflight and the candidate code inside a Daytona sandbox. It then compares the results against stored expectations. It is added beside resources_servers/critpt, which stays unchanged. It does not call the Artificial Analysis grading service and it does not claim parity with the hidden CritPt grader.

How it works

1. /verify receives an already generated response. The grader never generates candidates.
2. Extraction: first python fence, else first bare fence, else raw text. A completed response with no code scores 0.0.
3. A reference preflight runs first. Candidate code runs only after the preflight succeeds.
4. Values return through a lossless codec. Dictionary keys of scalar types are stringified. Symbolic answers use a restricted grammar. No eval, no unrestricted sympify.
5. Comparison is per leaf. Tolerance precedence: statement promise, then per-leaf entry, then task-level tolerance, then the server default (rtol 5e-12, atol 0). Integer and boolean expectations compare exactly. String expectations compare as categories.
6. Exception expectations are scored. An expected exception passes when the candidate raises and fails when it returns a value.
7. Candidate faults after a successful preflight score 0.0. Reference faults and comparator uncertainty are unscorable and never score.
8. Every sandbox is owned, journaled, and cleaned up. Every sandbox command runs through the session API with polling, so a run longer than the provider's blocking exec ceiling is not lost. A row the grader could not score (provider_unavailable) is recovered by re-verifying the saved candidate without a new model call. A lost outer /run response is a documented at-least-once limitation (see README).
9. Rows that are not a clean score set _ng_failure_class and failure_kind to one of three names. provider_unavailable and verifier_error come from core's FAILURE_KINDS registry. critpt_custom_grader:response_incomplete uses the server prefix that the registry asks for.

Execution limits

Bound	Value
Suite wall clock	1800 s
CPU time	1800 s
Memory per process	1536 MiB
Job timeout	5900 s
Queue timeout	6020 s

The README documents how the derived bounds follow from the suite limit.

Core changes

None. The grader is self-contained against unmodified core. Two small local modules replace what earlier drafts patched into core: a Daytona provider subclass that reads a missing or non-integer exit code as a sandbox failure and keeps its diagnostics category-only, and a module that holds the grader's request policy and privacy settings and its private error responses.

Tests and evidence

• Grader suites: unit, integration (Daytona-backed), and a remote runner suite that runs only inside a sandbox with NG_CRITPT_RUNNER_REMOTE=1.
• Full Gym suite run in a sandbox from the uv.lock pins. One pre-existing failure (test_slurm_launcher_submits_one_dependent_cpu_cleanup_job) fails the same way on the base commit.
• Five real example rollouts in data/example_rollouts.jsonl, collected through simple_agent. example_metrics.json reports 5 examples.
• Acceptance: every row of a private set and the public set was graded in Daytona on the frozen build 20260916T023429Z, before the review fixes below. A 20-row smoke on the review-fix commit (10 public tasks, 6 reference answers and 4 canaries, plus 10 private rows) agreed with the earlier verdicts on all 20 rows, with every sandbox cleaned up.

Public set: the 70 answer rows, 73 alternates on their own test cases, 73 alternates against the primary test cases, and 70 canaries.

Private set, reward agreement with the historical grader: above 99.7% of rows. The 20 other rows are triaged in the acceptance report:

• 10 reference-side faults. The stored reference does not reproduce its own expectation. A task-data defect, unscorable by design.
• 5 deliberate tolerance-policy differences. Integer expectations compare exactly (3). The silent numeric default rtol 5e-12 is stricter than the baseline (1). A statement precision promise is honored (1).
• 1 comparator_uncertain on a symbolic answer, unscorable by design.
• 3 rows where the grader passes a submission the baseline rejected as wrong or too slow. 1 row where the candidate source fails under the pinned runtime while the reference runs.
  Re-runs on the frozen build: the 7 candidate timeouts reproduced at 1823 to 1844 s. Of the 4 rows above, 3 reproduced. 1 submission passed on the first pass and timed out at the 1800 s suite limit on the second pass. Its verdict is timing-bound.
  A further 54 rows differ only in the rejection label. Both graders give reward 0.

Public 70: all 70 primary solutions pass. All 70 canaries reject. 71 of 73 alternates pass on their own test cases. The 2 others are reference faults (public set question #2, alternate 0, and public set question #62, alternate 0). 71 of 73 alternates reject against the primary test cases. Public set question #68, alternate 0, passes and public set question #12, alternate 0, is uncertain.

Review fixes

Two independent model reviews were run and each finding was verified against the code before a fix. Fixed: the symbolic sampling fallback no longer accepts two expressions that differ by a branch function, an accuracy kind (absolute or relative) no longer bleeds across a sentence break, an unparseable cross pair no longer defeats a valid unordered matching, set inputs reach task code as Python sets, a symbolic infinity expectation is unscorable, a polynomial check that raises is treated as "not a polynomial", so the comparison goes on to the branch-function guard and then the bounded numeric fallback, a queue wait is bounded once, shutdown retries journal records recovered from disk, and a completed envelope with an incomplete final message is flagged for regeneration instead of scored zero.

Update 2026-09-23

• Rebased onto current main. The only conflict was the root README.md table, where the Deepsearchqa row landed next to this one. Both rows are kept.
• Failure-class names now follow nemo_gym/failure_kinds.py: verifier_unavailable → provider_unavailable, reference_failed → verifier_error, needs_regeneration → critpt_custom_grader:response_incomplete. failure_kind carries the same value. Routing, retry, and terminal behavior do not change.
• The README recovery section is corrected. The earlier text named a recover_failure_classes option that core does not have. Core's --judge-failed-only selects only judge_failed rows. The README now selects provider_unavailable rows from the failures file with jq and runs plain gym eval reverify on them, with no core change. The README also documents the policy_* overrides that gym eval reverify needs (it has no --model flag) and the quoting that gym eval aggregate needs for a comma list of files.

Scope notes

• Daytona is the only sandbox backend in this PR. The grader keeps its own SandboxBackend seam so other providers can follow.
• No private task or response data is in the repository.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yuhezhang-ai]

Thanks for adding this grading option. Being able to run CritPt with operator-supplied expectations and individual task scores would be useful. I reproduced several issues in the supplied Gym integration and answer comparison that should be addressed before merging; the inline comments include concrete examples and suggested fixes. The comments on masking unscorable responses and preserving error categories are smaller, non-blocking improvements.

Could you add regression coverage for these cases and validate the corrected configuration with a representative end-to-end model rollout on Daytona? My validation covered local tests and focused reproductions; live execution isolation still needs review by someone familiar with the sandbox runtime.

[yuhezhang-ai]

[P1] Add the required environment-server routing block

Please add the environment-server block that routes collection to this agent. With all four execution inputs supplied and a dummy model composed, the shipped YAML raises AgentWithoutEnvironmentServerError in GlobalConfigDictParser.parse(..., offline=True). Adding the usual legacy_agent environment server pointing to critpt_custom_grader_simple_agent makes the same parse succeed. The documented setup therefore cannot load on this revision. Please exercise the actual config parser in a regression test.

[yuhezhang-ai]

[P1] Accept the stock agent default when no tools are configured

Please allow tool_choice=auto when the tools list is empty. SimpleAgentRunRequest.model_validate(first_example_row).model_dump() fills this default, and simple_agent sends that full dump to seed_session and verify. This validator rejects it with tools are not supported, even though no tool can be called. Explicit none works, but the committed examples omit the field. Please test a request serialized by the real agent model so the default workflow is covered.

[yuhezhang-ai]

[P1] Exclude framework bookkeeping from task validation

Please strip framework bookkeeping before validating the task schema. normalize_task_fields does not remove _ng_attempt_index or _ng_environment_server at this base, and TaskData rejects extras. Adding either key to an otherwise valid committed example reproduces extra_forbidden in _task. Collection stamps the attempt key on resumed rows and the environment key for legacy routing, so those requests fail before grading. Preserve strict rejection of task-data typos while excluding framework-owned keys, and add resumed-request coverage.

[yuhezhang-ai]

[P1] Do not certify symbolic equality from fixed samples

Please keep unproven symbolic equality undecided instead of accepting agreement at fixed real samples. Production compare_request returns equal for x + sin(840pix) versus x, although the difference is exactly 1 at x=1/1680. It also accepts conjugate(x) versus assumption-free x, which differ at x=i. Both were reproduced with SymPy 1.14.0. Equal verdicts can become reward 1. Changing the fixed points or expanding the branch blacklist alone does not address the root problem; require an appropriate equivalence proof and add these regressions.

[yuhezhang-ai]

[P1] Keep unrelated percentages out of answer tolerances

Please avoid promoting an unrelated numerical statement to the output tolerance. With 'The trap frequency is stable to within 2% during the measurement. Report the ground-state energy with relative error below 1e-6.', expected 1, candidate 1.015, and authored rtol=1e-9, compare_request returns equal because the 2% clause becomes the blanket. Removing the first sentence makes the same answer mismatch. Only an accuracy requirement tied to the output should override its tolerance; ambiguous or incidental quantities must not widen it. Please add this mixed-prose regression.

[yuhezhang-ai]

[P2] Recognize simple mathematical legacy answers

Please handle mathematical legacy outputs such as -x, x, and unspaced x-1 consistently with their symbolic reference values. The current classifier leaves those legacy expectations as categorical strings, while the same SymPy output is parsed as an expression. Production reference-role comparisons for all three return invalid_reference, stopping a valid task before candidate execution. Preserve or correctly recognize the expected mathematical kind, retain explicit categorical behavior, and add reference-preflight regressions.

[yuhezhang-ai]

[P2] Mask unscorable responses for shared metric consumers (non-blocking)

Please set the inherited mask_sample flag for unscorable responses, consistently with scored=False. comparator_uncertain currently returns reward=0 and mask_sample=False; Gym's shared select_measured therefore retains it as a measurement. One success plus this unmeasured attempt implies mean reward 0.5 in that consumer. Normal collection's failure routing and this server's custom aggregation already filter it correctly, so this is a standard-consumer interoperability fix, not a claim that normal collection is corrupted.

[yuhezhang-ai]

[P3] Preserve the invalid-task error category (non-blocking)

Please retain task_invalid when the controller reports an invalid expected value or comparison policy, and map only reference_mismatch to reference_invalid. A task_invalid GradeResult currently projects to reference_invalid here, which directs operators to the reference solution when the defect is in the stored expectation or policy. This can be corrected without changing reward or retry behavior.

[osoblanco]

Thank you for this grader, and for the careful review it received. Its grading pipeline (reference preflight, candidate run, comparator) and its grading core carry over into the replacement, and this PR's commits keep their authorship there.

The replacement, #4629, takes that pipeline onto Gym's provider-neutral sandbox API, so the grader runs on any Gym sandbox provider without a backend of its own, and it addresses every finding of the review here, each with a regression test. Could a maintainer close this PR in its favour? Thanks again.