[mohsenhariri]

What does this PR do?

Add gym eval profile --bayes-at-n for complete evaluations with independent repeated generations and a binary correctness verifier. Each run's saved agent metrics gain bayes@N/reward and std/bayes@N/reward, with N replaced by its observed trial count. Both use reward units on [0, 1]. The second is posterior standard deviation, separate from empirical reward variance and across-repeat standard errors.

The option checks the full materialized-input manifest before masking or partial-data filtering. It rejects missing or duplicate rollout keys, masked or failed rows, missing/nonbinary rewards, unequal trial counts and changed question inputs. Rollout indices, _ng_rollout_id and per-rollout RNG seeds may vary; supplied seeds must be distinct within a question/run. Fan-out and Environment Server runs stay separate. Default profiling output is unchanged.

This is the binary uniform-prior subset of Bayes@N, using independent Beta(s+1, N-s+1) posteriors per question. It averages question means and combines question variances before taking the corpus standard deviation. The formula was implemented independently; the MIT-licensed Scorio reference at 7278f67 is used for parity checks. No runtime dependency is added.

gym eval profile \
    --inputs results/rollouts_materialized_inputs.jsonl \
    --rollouts results/rollouts.jsonl \
    --bayes-at-n

The branch includes upstream 0c7ac5119c99e0aa0dfeda7be59157c589680ffd, including the metrics package move in #5211 and saved-run recovery in #3302. The Bayes check still precedes measured-row selection, and ordinary coverage reporting is preserved. Collection now stamps a distinct _ng_rollout_id on repeated copies; the input comparison allows that runtime field while retaining prompt/configuration checks. Regressions use RolloutCollectionHelper.preprocess_examples through posterior calculation and profile CLI export.

Closes #4810.

Validation

Current merged tree 293182126b13599f7f648169e1854fe05a89e1c6, CPython 3.13.14, repository-local environment synced with .tools/uv-env/bin/uv sync --locked --extra dev --extra wandb:

• OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 .venv/bin/python -m pytest tests/unit_tests/metrics/test_reward_profile.py tests/unit_tests/test_aggregate_metrics.py tests/unit_tests/test_cli_main.py tests/unit_tests/test_cli_eval.py --cpu -q: 483 passed. This includes 69 Bayes cases covering exact moments, complete cohorts, collection-generated rollout IDs, seed policies and actual CLI JSON export. The earlier base's CLI logging failure is absent on this tree.
• .venv/bin/python -m pytest tests/unit_tests/test_public_api.py tests/unit_tests/test_public_api_facades.py tests/unit_tests/test_import_linter_coverage.py tests/unit_tests/test_fern_docs_links.py --cpu -q: 423 passed and 24 subtests passed.
• 24 seeded workloads match independent Fraction calculations and the pinned Scorio implementation, including 164 questions by 64 trials, with rtol=1e-14 and atol=1e-15.
• PRE_COMMIT_HOME=.cache/pre-commit .venv/bin/pre-commit run --all-files: passed, including package import layering. The private guardwords catalog was unavailable, so its catalog check was skipped.
• npm --prefix fern run check: zero errors, one warning that the authenticated missing-redirect check was skipped. This change adds no page or redirect.
• git diff --check passed. Applying the saved complete patch to an index of the pinned upstream base reconstructs the exact tested tree.

A separate agent reviewed the method and public path. It found that collection-generated _ng_rollout_id values were rejected as changed inputs; this is fixed. Both collection regressions fail before the repair and pass after it. Final independent review passed 40 exact Fraction workloads and five CLI scenarios, covering complete, masked, omitted, missing-question and ragged inputs. A further CLI run with collection-generated rollout IDs and seed insertion confirmed posterior export and default-output preservation.

The full core/sandbox coverage and eight-shard component suite were not run locally. Trusted NVIDIA CI still requires a qualified vetter's approval of the new full head 1bf41a40251a75ea1ac85f25157722d8b4ade02f.

Rollouts

N/A: this change analyzes saved rollout artifacts. It changes neither generation, agent behavior, environment state nor verification. CPU fixtures exercise collection preprocessing and the actual profile command through exported metrics without model inference.

Compatibility and benchmark impact

The option is disabled by default. It requires the complete materialized-input file from one fixed evaluation and the same positive N per question within each run. The binary verifier contract, fresh independent generations and fixed external model/agent/verifier/environment remain caller obligations; saved files cannot certify them. Continuous graders, including GraphWalks F1, are unsupported even if one observed file contains only 0 and 1. Continuous-reward thresholds, categorical rubrics, historical observations and credible intervals are outside this binary option.

I am a coauthor of Bayes@N and maintain Scorio. The source investigation, implementation, tests and text were prepared with Codex.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated edits are outside this contribution.
• Tests added or updated and pass locally; unrun broader suites are disclosed above.
• Documentation added or updated.
• Pre-commit checks pass locally; the private guardwords limitation is disclosed above.
• No new source file requires an additional Apache-2.0 SPDX header.
• All contribution commits have DCO sign-off, explicitly authorized by the human author.

Proposed labels: feature, area:evaluation. Maintainer CI and review remain pending.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mohsenhariri]

DCO passes, but the lint/copyright/secrets runs require approval and NVIDIA trusted-runner vetting is still pending. Could a qualified NVIDIA-NeMo vetter review this head for CI, and an evaluation-area maintainer review the optional reward-profile summaries?

The current head is 83804fdf4104b33424965ad82d135fab2da5e7ae. The PR body records the profile/CLI validation and the logging failure reproduced on the unchanged base.

[mohsenhariri]

Updated this branch through upstream 0c7ac5119c99e0aa0dfeda7be59157c589680ffd at 1bf41a40251a75ea1ac85f25157722d8b4ade02f.

The update follows the metrics package move and preserves complete-manifest validation before measured-row filtering and coverage reporting. Independent review found that collection now gives repeated copies distinct _ng_rollout_id values; the Bayes input comparison now permits that runtime field. New regressions use collection preprocessing through posterior calculation and the profile CLI, while changed prompts/configuration, masked results and incomplete cohorts remain rejected.

On the final tree, 483 affected tests and 423 interface/import/docs-link tests passed, with 24 additional subtests. All-files pre-commit, Fern configuration and 24 exact/reference workloads passed. The private guardwords catalog and authenticated Fern redirect check were unavailable; the full core/sandbox and component suites were not run locally. No model rollout is needed for this saved-artifact analysis.

The PR body has the current commands and independent review findings. A qualified NVIDIA vetter still needs to approve the new full head 1bf41a40251a75ea1ac85f25157722d8b4ade02f before trusted CI can run.