Repository navigation
[GDPVal] Allow partial_completion on the final stage - #3770
Merged
Kh4L merged 1 commit intoSep 29, 2026
Merged
Conversation
The final stage had no coverage floor: a run either reported exactly N/N rows or was flagged final_stage_degraded, with nothing in between. Three 0.16.0 runs landed 215-217/220 and were blocked from certification. partial_completion may now be set on the final stage. When the policy accepts it, the orchestrator stamps accepted_stage_row_count on the kept rows, and aggregation reports final_stage_partial_accepted=1, final_stage_degraded=0, and emits the flat headline. Below the floor the stage is still scored but degraded, as before. final_stage_complete stays exact. Refs #3341 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
Kh4L
approved these changes
Sep 29, 2026
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 2, 2026
…o#3912) ## Summary Ports NVIDIA-NeMo#3770 from `gdpval-devel` to main. NVIDIA-NeMo#3687 made `partial_completion` unavailable on the final stage; this lets the final stage use it too. - A final stage the policy accepts reports `comparison/final_stage_partial_accepted: 1` and `comparison/final_stage_degraded: 0`, and emits the flat `comparison/eval_elo` / `comparison/normalized_elo`, even when the observed row count is below the expected count. `comparison/final_stage_complete` stays exact. - A final stage below the floor is still scored, but reports `final_stage_degraded: 1` and no flat headline. - The accepted row count is recorded per row (`accepted_stage_row_count`) and in the stage journal, so resume cannot change which rows count. - README documents the final-stage policy; the `partial_completion` entry in the config table is updated to match. Two files conflicted with main (the README `stages` row and a test import); both are resolved to keep main's text plus the new behavior. ## Test plan - `resources_servers/gdpval/tests` + `tests/unit_tests/test_rollout_collection.py` + `test_dispatch_latency.py`: 908 passed. - `ruff check` and `ruff format --check` clean. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Alexey Gronskiy <agronskiy@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
partial_completionwas rejected on the final stage, and aggregation treated anything short of exactly N/N final-stage rows asfinal_stage_degraded=1. So the final stage had no coverage floor at all. Three super3.5@0.16.0 GDPVal-AA v2 runs landed at 215, 217 and 216 out of 220 and were markedcannot_certify.This PR allows
partial_completionon the final stage, using the same policy fields as the calibration stage:partial_completeoutcome and stampsaccepted_stage_row_counton every row it keeps, in both the fresh path and the cached-resume path. If the policy rejects it, the stage behaves as before: it is scored and can finish, but is degraded.app.py: addscomparison/final_stage_partial_accepted. The flag is 1 when every final-stage row carries the sameaccepted_stage_row_countand that count equals the observed row count. In that casefinal_stage_degraded=0and the flatcomparison/eval_elo/normalized_eloare emitted.final_stage_completestill means exactly N/N.Example (95% = 209/220):
Refs #3341. This covers defect (b), the final stage's missing coverage floor. Defect (a),
timeout_exceededconsuming the rollout attempt budget, is not addressed here.Tests
pytest resources_servers/gdpval/tests/test_multistage_orchestrator.py: 140 passed. The two tests asserting that a final-stage policy is rejected are replaced with tests for acceptance and stamping, a cached-resume replay with no re-dispatch, and a below-floor run that stays unstamped.pytest resources_servers/gdpval/tests/test_app.py: 100 passed. New parametrized cases cover an accepted count, a missing count, and a count that doesn't match the observed rows.test_comparison_file_handling.py::test_oversize_xlsx_does_not_attempt_structured_extractionalready fails ongdpval-develbefore this change and is unrelated.🤖 Generated with Claude Code