Repository navigation
ci: add harness conformance probes and a P0 check - #4192
Merged
Merged
Conversation
lafrancef
added
ci
Contributor
Author
|
/ok to test c1b6046 |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4192-34716a0d6924.docs.buildwithfern.com/nemo/gym |
lafrancef
changed the title
Contributor
Author
|
/ok to test d768d1f |
Contributor
Author
|
/ok to test 34716a0 |
Run affected registered harnesses in a non-blocking CPU matrix and publish warnings, summaries, and probe artifacts. Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
yaoyu-33
added
the
complexity:medium
yaoyu-33
reviewed
Oct 8, 2026
| env: | ||
| HARNESS: ${{ matrix.harness }} | ||
| run: | | ||
| if .venv/bin/python scripts/run_harness_conformance.py \ |
Contributor
There was a problem hiding this comment.
Non-blocking follow-up: is there a plan to cover the native EnvironmentServer/session path as well? The current probes use legacy_agent, so native setup, cancellation, and cleanup are not covered. Could a small smoke test exercise that path with a local provider, if feasible?
Contributor
Author
There was a problem hiding this comment.
In principle yes, although this sounds like it would belong in a harness integration test instead.
Contributor
Author
|
/ok to test bed9622 |
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Contributor
Author
|
/ok to test 90170d6 |
Contributor
|
/review role=automation |
svcnvidia-nemo-ci
approved these changes
Oct 8, 2026
svcnvidia-nemo-ci
left a comment
svcnvidia-nemo-ci
left a comment
Contributor
There was a problem hiding this comment.
SHIP — no material reliability issues found.
Reviewed all 12 changed files and traced CI selection, runtime setup, probe exit codes, current-attempt evidence handling, and check publication. The custom P0 check remains independent of the existing CI gate as intended.
Static review only; tests and live probes were not executed.
Review generated by codex
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37806862272 |
This was referenced Oct 8, 2026
yaoyu-33
added a commit
to yaoyu-33/Gym
that referenced
this pull request
Oct 9, 2026
…DIA-NeMo#4251) ## What does this PR do? Align conformance with NVIDIA-NeMo#4045: retained HTTP errors belong to an invocation and may have zero or one step reference. Successful policy responses still require a step; compaction calls remain separate. This corrects the P0 check introduced in NVIDIA-NeMo#4192 without inventing turns for failed HTTP exchanges. Also pass the configured assistant-message header to the scripted Model Server's capture middleware, matching normal Model Server startup. Tests cover header retention on HTTP success/failure and reject duplicate links or missing ownership. The OpenCode plugin-loading commit was removed and replaced by the separate shared-runtime PR NVIDIA-NeMo#4288. This PR only changes conformance checks, probe wiring, their tests and documentation. It is rebased on main `77509af61`. ## Validation After removing the plugin-loading commit: ```bash PYTHONPATH=$PWD python -m pytest \ tests/unit_tests/harness_capabilities \ scripts/harness_conformance/tests \ tests/unit_tests/test_harness_conformance_checks.py -q ``` **342 passed, 2 skipped.** Full core and all-server suites were not run locally. The all-files pre-commit check passed on the rebased branch before removing the isolated plugin-loading commit. The fresh four-harness probe run on the rebased branch, before that split, gave OpenCode 5/7, Hermes 7/7, Pi 6/7, Codex 0/7. These are not results for this final split: OpenCode additionally needs the separate plugin setup. Hermes benefits from NVIDIA-NeMo#4048 now on main. Pi's two provider-error classification unit failures remain separate (NVIDIA-NeMo#4285); its HTTP-400 health expectation also remains unresolved. Codex retains independent evidence gaps. No fresh generative-model benchmark is needed for this diagnostic/probe correction. No API migration or reward/scoring change. No new source files; existing licenses retained. All commits have DCO sign-off. --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
tanishku
pushed a commit
to tanishku/Gym
that referenced
this pull request
Oct 14, 2026
…A-NeMo#4500) Refresh the main harness conformance tables from the existing four-hour CI schedule. A separate job with read-only credentials renders one table artifact with the same renderer used by local regeneration; Fern downloads and inserts it without importing Gym or rerunning probes. Follow-up to NVIDIA-NeMo#4192; no separate issue is needed. The obsolete C-table, path qualifications, gap/aggregate guidance, and stale generated snapshot are intentionally removed. The page now presents measured checks with a short scope note and freshness footer. Rendering stays under `scripts/harness_conformance/`; the CI job produces only the rendered table. The five-minute P0 reporting job stays dependency-free with exclusive `checks: write`. Both reporting and docs jobs use the shared per-attempt downloader. Missing or invalid harness measurements display as Unavailable without discarding other results. Periodic publication uses the triggering run and attempt and accepts measured P0 failures. Ordinary docs pushes, tags, and manual publications warn and display unavailable measurements if an artifact cannot be obtained. The publishing helper comes from main so older release refs can still publish; release snapshots remain unchanged. Running publications are not cancelled. GitHub may supersede a pending publication with a newer event. Render a fresh local preview with `python -m scripts.harness_conformance.table` (diagnostic reports stay in its output directory), or insert a prepared CI table using `python -m scripts.harness_conformance.publish --run-id <main-ci-run-id> --strict` with authenticated `gh`. Validation: replayed the [existing CI evidence](https://github.com/NVIDIA-NeMo/Gym/actions/runs/38223655321) through rendering and insertion, checked the generated Fern page and internal links, and passed formatting and workflow validation. Fern's authenticated redirect check remains unavailable locally. Model rollout evidence: N/A; this changes reporting and docs publishing. --------- Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3766: run its existing P0 harness conformance suite in Gym CI and publish Harness conformance P0 directly in the PR checks list.
Approved PR mirrors and merge-queue revisions probe affected registered harnesses; shared code and dependency changes select all four. Probe jobs install pinned runtimes and retain summaries, logs, and evidence. A separate reporting job publishes a failed P0 check for conformance violations or unavailable measurements, and success when all selected harnesses pass. The check includes failed scenario counts, the first failing assertion or execution issue per harness, copy-pastable local rerun commands, and direct links to the evidence artifacts. After publication, the reporting job also fails if P0 fails or a probe job fails or is cancelled, making the outcome visible in the workflow job list. Only that reporting job receives permission to write checks.
The P0 check is optional until maintainers add its fixed name to the repository's required checks; the existing
Nemo_CICD_Testgate remains independent. Scheduled and manual CI runs exercise all harnesses. Documentation-table refresh is deferred.The probes were exercised on GitHub-hosted Linux in this CI run. Replaying that evidence through the new reporter produces 1 passed, 2 failed, 1 unavailable: Pi passes, Codex and OpenCode fail conformance, and Hermes's terminal-model-error scenario yields no rollout. The updated CI run published the P0 check with that same breakdown. The required CI aggregate passed; Hermes's probe job correctly reports the execution error separately.