fix(nooa): preserve native evidence and add conformance probes by init-nikhil · Pull Request #4813 · NVIDIA-NeMo/Gym

init-nikhil · 2026-10-20T23:00:21Z

NOOA now retains native tool identities, authoritative history, model-visible feedback, and terminal provider outcomes through its sandboxed worker. Shared harness conformance exercises its production borrowed-sandbox path and saves independent execution evidence.

Terminal model rejections retain their numeric HTTP status and follow the existing default/fail/retry policy. Worker, transport, resources, and cleanup failures remain ungradable. Shared grading policy is unchanged.

The conformance workflow runs NOOA in a disposable Linux Python container. Runtime installation follows main's locked uv sync setup. Runtime provenance hashes pyproject.toml and uv.lock; it no longer reads the removed requirements.txt that previously prevented all NOOA scenarios from starting.

Validation on final head 13f0f9f2d9563b1c07eee683877cae60f061a912:

Required DCO, Lint check, and Nemo_CICD_Test all passed. The complete CICD workflow succeeded, including all core, sandbox, server, lock, and conformance jobs.
Linux NOOA conformance met the configured P0 gate across all nine scenarios. Eight scenarios fulfilled their complete verdict; the higher-tier context-compaction verdict remains not fulfilled. The uploaded harness-conformance-nooa-42118673574-1 artifact contains the report, saved model calls, and independent witnesses. This is not a claim of complete P1 conformance.
457 focused NOOA, conformance fixture/runner, and behavior tests passed locally after refreshing onto main fd1fa273, including the dependency-fingerprint regression and current session lifecycle naming.
Scoped pre-commit passed; the private guardword catalog was unavailable. The tree is clean and diff checks passed.

Main supplied the OpenClaw integration-test cleanup-budget fix that addressed the repeated CI timing failure; the duplicate local timing change was dropped. One added NOOA test assertion follows main's renamed fenced-session collection. No production timeout was changed by this PR.

The five lower stack PRs have merged, and this PR is based on main. The workflow change is unchanged from the version previously reviewed by automation. Fresh current-head code-owner approval remains the landing gate. Earlier benchmark evidence is historical and is not a readiness claim for this head.

copy-pr-bot · 2026-10-20T23:00:25Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

init-nikhil

CODEX SELF REVIEW

Reviewed head: 441e779bc57f5c49558b8edd1378ffbfd0c92e79
Actual PR base: 634921ea21d8221c6e62461193841cccc37d1e7d

The official Codex review engine (codex-cli 0.154.0, fresh codex exec review --base process, read-only sandbox, inherited account/model configuration, GYM_DEV_REVIEW_CHILD=1) returned no actionable findings.

The engine ran 32 focused tests successfully: 26 in tests/unit_tests/harness_capabilities/test_behavior.py and 6 in scripts/harness_conformance/tests/test_ci_install.py. Full NOOA runtime validation was unavailable because the pinned dependencies were not installed in the read-only review environment. This result does not guarantee correctness or complete the remaining workflow validation.

Run evidence: gymcheck-nooa-conformance-20261020T225915Z; engine exit status 0; completed 2026-10-21 00:22:35 UTC. The checkout remained clean.

Final-output SHA-256: b87a7ff402fda7d3c88c3efb4250a587496cd95dcbdb63156f1048427f8a37ea
Raw engine-output SHA-256: 5dde3cb05bd04c025329c63ff4d496729a979858ad39c93c0fe9540d4839d247

init-nikhil · 2026-10-21T00:44:11Z

/ok to test 4e9dd08

init-nikhil · 2026-10-21T01:40:08Z

/ok to test f627fdb

init-nikhil

CODEX SELF REVIEW

Reviewed head: f627fdbfce09d85a91c55378c4b45ad8157bd56c
Actual PR base: 634921ea21d8221c6e62461193841cccc37d1e7d

The official Codex review engine (codex-cli 0.154.0, fresh codex exec review --base process, read-only sandbox, inherited account/model configuration, GYM_DEV_REVIEW_CHILD=1) returned no actionable findings for the full 21-file PR.

All 32 focused behavior and installation tests passed. Full NOOA integration validation was unavailable because the installed NOOA version lacks required APIs. This result does not guarantee correctness or complete the remaining workflow validation.

Run evidence: gymcheck-nooa-conformance-20261020T225915Z, reviewer-repair2; engine exit status 0; completed 2026-10-21 01:45:38 UTC. The checkout remained clean.

Final-output SHA-256: 767587babe54e152320cc47479761d90169dc990ce7993d5bf31db1abce0bb37
Raw engine-output SHA-256: 4720ebb8e0b7f3125bfcb2fd143f70e1b08adbc7a82045bf15795807fe64f856

init-nikhil · 2026-10-21T04:24:47Z

/ok to test 2b7f9b4

init-nikhil

CODEX SELF REVIEW

Reviewed head 2b7f9b446f6679df2ce97379511548965a625bf8 against actual stacked base 4c632f32c37e52485dfc002a985823ed144b16fe with the official Codex CLI 0.154.0 (gpt-6-astra, fresh read-only review process). No actionable findings were returned.

The engine performed static source inspection; it did not execute tests. Local, live-model, and CI validation are tracked separately. This is not a readiness or correctness guarantee.

init-nikhil · 2026-10-21T08:19:18Z

/ok to test 46d9db6

init-nikhil

CODEX SELF REVIEW

Reviewed head 46d9db6a3049710394741593c87013751ebd839e against actual stacked base c5e26e893cec11c6e9709832e1e6b52cc2533585 with official Codex CLI 0.154.0 (gpt-6-astra, fresh read-only review process).

[P2] Preserve previously captured Python feedback across rerenders — responses_api_agents/nooa_agent/observability.py:271-273. A differently truncated rerender can replace previously captured feedback even when the next model call is stopped by the policy-call budget. Projection can then discard that unsent replacement and lose feedback that reached an earlier request. Independent reproduction is in progress.

The engine performed static source inspection; it did not execute tests. Model, benchmark and CI validation are tracked separately.

init-nikhil · 2026-10-21T08:54:07Z

/ok to test 5f3e148

init-nikhil · 2026-10-29T21:56:23Z

/ok to test eba0827

furgalep

LGTM

init-nikhil · 2026-11-04T02:48:30Z

/ok to test d8b273e

init-nikhil · 2026-11-04T03:24:36Z

@NVIDIA-NeMo/automation can someone review this? need codeowner approval for the workflow change that runs nooa conformance in a container.

init-nikhil · 2026-11-05T08:13:22Z

/ok to test bca323f

init-nikhil · 2026-11-05T21:39:26Z

/ok to test e69e63a

init-nikhil · 2026-11-05T22:02:51Z

/ok to test 1975665

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

init-nikhil · 2026-11-05T22:06:50Z

/ok to test 13f0f9f

github-actions · 2026-11-06T00:10:53Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42134907028

init-nikhil added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior complexity:medium Single-domain change with interacting parts or a moderate review surface labels Oct 20, 2026

init-nikhil commented Oct 21, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public October 21, 2026 00:44 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 00:46 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 00:47 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 01:40 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 01:42 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 01:43 Inactive

init-nikhil commented Oct 21, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public October 21, 2026 04:25 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 04:27 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 04:28 Inactive

init-nikhil added this pull request to stack #4825 October 21, 2026 04:33

init-nikhil commented Oct 21, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public October 21, 2026 08:19 Inactive

copy-pr-bot Bot temporarily deployed to public October 21, 2026 08:22 Inactive

init-nikhil commented Oct 21, 2026

View reviewed changes

init-nikhil dismissed furgalep’s stale review via eba0827 October 29, 2026 21:54

copy-pr-bot Bot temporarily deployed to public October 29, 2026 21:56 Inactive

copy-pr-bot Bot temporarily deployed to public October 29, 2026 21:58 Inactive

copy-pr-bot Bot temporarily deployed to public October 29, 2026 21:59 Inactive

init-nikhil force-pushed the codex/nooa-conformance branch from eba0827 to 5efa36d Compare November 2, 2026 20:32

init-nikhil force-pushed the codex/nooa-conformance branch from 5efa36d to 231ff41 Compare November 2, 2026 20:39

furgalep previously approved these changes Nov 3, 2026

View reviewed changes

init-nikhil dismissed furgalep’s stale review via d8b273e November 4, 2026 02:16

init-nikhil force-pushed the codex/nooa-conformance branch from 231ff41 to d8b273e Compare November 4, 2026 02:16

copy-pr-bot Bot temporarily deployed to public November 4, 2026 02:49 Inactive

copy-pr-bot Bot temporarily deployed to public November 4, 2026 02:51 Inactive

copy-pr-bot Bot temporarily deployed to public November 4, 2026 02:52 Inactive

kajalj22 previously approved these changes Nov 5, 2026

View reviewed changes

init-nikhil added 3 commits November 5, 2026 14:04

fix(nooa): preserve native evidence and add conformance probes

6b3314d

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

fix(conformance): fingerprint NOOA locked dependencies

2cf547e

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

test(nooa): follow main session lifecycle naming

13f0f9f

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

kajalj22 approved these changes Nov 5, 2026

View reviewed changes

Conversation

init-nikhil commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil commented Oct 29, 2026

Uh oh!

furgalep left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Nov 4, 2026

Uh oh!

init-nikhil commented Nov 4, 2026

Uh oh!

init-nikhil commented Nov 5, 2026

Uh oh!

init-nikhil commented Nov 5, 2026

Uh oh!

init-nikhil commented Nov 5, 2026

Uh oh!

init-nikhil commented Nov 5, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

init-nikhil commented Oct 20, 2026 •

edited

Loading