feat(nooa): restrict SWE task networking and prepare offline runtime by init-nikhil · Pull Request #4088 · NVIDIA-NeMo/Gym

init-nikhil · 2026-10-06T08:01:53Z

Current stack revision — November 3, 2026

Head: d70ea7e9d3926556cf14997ef5a3446dfe67a2b2. Stack refreshed onto main 3880fa5024bf97199cba96ade2e0fa6ec9eb6fbf with current import locations and merge conflicts resolved. Each PR remains a separate layer; landing uses native stack merge.

#4088 now packages the launching Gym checkout using its unchanged root pyproject.toml and full core dependencies. The separate responses_api_agents/nooa_agent/runtime/pyproject.toml was removed. Tracked source and package assets are copied with their current contents; untracked files, Git metadata, and caches are excluded. This requires Git on the controller. Sandbox installation stays in its private Python, and source changes invalidate prepared offline wheelhouses. #4813 retains its workflow change and code-owner review requirement.

Validation: 87 focused runtime, offline-bundle, and task-network-policy tests passed after the refresh; all-files pre-commit passed (private guardword catalog unavailable). A real Gym wheel built successfully from the new archive format, and its NOOA entrypoint, resource dispatcher, and core package asset were verified. No model rollout or Linux installation was run for this packaging change. Earlier evidence below is historical and does not establish fresh results for this head.

GitHub dismissed earlier reviews after this update; current-head reviews and required CI are still needed.

Stack refresh — 2026-11-02

Current head: ce3f4dcc670265d076379588d021666515a0cef9. Stack refreshed onto main 2b8efc987af03951a80f941f2a278317badfe085. This PR remains a separate layer in the intended stack merge: #4088 → #4820 → #4821 → #4351 → #4357. #4813 is excluded.

Reconstructed each layer from its final reviewed patch instead of replaying foundation commits already merged into main. Resolved current package-import and legacy-environment renames while retaining upstream checkpoint handling and newer harness support. #4088 also exports its networking helpers through the current public API and mechanically updates existing callers; #4351 exports its newly used response types.

Fresh validation on the combined refreshed stack:

Full NOOA suite plus conformance runner/probe and harness-behavior tests: 913 passed, 10 Linux-only skips with pinned native NOOA 19caab169b018476ac433d040f6ae3f06aeff101.
Public-API facade suite and corrected GDPval recipe fixture: 372 passed.
Approved-prefix NOOA/network/recipe/metrics run: 671 passed, 10 skipped, and one GDPval fixture failure caused by the renamed legacy environment config; corrected and passed in the separate recipe run above.
pre-commit run --all-files: passed independently at all six PR layers, including package layering, guardwords, lint and format.
Main advanced during validation; its intervening paths were disjoint from this stack. The final rebase was conflict-free, with the validated stack files unchanged.

No fresh real-model benchmark was run for this conflict/import refresh. Prior live-rollout evidence below is historical. Fresh reviews and required GitHub CI remain outstanding; this is not a merge-readiness claim.

Earlier implementation and validation (before this refresh)

restrict swe task egress to the gym model host while preserving grader networking
prepare an offline runtime and account for context-window and output-token reserve
preserve the complete native benchmark task while reusing canonical benchmark implementations

Tests and validation

Head 3057bb5ac3c2825799e7a2a9bdec9aaf334ddbb3; base a7a6ba544f2592c1bb2712a765f48db5b8e8b168. The existing stack tracks this work; no separate issue is needed.

145 focused checks passed across offline packaging, runtime staging, native task adaptation, network configuration and token budgets. Independent concurrency/cancellation probes also passed. Production prepare_runtime_wheelhouse built a fresh source-bound bundle; production prepare_nooa_runtime installed, imported and reused it inside an owned Docker sandbox with network=none. Archive construction, validation and staging left the event loop responsive.

A separate fresh five-call model rollout exercised the actual adapter/worker, native tools, a 2,048-token output cap and 65,536-token context setting. The synthetic SWE task earned canonical verifier reward 1 and completed cleanup.

The offline build used a custom manylinux 2.28 amd64 builder; default manylinux2014/glibc 2.17 binary compatibility was not demonstrated. The live rollout used local provisioning and a preselected runtime. Current OpenSandbox provider egress enforcement was not requalified; older network evidence remains historical. Host-wide model egress is deliberate, TCP-port enforcement remains deferred pending provider support, and that review thread remains unresolved. Explicit Docker use requires the documented unrestricted policy configuration. This is integration evidence, not an official benchmark score.

pre-commit run --all-files passed. Fresh official Codex review returned no actionable findings; the independent bug hunt passed for this head and base.

CI snapshot (2026-10-24 03:47 UTC) on this head: failing: Harness conformance (non-blocking) / kilocode conformance probes; Harness conformance (non-blocking) / Publish check; Harness conformance (non-blocking). Current checks. The all-green readiness gate remains unmet; no final verification-success post or new review request is implied.

copy-pr-bot · 2026-10-06T08:01:57Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

copy-pr-bot · 2026-10-13T08:22:34Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

init-nikhil · 2026-10-21T02:25:28Z

/ok to test e3645b6

github-actions · 2026-10-21T02:28:22Z

🌿 Preview your docs: https://nvidia-preview-pr-4088-d70ea7e9d392.docs.buildwithfern.com/nemo/gym

init-nikhil

CODEX SELF REVIEW

Reviewed head e3645b69cb70e3ad6781cb9b972ca9771c5a26ce against actual stacked base 53196dcf6bdb8c923060b0c517612d24aaa53803 using official Codex CLI 0.154.0 (gpt-6-astra, fresh read-only process).

[P2] Preserve host ownership of Docker build artifacts — responses_api_agents/nooa_agent/offline_runtime.py:165-169. On a non-root Linux controller using ordinary rootful Docker, the builder creates root-owned files in its bind-mounted temporary directory. Cleanup can raise PermissionError after bundle publication, and failed builds can leave undeletable artifacts. Preserve controller ownership through success and failure cleanup.

This is the engine's static finding; independent reproduction and repair validation are tracked separately. Tests were not executed by the review process.

init-nikhil · 2026-10-21T08:19:09Z

/ok to test 22dedb4

init-nikhil

CODEX SELF REVIEW

Reviewed head 22dedb4ab7477df679e994ed314bd568c2abff2f against actual stacked base 53196dcf6bdb8c923060b0c517612d24aaa53803 with official Codex CLI 0.154.0 (gpt-6-astra, fresh read-only review process).

No actionable findings were returned.

The engine performed static source inspection; it did not execute tests. Model, benchmark and CI validation are tracked separately.

init-nikhil · 2026-10-21T08:53:57Z

/ok to test 125ce2d

init-nikhil

CODEX SELF REVIEW

Reviewed head 125ce2d7d9991f7d585a0ff4d8fd0bd2ddf6935d against actual stacked base 53196dcf6bdb8c923060b0c517612d24aaa53803 with official Codex CLI 0.154.0 (gpt-6-astra, fresh read-only review process).

No actionable findings were returned.

The engine performed static source inspection; it did not execute tests. Model, benchmark and CI validation are tracked separately.

svcnvidia-nemo-ci

SHIP — no actionable reliability findings.

Completed the light static review of all 32 changed files and the directly affected networking, runtime-installation, and context-planning paths. The historical Docker artifact-ownership issue is addressed by the owner probe and user-scoped build in offline_runtime.py.

Tests, offline-container installation, and live OpenSandbox network enforcement were not executed during this review.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

furgalep

I reviewed this by reading the code and tracing callers; nothing was run. The two I'd look at first are the event-loop blocking in prepare_nooa_runtime and the breadth of the egress allowlist.

furgalep · 2026-10-22T07:32:48Z

+            model_url = sandbox_server_url(self.config.task_network_policy.model_server.name, require_reachable=True)
+            # Replace rather than extend any baseline allowlist. Grading keeps the original options.
+            provider_options = dict(provider_options or {})
+            provider_options["network_policy"] = restricted_network_policy(next(iter(provider_config)), [model_url])


The allowlist is the model server's whole host, on all ports. In the recipe, policy_model is a Gym server on the controller, and that host also runs the head server, the SWE Resources server and the agent server. Task code could reach all of them, including the head server, whose config may hold OPENSANDBOX_API_KEY / POLICY_API_KEY.

The docs acknowledge this, but nothing enforces it. Could the policy be limited to the model server's port, or could the model endpoint be exposed through a dedicated proxy?

the open sandbox version we use right now does not support this, but there is a release candidate being tested. this version of open sandbox will provide support for specific ports.

init-nikhil

CODEX SELF REVIEW

Reviewed head 3057bb5ac3c2825799e7a2a9bdec9aaf334ddbb3 against actual base a7a6ba544f2592c1bb2712a765f48db5b8e8b168 using a fresh official codex exec review process.

No actionable regressions were identified. All 83 focused model-adapter, network-policy, and observability tests passed; live Docker builds and OpenSandbox enforcement were not validated.

This is a source-review result, not an approving review or a substitute for the separate validation and CI gates.

furgalep

LGTM

init-nikhil · 2026-10-28T02:40:39Z

/ok to test 1a2491c

init-nikhil · 2026-10-29T07:09:07Z

/ok to test 09862c8

furgalep

LGTM

furgalep

LGTM

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

init-nikhil · 2026-11-04T02:48:21Z

/ok to test d70ea7e

github-actions · 2026-11-04T03:19:42Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41766407359

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch from 1aff24a to c2de575 Compare October 8, 2026 02:23

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch from c2de575 to f8c3c5b Compare October 8, 2026 21:01

init-nikhil mentioned this pull request Oct 8, 2026

feat(environment): [wip] retain agent evidence after session cleanup #3847

Closed

drazvan mentioned this pull request Oct 9, 2026

fix(nooa): retain task services through grading and preserve terminal failures #4291

Closed

4 of 5 tasks

init-nikhil mentioned this pull request Oct 9, 2026

fix(nooa): preserve complete assistant history across persistence #4351

Merged

init-nikhil added this pull request to stack #4352 October 9, 2026 20:58

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch from f8c3c5b to 070c74a Compare October 9, 2026 21:58

This was referenced Oct 9, 2026

feat(nooa): use standard single-agent-turn sessions #3846

Merged

feat(nooa): stage a portable sandbox runtime #3844

Merged

feat(nooa): wire sandbox execution into agent sessions #3849

Merged

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch 3 times, most recently from eeaf374 to db166b3 Compare October 13, 2026 08:22

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch from 49a8ff0 to f8fbeec Compare October 20, 2026 02:10

init-nikhil force-pushed the codex/nooa-eval-reproducibility branch 2 times, most recently from 10a2a19 to 4c56c44 Compare October 20, 2026 07:30

init-nikhil changed the title ~~feat(nooa): add reproducible three-benchmark evaluation recipes~~ feat(nooa): restrict SWE task networking and prepare offline runtime Oct 21, 2026

This was referenced Oct 21, 2026

feat(nooa): add a canonical Terminal Bench recipe #4820

Merged

feat(nooa): add a canonical GDPval recipe #4821

Merged

init-nikhil commented Oct 21, 2026

View reviewed changes

svcnvidia-nemo-ci reviewed Oct 21, 2026

View reviewed changes

furgalep reviewed Oct 22, 2026

View reviewed changes

Comment thread benchmarks/swebench/pro/nooa.yaml Resolved

furgalep reviewed Oct 22, 2026

View reviewed changes

init-nikhil commented Oct 24, 2026

View reviewed changes

furgalep previously approved these changes Oct 25, 2026

View reviewed changes

furgalep previously approved these changes Oct 29, 2026

View reviewed changes

init-nikhil mentioned this pull request Nov 2, 2026

fix(nooa): preserve native evidence and add conformance probes #4813

Merged

furgalep previously approved these changes Nov 3, 2026

View reviewed changes

feat(nooa): restrict SWE task networking and prepare offline runtime

d70ea7e

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

adil-a approved these changes Nov 4, 2026

View reviewed changes

Conversation

init-nikhil commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current stack revision — November 3, 2026

Stack refresh — 2026-11-02

Earlier implementation and validation (before this refresh)

Tests and validation

Uh oh!

copy-pr-bot Bot commented Oct 6, 2026

Uh oh!

copy-pr-bot Bot commented Oct 13, 2026

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 21, 2026

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

svcnvidia-nemo-ci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

furgalep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

furgalep Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

init-nikhil Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

init-nikhil left a comment

Choose a reason for hiding this comment

Uh oh!

furgalep left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Oct 28, 2026

Uh oh!

init-nikhil commented Oct 29, 2026

Uh oh!

furgalep left a comment

Choose a reason for hiding this comment

Uh oh!

furgalep left a comment

Choose a reason for hiding this comment

Uh oh!

init-nikhil commented Nov 4, 2026

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

init-nikhil commented Oct 6, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading