Repository navigation
Tb e2e - #1939
Open
jialeiwang wants to merge 17 commits into
Open
Tb e2e#1939jialeiwang wants to merge 17 commits into
jialeiwang wants to merge 17 commits into
Conversation
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This reverts commit c34d56c.
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Port the validated oracle-run config into the gym eval/agent path (swe_agents/app.py), all gated on dataset_name=='terminal-bench': - NETWORK (new): allow_internet=false tasks run with '--net --network=none' (private loopback netns) so services bind privileged ports (nginx :80) + parallel workers don't collide; allow_internet=true keeps shared host netns. Default True => unchanged for existing rollouts. (CNI/slirp blocked on these nodes; only --network=none permitted.) - /etc/hosts: add $(hostname) so sudo/tools resolve the host under --network=none. - service boot: for offline service tasks, run the image runscript (Docker CMD) in the background before the agent so boot services (postgres/nginx/...) are up (real-TB parity). - (already present from Path-2 work: persistent overlay, single-exec for daemon persistence, TEST_DIR/T_BENCH_TEST_DIR, apt _apt-sandbox=root, systemctl->service shim, /var/run/postgresql EOVERFLOW pre-create.) vllm_model/app.py: normalize multi-part tool-result content so opencode rollouts don't 500. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…twork fix Full arm64 SIF inventory (tb_sifs_arm64 + ots sifs_arm64 + image-conversion-HSG), tb_single_exec=true (daemon persistence). Use with build_rollout.py output (rows carry allow_internet from oracle_index -> gym branches --net --network=none per task). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…del conn) The 946-task re-run gave 0% solved: --network=none isolates the netns so the opencode agent cannot reach the policy model at host localhost:8000 (connection refused). The service-mode network fix is valid ONLY for the standalone oracle check (no model), not the agent rollout. Disabled both for the gym (clear comments + a path: socat-bridge the model into a private netns to re-enable). Kept the harmless /etc/hosts $(hostname) + the pre-existing Path-2 fixes. The old shared-netns rollout data stands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rint) The --network=none service-mode fix does not apply to the agent rollout (agent needs the host model; isolated netns breaks it). Revert ALL of my edits to the pre-change state: /etc/hosts $(hostname) (only needed under --network=none), the net_flag/--network=none branch, and the service-boot. Remove terminalbench_netfix.yaml. nemo-gym is back to its prior working state; the finding is recorded in the logbook + memory, not the gym code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…path setup Adversarially audited (13-agent hunk audit + 3-refuter re-verify) and remediated so every terminal-bench change is dataset- or config-gated and non-TB (SWE) runs behave identically to fork point f067661: - setup_scripts/opencode.sh: dual path. The hardened supply-chain path (pinned commit archive + sha256, ghostty pin, run_infer BENCH_CLI bundle patch, frozen-only installs, setup receipt) runs ONLY when (repo, commit) equals the preauthorized pin 0c088fd1; any other config takes the original git-clone path unchanged and is never hard-failed. Smoke-verified both paths: stock sdd/dev -> clone path, cli.ts launcher untouched, no receipt; pinned commit -> tarball path, receipt written, runtime-matched, authorized. - OpenCodeHarnessProcessor.setup(): receipt verification is opt-in via new opencode_setup_receipt_enforce (default False = fork-point existence-checked setup, no receipt code reachable). Enforced setups build into swe_opencode_setup_preauthorized so receipt-invalidation rmtree can never poison the shared SWE cache. Dropped the NEMO_GYM_OPENCODE_SETUP_DIR env override (config-via-YAML convention). - Receipt authorization pins build INPUTS (source tree, bun binary, repo/commit) plus build-tree shape, not bundle bytes: bun build output is not byte-reproducible across environments (confirmed empirically: identical inputs produced an opencode.js differing by ~850 bytes). Receipt-to-disk consistency is still enforced by runtime matching. - _inner_responses: trajectory repair (_append_terminal_tool_outputs + synthetic_terminal_tool_outputs metric) is now terminal-bench-gated, and the new TB-gated _tb_flatten_multipart_content flattens opencode's list-form tool-result notices (image/PDF placeholders for text-only models) so the Responses conversion cannot 500 whole collection shards. NOTE: the flatten is correct for text-only policies; reconsider the implementation path (end-to-end multipart image support) before serving vision policies. - vllm_model/app.py: reverted to fork point (the earlier shared-converter normalizer is superseded by the TB-gated guard). - configs: terminalbench{,_netbridge,_all}.yaml pin agent_framework_commit 0c088fd1 and set opencode_setup_receipt_enforce: true; superseded terminalbench_{multiset,opencode,tb21}.yaml removed; the preauthorized receipt reference opencode_setup_receipt_expected.json is committed. - tests: terminal-bench isolation suite covering hidden-test isolation, receipt authorization semantics, dual-path setup contract, dataset gating, and flatten semantics; tomlkit importorskip keeps clean-env core-suite collection green. Full unit suite: 222 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Test User <test@example.com>
…fixes all-zero rollouts) HSG smoke job 4127557 failed 24/24 rollouts in ~29s each with 'Module not found /opencode_setup/opencode/index.ts': the hardened run_infer.sh patch launched the BUNDLED bench-cli.js, inside which detectOpencodeBin()'s import.meta.url root walk (4 levels up from packages/opencode/src/bench) overshoots from .bench-build/ to /, misses the opencode.js bundle, and falls back to a nonexistent <root>/index.ts. The 46k-rollout campaigns launched bench/cli.ts from source, where the walk lands correctly — the bundle-launch path had never been validated in-container (flagged as audit finding M2). Fix: drop the run_infer.sh BENCH_CLI patch entirely; the launcher runs bench/cli.ts from the receipt-pinned source tree on every path (campaign-proven, and removes the SWE-vs-TB launcher divergence). bench-cli.js is still bundled (receipt build-tree shape) but unused at runtime. Regenerated opencode_setup_receipt_expected.json + H8_OPENCODE_SETUP_RECEIPT_SHA256 from a fresh hardened build (source tree now carries the unpatched run_infer.sh); isolation test asserts the patch is absent. Existing preauthorized caches fail authorization against the new reference and rebuild automatically. 222 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Test User <test@example.com>
github-actions
Bot
added
the
sla:triage-overdue
Contributor
|
there is multi harness tb support in anyterminal_agent if youd like to try |
This was referenced Sep 8, 2026
svcnvidia-nemo-ci
added
area:agent
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.