Repository navigation
fix(terminal-bench-science): raise sandbox disk limits and drop the S3 mount - #5334
Conversation
sandbox_min_disk_gib raises a role's ephemeral-storage limit to at least that many GiB while the declared storage_mb stays its request. Harbor's Docker environment enforces no storage limit, but OpenSandbox evicts the whole sandbox once it writes past its limit, with no ENOSPC first. With resource_requests: limits the request becomes the declared resources, so the node reserves only the declared disk. Roles without storage_mb stay unlimited. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Every sandbox gets nemo.nvidia.com/s3-hostpath: 'false', so the cell leaves out its shared read-only S3 bucket at /mnt/s3-data (data unrelated to the benchmark; agents searched it for answers, and whole-filesystem scans stalled in it). Disk limits rise to 100 GiB (++tbsci_min_disk_gib) while the declared storage_mb is reserved: 26 tries in seven full runs were evicted for writing past 2-12 GiB, which Docker allows. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
svcnvidia-nemo-ci
added
area:environment
|
/ok to test 4e7992d |
|
The two pending items are done, and the description is updated.
|
svcnvidia-nemo-ci
left a comment
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
One RISK: enabling disk headroom freezes CPU/memory requests before Compose overrides, unintentionally changing main-service reservations.
Reviewed all five changed files and their directly affected provider, Compose, and log-helper paths. No other qualifying findings. Static review only; tests and reported rollouts were not executed or independently reproduced.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| if resources.disk_gib is not None and resources.disk_gib < (config.sandbox_min_disk_gib or 0): | ||
| if options.get("resource_requests") == "limits": | ||
| # Reserve only the declared disk on the node; the higher limit is headroom. | ||
| options["resource_requests"] = {k: v for k, v in asdict(resources).items() if v is not None} |
There was a problem hiding this comment.
RISK — Preserve final Compose resource requests when adding disk headroom
Failure: With task resources cpus=2, memory_mb=4096, storage_mb=15000, floor 100, and resource_requests: limits, this branch freezes requests at {cpu: 2, memory_mib: 4096, disk_gib: 15}. A supported Compose main overlay of cpus: 4, mem_limit: 8g subsequently produces limits (4, 8192, 100) through the adapter's resource merge, without updating those requests. OpenSandbox's explicit-map branch only clamps requests downward, so it sends CPU/memory requests of 2/4096, not 4/8192. Before this change, limits remained deferred until after the Compose overlay. Affected main services are therefore under-reserved for CPU/memory merely because disk headroom was enabled.
Correction: Could the automatically derived requests be finalized after the main service's resource overlay? Coordinate Environment.start's service specification with AsyncSandboxCompose._prepare, using the normalized Compose resource contract. Reserve the effective CPU/memory/GPU values, but retain the original declared disk reservation. The invariant is that this option changes only disk limit versus request—not other resource_requests: limits semantics. The exact drop-in implementation across these layers remains uncertain; a blanket replacement of every request map would break explicitly configured requests.
Preservation and acceptance — proposed checks, not run:
- Add an Environment-to-Compose regression for the setup above: final limits must be
(4, 8192, 100)and requests(4, 8192, 15). Repeat with equivalentdeploy.resources.limitsvalues. - With a lower main override
(1, 2048), requests must remain(1, 2048, 15); preserve the existing downward-clamping behavior. - An explicit request map such as
{cpu: 1}must remain explicit and unchanged; omitted requests must remain omitted. Sidecars must continue deriving requests from their own effective resources, not the main role's map. - Retain the non-Compose disk-test assertions: raised limit
100with disk request15, immutable config options, no-storage verifier diskNone, declaration200unchanged, and unset floor retaining disk15. Also check a declaration equal to the floor remains unchanged.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41097568760 |
What changed
resources_servers/terminal_bench_4: a new environment option,sandbox_min_disk_gib(unset by default, which changes nothing). When it is set, a role that declaresstorage_mbgets anephemeral-storagelimit of at least that many GiB. Under the provider optionresource_requests: limits, the request becomes the declared resources, so the node still reserves only the declaredstorage_mband the higher limit is headroom. Roles that declare nostorage_mbkeep having no limit.benchmarks/terminal_bench_science: setssandbox_min_disk_gib: 100(override with++tbsci_min_disk_gib=) and adds the sandbox labelnemo.nvidia.com/s3-hostpath: 'false'beside the existingoom-killlabel. Both changes are new rows in the README's table of deviations from the packages.Why
Disk. Harbor's Docker backend, which the TB-Science leaderboard ran on, applies a task's CPUs and memory but not its
storage_mb. OpenSandbox enforces theephemeral-storagelimit Gym sends by evicting the whole sandbox, with no "no space left" error first. In a probe, a sandbox with a 1 GiB limit wrote 2 GiB without a write error and was evicted about 10 s after it passed 1 GiB. Its status then read Pending for 15 minutes, so the runner learns of it only from a failed command. In seven full TB-Science runs on OpenSandbox, 26 tries were lost this way, on 11 tasks declaring 2 to 12 GiB (microarch-modeling 6 times in 5 runs). The cell accepts a 100 GiB limit; a sandbox with no limit kept 7 GiB.S3 mount. By default the cell mounts a shared read-only S3 bucket at
/mnt/s3-datain every sandbox. Its data is unrelated to the benchmark and Docker sandboxes have no such mount, but agents searched it for answers and whole-filesystem scans (find /) stalled in it. The cell's per-sandbox labelnemo.nvidia.com/s3-hostpath: 'false'leaves it out, and/mntis then empty. A deployment that does not know the label ignores it, as withoom-kill.Issue
None. Found while running TB-Science on OpenSandbox; the evidence is below.
Validation
resources_servers/terminal_bench_4/tests: 264 passed, 1 skipped. New:test_min_disk_raises_the_limit_but_not_the_requestcovers the raised limit, the request staying at the declared resources underresource_requests: limits, explicit request mappings and no requests left unchanged, the provider config not mutated, roles withoutstorage_mbstaying unlimited, and larger declarations kept. The TB-Science profile test now checks the label on every role.pre-commit run --all-files: passed, no changes./mntwith the label./mnt.Rollout evidence
Codex 0.154.0 with self-hosted DeepSeek V4.1 Flash on OpenSandbox, on these two commits applied to an earlier
main. 13 tasks x 3 trials at the 8-hour budget, chosen because earlier runs repeatedly lost them to evictions (7 tasks) or searched/mnt/s3-datain them (6 tasks). All 39 trials were graded.storage_mb, so the previous limit would have evicted them, and all three were graded:/mnt/s3-dataand none received an S3 path in a tool output (0 of 39). On the same 13 tasks, an earlier run with the same model had 7 of 78 trials name the path and 21 receive S3 file contents or paths.find /returned in at most 5.4 s over 292 calls. Before, at least 10% of 478 calls ran past Codex's 30 s wait.SandboxUnresponsiveError), and its retry is the 21.3 GB trial; one trial ended on a gateway 502.Compatibility and benchmark impact
sandbox_min_disk_gibis unset by default, so other TB4 profiles are unchanged.resource_requests: limitsonlystorage_mbis reserved, so the higher limit does not change packing. A sandbox that writes past its declared size uses node disk beyond its reservation.