feat(sandbox): lifecycle telemetry: durations, retries, sandbox ids and a stop span by iguehring-nvidia · Pull Request #3633 · NVIDIA-NeMo/Gym

iguehring-nvidia · 2026-09-22T22:02:05Z

Supersedes #3581 (its two commits are the first two here) and takes the sandbox part of #2935, with @oyilmaz-nvidia as co-author on the hunks from there. Agreed with Onur as the split of #2935 that this side owns.

What

One coherent "sandbox lifecycle telemetry" set in AsyncSandbox and the providers, all under the existing sandbox span group, all attributed by nemo.gym.sandbox.provider:

Instrument	Kind	Where	Origin
`gym.sandbox.active`	up-down counter: +1 when a provider hands back a handle, −1 when Gym releases it	facade `start()` / `stop()`	this PR (was #3581)
`gym.sandbox.startup_duration_ms`	histogram	facade `start()`	#2935
`gym.sandbox.exec_duration_ms`	histogram	facade `exec()`	#2935
`gym.sandbox.create_retry_total`	counter, one per retried create attempt	OpenSandbox and Daytona retry callbacks	#2935 (Daytona), new (OpenSandbox)
`nemo.gym.sandbox.id`	span attribute on start / exec / stop	facade	#2935
`gym.sandbox.stop`	span around `provider.close`	facade `stop()`	new

Supporting change: nemo_gym/telemetry/gym_metrics.py, a small registry for attributed instruments (per-meter cache, histogram / counter / up-down-counter recorders, no-op without telemetry, never raises), taken from #2935. record_sandbox_active lives there; telemetry/metrics.py keeps forwarding to nemo-lens's five fixed instruments and points here for anything attributed.

Why

Gym's sandbox API emitted gym.sandbox.start and gym.sandbox.exec spans and nothing else: no stop span, no metric. Three questions a dashboard could not answer: how many sandboxes exist right now (dashboards approximated it from in-flight /run requests, which undercounts by more than two times on Terminal-Bench 4, where a session holds an agent sandbox, a shared-logs helper and later a verifier sandbox); how long provisioning and commands take as a metric rather than only as spans, which need a span-metrics connector somewhere to become one; and how often creation has to be retried. On a degraded provider the last one is the difference between "39 rollouts dropped" and "101 retries, then 39 dropped".

Design points:

Up-down counter, not a gauge, for gym.sandbox.active: several server processes each hold a share of the sandboxes; a backend sums counters correctly across processes, while a gauge shows whichever process wrote last. −1 happens once the provider has actually released the sandbox: a close() the provider rejected has released nothing, so the sandbox stays started and counted and a later stop() retries it (the compose group depends on that retry). A connect()ed sandbox is neither counted nor subtracted, since the starting process counted it.
Explicit bucket boundaries up to thirty minutes on both duration histograms. The SDK default ends at 10 s; a remote sandbox start takes about a minute, so every sample would otherwise land in the overflow bucket and a p95 would read "10 s" whatever the truth. (feat(telemetry): full correlation chain, saturation, concurrency, and Pareto-efficiency metrics #2935 used the defaults; fixed while consolidating.)
Retry counting sits in the providers because retrying is provider-internal: the facade calls create() once. The two providers that retry (OpenSandbox, Daytona) record from their tenacity before_sleep callback, next to the warning they already log, so the counter can never disagree with the log.
The stop span wraps only provider.close; a failing close yields an error-status stop span and leaves the count unchanged until the retry succeeds.

Deliberately not in this PR (they belong to other splits of #2935): the CPU / memory / GPU sampling at span close, and the TimedSemaphore swap in the providers.

Validation

pytest tests/unit_tests/telemetry tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py: pass locally (nemo-lens-gated span tests skip locally and run on the cluster venv). Tests, all against a real in-memory OTel reader:
- test_sandbox_active.py (10): start then stop moves 1 → 0 and a second stop stays 0; failed create counts nothing; failed initial upload releases the count; a failed close keeps the sandbox counted until a retried stop releases it; group off records nothing; connect()ed sandbox not subtracted; no-op without telemetry; errors never reach the caller; non-monotonic and dimensioned by provider; one instrument per meter.
- test_sandbox_lifecycle_metrics.py (4): one startup sample per start and one exec sample per command with the provider attribute; explicit bounds end at 1,800,000 ms; failed create records no startup sample; group off records nothing.
- test_sandbox_create_retry.py (3): both providers' callbacks count by provider when the group is on, silent when off.
- test_call_sites.py (+1 test, +3 assertions): nemo.gym.sandbox.id on start and exec spans; a gym.sandbox.stop span with provider and id.
pre-commit run --files ... on the changed files: pass.

Rollout evidence

Slurm cluster, Terminal-Bench 4 (mini-SWE), OpenSandbox provider, sandbox span group on, metrics exported through an OTLP collector that also derives span metrics, so every Gym-emitted number has an independent counterpart to check against.

8-task run (concurrency 8, 24 min, all 8 rollouts scored, collector shut down cleanly):

Instrument	Gym-emitted	Independent source	Result
`gym.sandbox.create_retry_total{provider="opensandbox"}`	10	10 "Retrying OpenSandbox sandbox create" lines in the driver log	exact match
`gym.sandbox.startup_duration_ms`	25 samples, mean 91.6 s, p50 ≤ 60 s, p95 ≤ 300 s	span-derived `gym.sandbox.start` duration: 25 samples, mean 91.6 s, same p50 / p95	identical
`gym.sandbox.exec_duration_ms`	580 samples, mean 2.4 s, p95 ≤ 5 s	span-derived `gym.sandbox.exec`: 581 successful + 1 error span, p95 ≤ 5 s	agree (see note)
`gym.sandbox.stop` span	25 spans, all OK	25 starts	every sandbox released
`gym.sandbox.active`	0 → peak 14 → 0	8 sessions, each holding an agent sandbox, a shared-logs helper and later a verifier sandbox	consistent, no leak

With the SDK's default buckets, the startup histogram (mean 92 s) would have placed every sample in the overflow bucket and reported "10 s" for any percentile; with the explicit boundaries the p50 / p95 land in the 60 s / 300 s buckets, matching the span-derived series bucket for bucket.

Note on the exec count: the duration histograms are recorded after the span closes, so an operation that raises never reaches the record and is absent from the histogram; it does appear in the span-derived series with STATUS_CODE_ERROR. The one error exec accounts for the gap. This is deliberate: a failure's duration is not a latency, and the failure itself is counted where the status lives.

66-task run at concurrency 16 (earlier, counter only): gym.sandbox.active rose from 8 to a peak of 39 while the in-flight-rollouts proxy reported 16, and returned to 0 through 6 SANDBOX_NOT_FOUND rebuilds and 3 failed starts.

Compatibility

Additive. Nothing is recorded unless telemetry is on and the sandbox span group is enabled. New module telemetry/gym_metrics.py; no public API changes. AsyncSandbox.stop() semantics are unchanged from main: a failed close() leaves the sandbox started so the call can be retried (an earlier revision of this PR broke that and failed test_docker_compose; fixed in the last commit).

copy-pr-bot · 2026-09-22T22:02:09Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

copy-pr-bot · 2026-09-22T22:15:12Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

oyilmaz-nvidia · 2026-09-23T03:15:29Z

/ok to test 28b9946

oyilmaz-nvidia

LGTM, thanks!

## Background / motivation - Adopt released [FW-CI-templates v1.11.0](https://github.com/NVIDIA-NeMo/FW-CI-templates/releases/tag/v1.11.0), containing NVIDIA-NeMo/FW-CI-templates#582's SSO asset discovery/retry fix. - Replaces this PR's temporary candidate override after successful integration testing. Gym #3633 previously failed on deleted SSO asset `583229410`. ## What changed - Pin both `_cicd_preflight.yml` callers (`cicd-main.yml` and `release.yaml`) to release commit **`037094bcc9138c72bc125418e6e3653f9e1c4160`**. - Pin each caller's `workflow_ref` to the same release commit, so the inner action checkout cannot silently follow `main`. - Leave unrelated workflow versions, runner configuration, credentials, membership policy, and test jobs unchanged. ## Details - Public SHA literals carry the standard secrets-scanner false-positive annotation. - CI scope: **full** (workflow configuration). Model rollouts: **N/A**; no model, environment, agent, verifier, or benchmark behavior changes. - No user-facing API or migration impact. No merge or deployment performed. ```mermaid flowchart LR Gym[Gym CI or release] --> Workflow[Preflight at v1.11.0 SHA] Workflow --> Checkout[Inner checkout at same release SHA] Checkout --> SSO[Current SSO asset discovery and retry] SSO --> Gate[Existing membership and runner selection] ``` ## Tested - Release tag resolves to `037094bcc9138c72bc125418e6e3653f9e1c4160`; released downloader exists at that commit. - `pre-commit run --all-files`: passed. - `actionlint -shellcheck=''` on both changed workflows: passed. Full actionlint's pre-existing SC2129 warning in the unrelated container-build block was reproduced on the base. - Scoped `detect-secrets-hook` and `git diff --check`: passed. - Current Gym head: `c97b6e0398f8fff05df4f14ee852a060d5b2d671`; release-pin CI pending. - Historical proof of the same SSO fix before release: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35845632625/job/107131012058 — candidate checkout verified, **8,177-user roster downloaded**, membership and runner selection passed. This is not a claim that current-head or broader Gym CI has passed. --------- Signed-off-by: oliver könig <okoenig@nvidia.com>

…n counter Gym had no "sandboxes running now" signal: the sandbox span group emits gym.sandbox.start and gym.sandbox.exec spans but no stop span and no gauge, so dashboards had to approximate the count from in-flight /run requests on the agent server. AsyncSandbox now adds +1 to gym.sandbox.active when a provider hands back a handle and -1 when Gym releases it (also when the provider's close fails or the initial file upload fails), dimensioned by nemo.gym.sandbox.provider. An up-down counter rather than a gauge so a backend sums correctly across the agent-server processes that each hold a share of the sandboxes. A connect()ed sandbox is not subtracted, since the starting process counted it. The instrument lives on the lens meter but is Gym-owned; it follows the existing sandbox span-group gate and is a no-op without telemetry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com> (cherry picked from commit 2e188aa)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

…rs with real buckets nemo_gym/telemetry/metrics.py can only reach nemo-lens's five fixed, undimensioned instruments. Anything that needs an attribute (a sandbox provider, a site) has to be created directly on the lens meter, so this adds gym_metrics.py: a per-meter instrument cache and histogram / counter / up-down-counter recorders that are no-ops without telemetry and never raise. The sandbox lifecycle set lives here: gym.sandbox.active moves in from metrics.py unchanged in behaviour; gym.sandbox.startup_duration_ms, gym.sandbox.exec_duration_ms and gym.sandbox.create_retry_total are added for the next commits to record into. The two histograms declare explicit bucket boundaries up to thirty minutes: the SDK default ends at ten seconds, and a remote sandbox start routinely takes a minute, so every sample would otherwise land in the overflow bucket and a p95 would read "10 s" whatever the truth. The registry helpers and the three duration/retry recorders are taken from Onur Yilmaz's #2935, which this series splits out. Co-authored-by: Onur Yilmaz <oyilmaz@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

…ans, a stop span AsyncSandbox records gym.sandbox.startup_duration_ms around provider.create and gym.sandbox.exec_duration_ms around each command, per provider, with the explicit bucket boundaries from gym_metrics. The start and exec spans carry nemo.gym.sandbox.id so a trace can be joined to the provider's own logs, and stop() now opens a gym.sandbox.stop span around provider.close, so a trace pairs a sandbox's start with its end. All of it sits under the existing sandbox span group; nothing is recorded with the group off. The duration histograms and the id attribute are taken from Onur Yilmaz's #2935. Co-authored-by: Onur Yilmaz <oyilmaz@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

Retrying a create is provider-internal, so the facade cannot see it; the two providers that retry record gym.sandbox.create_retry_total from their tenacity before_sleep callback, next to the warning they already log. On a full Terminal-Bench 4 run against a degraded OpenSandbox this is the difference between "39 rollouts dropped" and "101 retries, then 39 dropped" being visible on a board rather than only in the driver log. The Daytona hook is from Onur Yilmaz's #2935. Co-authored-by: Onur Yilmaz <oyilmaz@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

The counter change had moved `_stopped = True` into a finally, so a sandbox whose provider close() raised was marked stopped and a later stop() no longer retried the close. The compose group relies on that retry (test_docker_compose::test_failed_cleanup_can_be_retried). A rejected close has released nothing: the sandbox stays started, stop() stays retryable, and gym.sandbox.active drops on the attempt that succeeds, once. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

github-actions · 2026-09-23T10:27:14Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35848865661

## Summary Add an opt-in `sink_window` history policy to the OSWorld Nemotron adapter: retain the earliest screenshots alongside a recent-image window, with intervening turns represented as text in chronological order. This branch includes upstream `main` at `d4c0f86c7` through a normal merge, including the NumPy metadata fix #3649, CI preflight update #3658, and sandbox lifecycle telemetry #3633. The OSWorld foundation from #2554/#3140 was consolidated and merged through #3592, so this PR no longer carries the old exact-trace/Core stack or depends on #2554. It does not require #3591. No separate issue is needed for this follow-up policy extension. ## Changes - Add configurable sink, low-water and high-water counts. Both watermarks include the sink; equal watermarks slide, while unequal watermarks provide hysteresis. - Represent non-contiguous image selections with intervals and render folded text in chronological order. - Distinguish context overflow, transport failure, truncated output, empty output and unparseable output; retain `agent_outcome_family=model_response_invalid` for compatibility. - On context overflow, reduce the recent-image window within the existing retry budget, preserve the sink/current observation, record actual shrink events, and stop when further reduction is impossible. - Preserve main's immediate model-timeout propagation. Report the terminal attempt's completion fact so an earlier malformed sample cannot hide a subsequent transport failure. The latest synchronization also annotates six deterministic SHA-256 prompt goldens that the previous secrets-detector run falsely identified as credentials. Only those literal lines are annotated; the digest values, assertions, repository scanner configuration, and baseline are unchanged. ## Design and compatibility History policy selects image/text turns; the adapter owns model protocol rendering and parsing. The runner owns horizon and runtime failure handling, the evaluator owns reward, and runtime admission owns masking. This PR does not change those ownership boundaries or add a training/evaluation switch. Fixed-window and hysteresis policy IDs and normal-path rendered prompts remain covered by deterministic golden tests, including fixed-3 and 3-10-3. Sink-window is opt-in, not a new default or a claim of improved benchmark score. Context-overflow recovery and more precise failure outcomes are intentional behavioral changes and may affect failed-run outcomes. The legacy scalar `image_window_start` accessor rejects non-contiguous plans; consumers should use `image_intervals` or per-turn decisions. Scope relative to current main remains limited to `responses_api_agents/osworld_agent/`: two implementation files, documentation, and tests. No Gym Core, dependency, deployment, sandbox-provider or model-server changes are included in the PR diff. ## Validation Re-run on the final main-synchronized tree: - **440 passed, zero skipped** across OSWorld agent, resource-server and benchmark tests; one non-failing Starlette deprecation warning. - `pre-commit run --all-files`: passed. - `uv lock --check --offline`: passed. - `git diff --check`: passed. - The same `detect-secrets==1.5.0` hook and checked-in baseline used by CI passed on all nine PR-scoped files after the inline false-positive annotations. - Existing model-timeout propagation, runner admission, prompt-parity, sink selection, overflow recovery and policy-identity regressions passed. The unchanged-overflow retry and mixed parse/transport completion bugs had been reproduced before their earlier fixes. Test invocation, using an existing compatible development environment from this checkout: ```bash export PYTHONPATH="$PWD" export NEMO_GYM_EXTRA_ROOTS="$PWD" python -m pytest \ responses_api_agents/osworld_agent/tests \ resources_servers/osworld/tests \ benchmarks/osworld/tests \ --import-mode=importlib -o addopts='' -o log_cli=false -q pre-commit run --all-files ``` The explicit source root prevents an older editable checkout from shadowing the branch under test; importlib mode avoids the shared `osworld` test-package name collision. These checks are local evidence, not a claim that the upstream CI run has completed. ## Score reproduction OSWorld 361 tasks, Nemotron-3-Nano-Omni v3 (`24e67ea0`), fixed-3 history, 100 steps, one A/B run per arm. Control = main `d4c0f86c7` + #3591 (needed for the E2B `endpoint()` on AgentEnv); treatment = control + this PR. Raw SR 44.61% → 46.62% (Δ +2.01 pp, paired 95% CI [−2.12, +6.07] pp); binary 154 → 159 of 361 (McNemar p = 0.59); 0 masked and 0 missing in both arms; identical history-policy and agent-contract IDs. No degradation. The context-overflow shrink path was not triggered in either run. --------- Signed-off-by: Jeff Peng <jepeng@nvidia.com> Co-authored-by: Michał Bień <mbien@nvidia.com>

iguehring-nvidia added the area:sandbox Sandbox providers, isolation, and sandbox runtime contracts label Sep 22, 2026

iguehring-nvidia self-assigned this Sep 22, 2026

iguehring-nvidia force-pushed the iguehring/sandbox-telemetry branch from 8bbcaf2 to 1fd06b3 Compare September 22, 2026 22:15

iguehring-nvidia changed the base branch from iguehring/sandbox-active-counter to main September 22, 2026 22:15

iguehring-nvidia mentioned this pull request Sep 22, 2026

feat(sandbox): count running sandboxes in a gym.sandbox.active up-down counter #3581

Closed

iguehring-nvidia force-pushed the iguehring/sandbox-telemetry branch from 1fd06b3 to 28b9946 Compare September 22, 2026 22:17

iguehring-nvidia requested a review from hemildesai September 22, 2026 22:26

iguehring-nvidia marked this pull request as ready for review September 22, 2026 22:26

iguehring-nvidia requested a review from oyilmaz-nvidia September 22, 2026 22:26

iguehring-nvidia enabled auto-merge September 22, 2026 22:49

yaoyu-33 added complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 22, 2026

iguehring-nvidia mentioned this pull request Sep 22, 2026

feat(telemetry): count finished rollouts by outcome, class and reason #3638

Merged

nemo-automation-bot Bot requested review from a team September 23, 2026 03:15

copy-pr-bot Bot temporarily deployed to public September 23, 2026 03:16 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 03:17 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 03:18 Inactive

oyilmaz-nvidia previously approved these changes Sep 23, 2026

View reviewed changes

iguehring-nvidia dismissed oyilmaz-nvidia’s stale review via 878fa72 September 23, 2026 08:05

iguehring-nvidia force-pushed the iguehring/sandbox-telemetry branch from 878fa72 to ae7d23d Compare September 23, 2026 08:13

nemo-automation-bot Bot requested a review from a team September 23, 2026 08:13

copy-pr-bot Bot temporarily deployed to public September 23, 2026 08:13 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 08:15 Inactive

iguehring-nvidia force-pushed the iguehring/sandbox-telemetry branch from ae7d23d to 950e42b Compare September 23, 2026 09:00

copy-pr-bot Bot temporarily deployed to public September 23, 2026 09:00 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 09:03 Inactive

This was referenced Sep 23, 2026

fix: recover SSO roster downloads after release asset rotation NVIDIA-NeMo/FW-CI-templates#581

Closed

ci: bump preflight workflows to v1.11.0 #3658

Merged

iguehring-nvidia and others added 6 commits September 23, 2026 12:14

refactor(telemetry): shorter notes on the gym.sandbox.active instrument

7a90dbb

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

iguehring-nvidia force-pushed the iguehring/sandbox-telemetry branch from 950e42b to 22351ab Compare September 23, 2026 10:15

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:15 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:17 Inactive

copy-pr-bot Bot temporarily deployed to public September 23, 2026 10:18 Inactive

iguehring-nvidia added this pull request to the merge queue Sep 23, 2026

Merged via the queue into main with commit d4c0f86 Sep 23, 2026
41 checks passed

iguehring-nvidia deleted the iguehring/sandbox-telemetry branch September 23, 2026 10:39

JeffPengCoder mentioned this pull request Sep 23, 2026

feat(osworld): add sink-window history policy #3139

Merged

This was referenced Oct 1, 2026

feat(observability): instrument sandbox lifecycle durations retries and active counts #3866

Closed

[epic] Rollout observability & reliability fast-follows #2831

Open

iguehring-nvidia mentioned this pull request Oct 19, 2026

feat(telemetry): stamp the rollout id on sandbox spans #4752

Merged

sshiddib mentioned this pull request Oct 23, 2026

swe_agents: run agent and grading containers through the shared sandbox layer #5015

Open

4 tasks

Navigation Menu

Repository navigation

feat(sandbox): lifecycle telemetry: durations, retries, sandbox ids and a stop span - #3633

feat(sandbox): lifecycle telemetry: durations, retries, sandbox ids and a stop span#3633
iguehring-nvidia merged 6 commits into
mainfrom
iguehring/sandbox-telemetry

iguehring-nvidia commented Sep 22, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

oyilmaz-nvidia commented Sep 23, 2026

Uh oh!

oyilmaz-nvidia left a comment

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

iguehring-nvidia commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Validation

Rollout evidence

Compatibility

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

copy-pr-bot Bot commented Sep 22, 2026

Uh oh!

oyilmaz-nvidia commented Sep 23, 2026

Uh oh!

oyilmaz-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iguehring-nvidia commented Sep 22, 2026 •

edited

Loading