[JeffPengCoder]

Summary

This PR adds OSWorld as a first-class NeMo Gym benchmark and introduces a reusable Gym rollout runtime for desktop-computer agents.

OSWorld is a stateful desktop benchmark covering Chrome, LibreOffice, GIMP, VS Code, VLC, multi-application workflows, and operating-system tasks. Each task requires a real desktop VM, iterative visual observations and actions, task-specific setup, and final-state evaluation.

One Gym /run request corresponds to one complete OSWorld rollout:

1. create a fresh task environment;
2. reset it with the original OSWorld task definition;
3. run the selected model/agent loop;
4. execute actions through OSWorld;
5. call OSWorld's inline evaluator;
6. return the trajectory, evaluator score, Gym reward, failure classification, and artifact metadata in a BaseVerifyResponse-compatible shape.

The integration preserves OSWorld's task, controller, action, and evaluator semantics. Gym owns orchestration, model routing, concurrency, sandbox lifecycle integration, reproducibility, observability, and result accounting.

The full benchmark guide is in benchmarks/osworld/README.md. The reusable runtime contract is documented in responses_api_agents/osworld_agent/README.md.

Why the integration is structured this way

OSWorld is not a stateless verifier. Its evaluator depends on the live DesktopEnv, task setup state, files inside the VM, and application-specific controller endpoints.

For that reason:

• the complete rollout and evaluation stay in the same Ray worker;
• each rollout owns one independent VM lifecycle;
• OSWorld continues to own DesktopEnv.reset(), action execution, and DesktopEnv.evaluate();
• Gym wraps that lifecycle instead of translating OSWorld tasks or reimplementing evaluators;
• there is no separate resources_servers/osworld service.

Splitting evaluation into another service would require exporting live environment handles or reproducing task state outside the worker, without providing a useful stateless boundary.

Architecture

Benchmark preparation
  benchmarks/osworld/prepare.py
    |-- validates the selected task JSONL
    |-- optionally creates deterministic task shards
    |-- prefetches task setup and evaluator assets
    |-- records input, qcow2, and container identities
    `-- writes a private benchmark-local env.yaml
                         |
                         v
                    gym eval run
                         |
                         v
               OSWorld agent /run
                 async admission gate
                 concurrency semaphore
                         |
                         v
               one Ray task per rollout
                         |
                         v
                   RunnerRegistry
       +-----------------+------------------+
       |                 |                  |
 Gym-owned runner   upstream runners   model-specific adapter
 gym_pyautogui      Prompt/Pointer/M3   Nemotron Nano Omni
                    Qwen3 Omni
       +-----------------+------------------+
                         |
                  configured model
           Gym model server or runner-native
                    transport contract
                         |
                         v
                  OSWorld DesktopEnv
        reset -> observe -> model -> parse -> execute
                         |
                    evaluate()
                         |
             +-----------+------------+
             |                        |
     OSWorld provider          Gym Sandbox bridge
     owns VM lifecycle         GymSandboxDesktopProvider
                                      |
                               Gym Sandbox API
                                      |
                         Docker provider, local or SSH
                                      |
                         container -> QEMU/KVM -> desktop VM
             +-----------+------------+
                         |
                         v
                OSWorldVerifyResponse
          |-- raw OSWorld evaluator score
          |-- configured binary or raw reward
          |-- mask_sample and termination reason
          |-- executed model steps
          `-- optional artifact provenance

The physical roles are also explicit and may be co-located or split:

MODEL_HOST  <- model HTTP -  AGENT_CONTROL_HOST  - Docker API ->  ENVIRONMENT_HOST
                                  |
                                  `-- prepare / control / eval

• The model host serves a compatible vision-language model.
• The agent/control host runs benchmark preparation, Gym control, Ray workers, and evaluation.
• The environment host runs Docker, QEMU/KVM, and the OSWorld desktop VMs.

For a local smoke test, the agent/control and environment roles can be the same machine. For a split-host deployment, Gym uses Docker's standard SSH transport through DOCKER_HOST=ssh://...; no persistent interactive SSH session is part of the rollout path.

Ownership boundaries

Layer	Owns	Does not own
benchmarks/osworld	Dataset composition, profiles, task sharding, asset preparation, manifests, and launch recipes	Agent loop or evaluator semantics
responses_api_agents/osworld_agent	/run, concurrency, runner selection, model routing, rollout lifecycle, failure classification, and Gym response mapping	OSWorld task setup or final-state evaluation logic
OSWorld	DesktopEnv, controllers, setup, observations, action execution, and evaluators	Gym orchestration or cluster lifecycle
Gym Sandbox	Container lifecycle, endpoint discovery, status, and cleanup	Desktop task semantics
Runner/adapter	Prompt, history, model protocol, parser, and coordinate contract	Environment scoring
Model server	Model inference and optional model-call observability	VM or task lifecycle

Design principles

Preserve benchmark semantics

The complete task object from verifier_metadata.osworld_task is passed to DesktopEnv.reset(task_config=...) without translating setup or evaluator behavior. Final scoring still comes from DesktopEnv.evaluate().

Separate independent axes

The following are independent configuration choices:

• runner/model protocol;
• observation and action contract;
• VM lifecycle backend;
• local versus split-host placement;
• model-serving topology.

A remote environment host is not a different agent implementation, and selecting Gym Sandbox does not change the prompt, actions, or evaluator.

Adapt at stable contracts

RunnerRegistry makes each runner's environment class, agent class, observation type, action space, and default arguments explicit.

Where possible, Gym preserves an upstream OSWorld runner. Gym-owned adapters are used only when a model has a distinct prompt/history/parser contract that is not represented by an upstream runner.

The Gym Sandbox integration implements OSWorld's provider contract instead of replacing DesktopEnv or copying the rollout loop.

Distinguish model outcomes from infrastructure failures

A valid evaluator score of zero is still a valid benchmark or training sample.

Infrastructure failures, rollout timeouts, invalid environment configuration, and unfinished rollouts set mask_sample=true and retain a specific termination reason. They are not silently converted into ordinary model failures.

Task setup operations that explicitly declare on_nonzero: score_zero remain valid score-zero results rather than masked infrastructure failures.

Make scale reproducible

• One rollout owns one VM lifecycle.
• Agent concurrency is bounded explicitly.
• Task sharding is deterministic, disjoint, and exhaustive.
• Shard manifests include the source hash, selected task IDs, and task-ID digest.
• The OSWorld dependency and container image are immutable.
• Benchmark preparation records the qcow2 identity and reset semantics.
• Completed task IDs can be retained while recovery runs only the missing canonical set.

Keep diagnostics opt-in

Screenshots, video, VM execution logs, and full model I/O can be large and sensitive. They are disabled by default and enabled only through explicit environment variables.

What this PR adds

Benchmark composition

benchmarks/osworld/ provides:

• benchmark metadata and default composition;
• five-task example data and example rollouts;
• the 39-task test_small subset;
• prepare.py for input validation, profile selection, environment generation, deterministic sharding, VM identity recording, and asset preparation;
• model/runner profiles for the generic Gym runner, Pointer Agent, PromptAgent, and Nemotron 3 Nano Omni;
• model endpoint, environment, VM preparation, control, evaluation, and run-scoped cleanup tools;
• local and split-host deployment documentation.

The larger canonical task input is supplied explicitly at preparation time and is not committed as a generated repository artifact.

Reusable OSWorld runtime

responses_api_agents/osworld_agent/ provides:

• the Gym /run server and aggregate OSWorld metrics;
• the synchronous DesktopEnv rollout loop executed inside Ray;
• explicit raw versus binary reward handling;
• timeout and mask semantics;
• runner registration and validation;
• model-specific adapters;
• pyautogui and control-action parsing;
• proxy-task validation and non-secret provenance;
• shared setup-cache staging;
• task-scoped logs, screenshots, trajectories, manifests, and video hooks;
• Gym Sandbox compatibility for standard and Pointer DesktopEnv variants.

Registered runner contracts

Runner	Ownership and contract
gym_pyautogui	Gym prompt, screenshot observations, and Python/pyautogui actions
prompt_agent and prompt_agent_*	Upstream OSWorld PromptAgent observation/action variants, routed to the configured policy model
pointer_agent	Upstream PointerAgent planner/executor/verifier behavior and Anthropic-compatible model contract
m3_agent	Upstream MiniMax M3 scaffold, transport, prompt, and relative-coordinate protocol
nemotron_v3_nano_omni_agent	Gym-owned Nemotron 3 Nano Omni prompt, bounded image history, parser, and coordinate projection
qwen3_omni_agent	Upstream Qwen3VL scaffold with Gym-owned transport compatibility

The maintained prepare.py --profile entry points are currently default, pointer, and nano_omni. Other registered runner contracts can be selected through explicit compatible configuration.

Gym Sandbox integration

The runtime supports two execution backends:

Backend	Lifecycle owner	Intended use
osworld_provider	OSWorld provider implementation	Compatibility and direct OSWorld-provider deployments
gym_sandbox	Gym Sandbox, through GymSandboxDesktopProvider	Default benchmark workflow and provider-neutral lifecycle integration

In the Gym Sandbox path:

• OSWorld still owns DesktopEnv, setup, controllers, actions, and evaluators;
• Gym Sandbox owns container creation, endpoint discovery, status, and cleanup;
• the OSWorld qcow2 is mounted read-only;
• reset means closing the Sandbox and recreating a fresh VM from the same base;
• /dev/kvm is validated by Docker on the actual environment host;
• all required OSWorld service ports are published dynamically;
• local and remote Docker daemons use the same provider code.

Small generic Gym changes required by OSWorld

This PR also includes narrowly scoped core changes needed by the integration:

• optional Sandbox endpoint discovery as a provider capability;
• explicit Sandbox service-port declarations;
• Docker provider support for dynamic multi-port publication and remote environment hosts;
• positional compatibility for existing SandboxSpec construction;
• benchmark-config discovery in the generated environment table;
• configurable preservation of reasoning in multi-turn vLLM assistant history;
• opt-in correlated vLLM transport logging.

The endpoint capability is optional, model capture remains disabled unless configured, and existing non-OSWorld provider behavior is preserved.

Request, reward, and failure contract

The rollout request carries the complete OSWorld task under:

verifier_metadata.osworld_task

Per-task responses_create_params may override configured sampling parameters.

A completed response contains:

• reward;
• mask_sample;
• verifier_metadata.osworld_score;
• the number of executed OSWorld steps;
• completion/error and termination state;
• runner and model identity;
• proxy provenance without credentials;
• optional task artifact location;
• one assistant output item per model step.

reward_mode controls the top-level Gym reward:

• binary: reward is 1.0 only for a full OSWorld evaluator score of 1.0;
• raw: reward preserves fractional OSWorld evaluator credit.

Aggregate metrics always report both:

• osworld/binary_success_rate;
• osworld/raw_reward_rate.

This prevents partial evaluator credit from being confused with full task success.

Dataset and denominator convention

The canonical full input used by the validation runs contains 369 unique OSWorld task IDs with SHA-256:

3be0746d59fc5a311447e3e36241381c022932c624315b851fef2c28ecbf5ac9

Eight GDrive tasks require an external Google OAuth client_secrets.json that was not available in the validation environment.

Validation therefore reports two explicit views:

• full attempted input: 369 tasks;
• comparable no-GDrive view: the same 361 task IDs after excluding exactly those eight credential-blocked IDs.

Every run first validates all 369 canonical IDs. Only the frozen GDrive-eight set is excluded from the 361-task comparison. Non-GDrive model failures, masks, proxy failures, and evaluator-zero outcomes remain in the denominator as zero.

Reproducibility and dependency boundary

The agent installs OSWorld from the immutable archive:

JeffPengCoder/OSWorld@31b76bf1c4d4e589238b314caa91470afc52651e

This is a fork commit, not an upstream OSWorld release. It includes upstream OSWorld main 83e85344 plus the nv-gym provider/proxy integration changes.

Gym does not modify the installed OSWorld tree at runtime. The phrase "unmodified dependency" in this integration means that the pinned archive is installed and consumed without run-local patching; it does not mean that the pinned commit is byte-identical to xlang-ai/OSWorld main.

The default OSWorld container is pinned to:

happysixd/osworld-docker@
sha256:0e6497a9295647cf05bf2b2af522fdd79bdeba2737595259cab310a3bcf6baa9

Benchmark preparation records:

• task input SHA-256;
• generated shard membership;
• qcow2 path, size, and SHA-256;
• container digest;
• selected profile/backend;
• output and cache locations.

Asset preparation is idempotent. Prefetched assets change download timing and resilience, not task semantics.

Verification

Current-head contract tests

Current PR head:

32f37c0daea45daad7754ccf89e9f042f8b4b67c

The following targeted suite passes on the current checkout:

.venv/bin/pytest -q \
  responses_api_agents/osworld_agent/tests \
  benchmarks/osworld/tests \
  tests/test_update_env_list.py \
  tests/unit_tests/test_update_env_list.py \
  tests/unit_tests/test_sandbox.py \
  tests/unit_tests/test_sandbox_connect.py \
  tests/unit_tests/test_base_responses_api_model.py \
  responses_api_models/vllm_model/tests/test_app.py

Result:

392 passed, 11 skipped (optional tenacity sandbox dependency not installed)

The CI-equivalent isolated server suites also pass:

.venv/bin/gym env test +entrypoint=responses_api_agents/osworld_agent
.venv/bin/gym env test +entrypoint=responses_api_models/vllm_model
.venv/bin/gym env test +entrypoint=responses_api_agents/stirrup_agent

Results:

OSWorld: 153 passed
vLLM:    110 passed
Stirrup: 151 passed

The suite covers runner contracts, action parsing, request/response mapping, raw and binary metrics, masks, proxy handling, setup caching, task artifacts, Sandbox lifecycle adaptation, endpoint resolution, sandbox serialization and reconnection, model-call observability, task sharding, VM manifests, launch-script contracts, reasoning history, and model transport behavior.

git diff --check upstream/main...HEAD is also clean.

Live Gym Sandbox validation

A near-head branch ancestor, 663e53e62ed740d803c65ac6df5494da93a7c687, completed the full canonical Nano Omni validation using:

• nemotron_v3_nano_omni_agent;
• Gym Docker Sandbox;
• two deterministic 185/184 task shards;
• 16 concurrent environments per shard;
• 100 maximum actions;
• temperature 0.6, top-p 0.95;
• a shared TP=8 vLLM 0.23 model endpoint;
• the pinned OSWorld and VM/container identities documented above.

Result:

View	Tasks	Raw evaluator	Raw rate	Binary successes	Binary rate	Masks
Full canonical input	369	180.9759876	49.0450%	175	47.4255%	11
Exact no-GDrive view	361	180.9759876	50.1319%	175	48.4765%	3

Integrity checks:

• 369/369 unique canonical IDs;
• 0 missing IDs;
• 0 extra IDs;
• 0 duplicate IDs;
• exactly eight GDrive credential failures excluded from the comparable view;
• all three non-GDrive masked outcomes retained in the 361 denominator;
• zero run-owned containers remaining after cleanup.

Controlled repeated runs

The same fixed Nano Omni contract was repeated three times over the same canonical task set:

Run	Official tasks	Raw rate	Binary rate	Retained non-GDrive masks
Round 1	361	50.13%	48.48%	3
Round 2	361	49.48%	47.09%	11
Round 3	361	48.13%	45.98%	1
Mean +/- sample SD	361/run	49.25% +/- 1.02 pp	47.18% +/- 1.25 pp	-

Every round contains the same 369 canonical IDs before the exact GDrive-eight exclusion and the same 361 IDs afterward.

These are system-level benchmark results at nonzero sampling temperature. They validate end-to-end completeness, task accounting, reproducibility, recovery, and score stability; they are not presented as an isolated model-quality claim.

Pointer Agent compatibility evidence

A separate Pointer/Opus 4.7 experiment at Gym commit 09a895d7 completed all 369 tasks.

On the comparable 361-task view:

Gym Pointer/Opus raw rate:       296.5627129 / 361 = 82.1503%
Verified leaderboard reference: 301.94 / 361       = 83.64%
Delta:                                                -1.4897 pp

That experimental commit is not an ancestor of the current PR head, so this result is compatibility evidence for the Pointer wrapping design, not a current-head merge gate.

Backward compatibility

• sandbox_provider: null retains the OSWorld-native lifecycle path.
• Sandbox endpoint lookup is an optional provider capability.
• Existing SandboxSpec positional construction remains compatible.
• Existing Docker provider behavior is unchanged unless ports, devices, mounts, or a remote Docker daemon are explicitly configured.
• Reasoning-history preservation is opt-in.
• Model-call capture and OSWorld artifact logging are opt-in.
• Video recording has no runtime overhead when OSWORLD_RECORD_VIDEO_DIR is unset.
• Existing non-OSWorld agents and resources servers do not consume the OSWorld runtime.

Security and artifact handling

• Proxy credentials are external runtime configuration and are never committed.
• Result metadata records only non-secret proxy provenance.
• The pinned OSWorld revision prevents task setup DEBUG logging from serializing the complete worker environment.
• Full model I/O may contain prompts and embedded screenshots.
• Task artifacts and video may contain user-visible desktop content.
• Large screenshots, trajectories, VM images, model logs, and caches are intentionally not part of the normal repository artifact set.

Limitations and planned follow-ups

1. OSWorld fork dependency

   The runtime currently depends on the immutable JeffPengCoder/OSWorld commit above rather than an upstream release. Provider and proxy changes should be upstreamed or replaced with an official pinned revision before removing this dependency.

2. Full dataset publication

   The committed five-task example and 39-task subset are runnable. The complete generated benchmark JSONL is intentionally not committed, and the registry-backed train.jsonl and validation.jsonl artifacts have not yet been published.

3. GDrive credential tasks

   Eight canonical tasks require valid external Google OAuth credentials. This PR does not fabricate, distribute, or bypass those credentials.

4. Default platform

   The default OSWorld Docker image is x86_64 and expects Linux with Docker and KVM. ARM64 environment images are outside this PR.

5. Reset semantics

   Gym Sandbox resets by closing the current Sandbox and recreating it from the read-only qcow2 base. Live RAM/device-state snapshots are not implemented.

6. OpenSandbox backend

   OpenSandbox has been exercised downstream but its provider implementation is not included here.

7. Training recipes

   The response contains the reward, mask, trajectory, and model-step information needed by downstream training integrations, but a complete NeMo RL training recipe is outside this PR.

8. Sensitive diagnostics

   Full model I/O, screenshots, VM execution logs, and video should be enabled only for controlled diagnostic runs with appropriate storage and access controls.

Reviewer guide

A suggested review order is:

1. Request, reward, and failure semantics:

   • responses_api_agents/osworld_agent/app.py
   • responses_api_agents/osworld_agent/client.py

2. Runner ownership and model-facing contracts:

   • responses_api_agents/osworld_agent/runner_registry.py
   • responses_api_agents/osworld_agent/adapter_agents.py
   • responses_api_agents/osworld_agent/action_parser.py

3. Sandbox ownership boundary:

   • responses_api_agents/osworld_agent/sandbox_desktop_env.py
   • responses_api_agents/osworld_agent/sandbox_provider.py
   • nemo_gym/sandbox/
   • nemo_gym/sandbox/providers/docker/

4. Benchmark preparation and operations:

   • benchmarks/osworld/prepare.py
   • benchmarks/osworld/assets.py
   • benchmarks/osworld/configs/
   • benchmarks/osworld/tools/

5. Model-server compatibility and observability:

   • nemo_gym/base_responses_api_model.py
   • responses_api_models/vllm_model/app.py

6. Contract and regression tests:

   • responses_api_agents/osworld_agent/tests/
   • benchmarks/osworld/tests/
   • tests/unit_tests/test_sandbox.py
   • tests/unit_tests/test_docker_provider.py

Documentation

• OSWorld benchmark guide
• OSWorld runtime reference
• Deployment tool reference
• Gym Docker Sandbox provider notes

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

also could add this to benchmarks/

but needs further review from team

[JeffPengCoder]

also could add this to benchmarks/

but needs further review from team

Makes sense — happy to move it under benchmarks/ if that's the preferred home. I'll defer to the team on placement; let me know the call and I'll restructure.

[cmunley1]

cc @gchlebus

[Glorf]

I think the integration is close, but I would request changes for two concrete issues before merging:

1. The generic HeadServer service registry remains after its only intended in-tree consumer was deferred and removed.
2. The local /dev/kvm validation breaks the advertised split-host deployment.

Also, please try to pass a simplification round with the agent of your choice, as the amount of code is still very large

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feature-osworld.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/model-server/model-call-capture

[Glorf]

Cond. approved if sandbox changes are well tested. I'd strongly recommend merging only after 0.5.0 release freeze (today-tomorrow) to give some more time for testing (we don't need this in 0.5.0 AFAIK)