Moved to internal GitLab

benchmarks/enterpriseops_gym/README.md

@@ -5,7 +5,7 @@ and tool use in enterprise workflows: Customer Service Management (CSM), HR, IT
 Calendar, Teams and Drive, plus Hybrid tasks that span two of them. The agent operates live systems through MCP tools
 and is graded on the final state of their databases.
 
-This benchmark runs the public oracle-mode tasks the way Artificial Analysis runs
+This benchmark runs all 1,117 oracle-mode tasks, the set Artificial Analysis evaluates, the way it runs
 [EnterpriseOps-Gym-AA](https://artificialanalysis.ai/methodology/intelligence-benchmarking#enterprise-ops-gym-aa): the
 [Stirrup](https://github.com/ArtificialAnalysis/Stirrup) agent harness in a reason-and-act loop with a 100-turn cap,
 only the tools each task needs, 3 repeats, and strict pass@1 (a task passes only when every verifier passes).
@@ -42,15 +42,19 @@ only the tools each task needs, 3 repeats, and strict pass@1 (a task passes only
 
 **Provenance**
 
-- Tasks: [`ServiceNow-AI/EnterpriseOps-Gym`](https://huggingface.co/datasets/ServiceNow-AI/EnterpriseOps-Gym) at
-  revision `c8e538e`, config `oracle`, all eight domain splits: 649 tasks (Calendar 61, CSM 103, Drive 64, Email 67,
-  HR 102, Hybrid 88, ITSM 103, Teams 61), Apache 2.0. This is the public 60% of the benchmark; Artificial Analysis
-  evaluates 1,117 tasks, so its numbers are not directly comparable.
+- Tasks: [`nvidia/EnterpriseOps-Gym-Full`](https://huggingface.co/datasets/nvidia/EnterpriseOps-Gym-Full) at revision
+  `14e2a7e`, config `oracle`, all eight domain splits: 1,117 tasks (Calendar 104, CSM 186, Drive 105, Email 104,
+  HR 184, Hybrid 153, ITSM 181, Teams 100), CC BY-NC 4.0. It holds the 649 tasks ServiceNow publishes
+  ([`ServiceNow-AI/EnterpriseOps-Gym`](https://huggingface.co/datasets/ServiceNow-AI/EnterpriseOps-Gym) at `c8e538e`)
+  unchanged, and 468 more.
 - Tool schemas: the gym servers' `tools/list`, snapshotted in
   [`nvidia/NeMo-Gym-EnterpriseOps-Assets`](https://huggingface.co/datasets/nvidia/NeMo-Gym-EnterpriseOps-Assets) at
   revision `8918dc6`; identical to what the pinned images serve.
-- Task databases: `gym_dbs.zip` from [ServiceNow/EnterpriseOps-Gym](https://github.com/ServiceNow/EnterpriseOps-Gym)
-  at `271f2c3`, pinned by sha256.
+- Task databases: [`nvidia/EnterpriseOps-Gym-DBs`](https://huggingface.co/datasets/nvidia/EnterpriseOps-Gym-DBs) at
+  revision `8742ff2`, the SQL dumps of all 1,117 tasks. The dumps of the public tasks are byte-identical to
+  `gym_dbs.zip` in [ServiceNow/EnterpriseOps-Gym](https://github.com/ServiceNow/EnterpriseOps-Gym) at `271f2c3`.
+- Both datasets are private to NVIDIA: preparing the data and running the resources server need an `hf_token` with
+  access to them.
 - Gym servers: the upstream images `shivakrishnareddyma225/enterpriseops-gym-mcp-<domain>` (amd64), pinned by digest.
 
 **Task contract**
@@ -59,8 +63,9 @@ only the tools each task needs, 3 repeats, and strict pass@1 (a task passes only
   message. Stirrup puts its own instructions first: complete the task with the tools in 100 steps, call a finish tool
   as the last step, and do not interact with the user.
 - Tools: the task's oracle tools (`selected_tools` minus `restricted_tools`) with the gym servers' schemas, and
-  Stirrup's `finish(reason, paths)`. No shell. Two tasks select a tool no image serves (`list_team_apps`,
-  `remove_channel_member`) and run without it, as upstream.
+  Stirrup's `finish(reason, paths)`. No shell. As upstream, a task that selects no tools gets every tool of its gym
+  servers (one task), and a selected tool that no image serves is left out (four tasks: `add_teamwork_tag_member`,
+  `list_team_apps`, `remove_channel_member`).
 - Each gym call carries the task database's ID and the task's context headers, mapped from the task's `context` as the
   upstream harness maps them.
 
@@ -73,6 +78,9 @@ only the tools each task needs, 3 repeats, and strict pass@1 (a task passes only
 - Harness: Stirrup in Gym's `stirrup_agent`, 100 turns, thinking on, `top_p` 0.95. The temperature defaults to 0.6,
   Artificial Analysis's setting for reasoning models; use the model lab's recommendation where there is one (see
   [Run](#run)).
+- Reasoning effort: Artificial Analysis runs each model at a named effort (`max` for GLM-5.3 and GLM-5.3-Flash).
+  Set it on the model server, since an endpoint's default can be lower: on `inference-api.nvidia.com` the GLM
+  models' default spends a quarter to a third of the reasoning tokens of `max`.
 
 ## Prepare data
 
@@ -97,64 +105,72 @@ gym eval run \
     --split benchmark \
     --concurrency 64 \
     --output results/enterpriseops_gym.jsonl \
-    ++enterpriseops_gym_stirrup_agent.responses_api_agents.stirrup_agent.temperature=1.0
+    ++enterpriseops_gym_stirrup_agent.responses_api_agents.stirrup_agent.temperature=1.0 \
+    ++policy_model.responses_api_models.openai_model.extra_body.reasoning_effort=max
 ```
 
+Set `hf_token` (in `env.yaml` or as `++hf_token=...`) to a token with access to both datasets.
+
 Tasks run as environment-server episodes (`taskset: enterpriseops_gym`), so sampling is set on the agent rather than
 with `--temperature`, which overrides request fields of flat rows only. Add `++observability_enabled=true
 ++model_call_capture_dir=<absolute dir>` to record the model calls that rollout health checks and the harness evidence
-gate read.
+gate read. Recorded rollouts are large (about 2 MB each in the collector's memory); on a host with a memory limit, add
+`++retain_results_in_memory=false` so the collector writes each rollout out instead of holding all 3,351.
 
 Each rollout uses one sandbox (two for Hybrid tasks) with 2 CPUs and 4 GiB of memory for the length of the episode.
 
 ## Results
 
-GLM-5.3-Flash (`nvidia/zai-org/glm-5.3-flash`), temperature 1.0, 649 tasks x 3 repeats on OpenSandbox; 1,947 of
-1,947 rollouts completed with no infrastructure failure in 44 minutes at concurrency 64.
+GLM-5.3-Flash (`nvidia/zai-org/glm-5.3-flash`), reasoning effort `max`, temperature 1.0, all 1,117 tasks x 3 repeats
+on OpenSandbox; 3,351 of 3,351 rollouts completed. Artificial Analysis evaluates the same 1,117 tasks, so the numbers
+compare directly.
 
 | Domain | Success (this benchmark) | Success (EnterpriseOps-Gym-AA) |
 |---|---:|---:|
-| CSM | 30.7 | 47.3 |
-| HR | 31.4 | 34.8 |
-| ITSM | 37.2 | 38.1 |
-| Email | 60.7 | 37.5 |
-| Calendar | 36.1 | 5.8 |
-| Teams | 51.4 | 7.0 |
-| Drive | 54.2 | 58.1 |
-| Hybrid | 36.4 | 24.2 |
-| **Overall** | **40.5** | **33.2** |
-
-- Verifier pass rate 74.2%; success per repeat 40.4, 39.8, 41.3. Two earlier full runs scored 41.0 and 41.0.
-- A rollout averages 7.6 model calls, 12.2 tool calls and 2.3k output tokens (1.1k reasoning). Artificial Analysis
+| CSM | 44.3 | 47.3 |
+| HR | 41.7 | 34.8 |
+| ITSM | 37.0 | 38.1 |
+| Email | 62.8 | 37.5 |
+| Calendar | 43.3 | 5.8 |
+| Teams | 52.7 | 7.0 |
+| Drive | 62.2 | 58.1 |
+| Hybrid | 38.3 | 24.2 |
+| **Overall** | **45.9** | **33.2** |
+
+- Verifier pass rate 76.5%; success per repeat 45.6, 45.9, 46.3; 44.9% on the 649 public tasks and 47.4% on the 468
+  others.
+- A rollout averages 7.3 model calls, 13.0 tool calls and 9.6k output tokens (8.0k reasoning). Artificial Analysis
   reports 11.4 turns and 13.8k output tokens per task for this model.
-- On the six domains other than Calendar and Teams, weighted by the per-domain task counts that Artificial
-  Analysis's scores imply, the success rate is 39.2% here and 39.2% at Artificial Analysis. On Calendar and Teams,
-  Artificial Analysis reports identical scores (5.8% and 7.0%) for GLM-5.3 and GLM-5.3-Flash, while its other models
-  solve 36-52% there; that floor does not appear here.
-- The run recorded model calls: all 1,947 rollouts are healthy and pass the `gym-p0/v1` evidence gate.
+- The gap is in the domains where Artificial Analysis's GLM results have a floor. It reports 5.8% on Calendar, 7.0%
+  on Teams and 37.5% on Email for GLM-5.3-Flash, and nearly the same for GLM-5.3, while its other models solve 39-70%
+  there. 117 of the 153 Hybrid tasks use one of those three gym servers. On CSM, HR, ITSM and Drive the success rate
+  is 44.4% here and 43.0% at Artificial Analysis.
+- The run recorded model calls: 3,331 of 3,351 rollouts are healthy; the other 20 have a model call that used its
+  whole output budget.
 
 ## Validation
 
-The public task set has no reference solutions, so the integration is checked against the upstream harness's own
-code instead:
+There are no reference solutions, so the integration is checked against the upstream harness's own code instead:
 
 - **Verifier goldens.** `resources_servers/enterpriseops_gym/tests/verifier_golden.json` records what upstream's
   `VerifierEngine` extracts from 12 kinds of `/api/sql-runner` answer and decides for 660 combinations of actual
   value, expected value and comparison. The server's verifier matches all of them except the 21 comparisons of a
   number with a numeric string.
-- **Replay.** The first repeat of every task in the results run was replayed with upstream's `MCPClient`,
-  `create_database_from_file` and `VerifierEngine` against fresh gym servers. 647 of 649 tasks replay (upstream
-  cannot send the two whitespace headers). After normalizing generated IDs and timestamps, 99.8% of the tool outputs
-  are identical; the others differ only in generated values (random IDs and the order they sort in, PDF bytes).
-  Upstream's verifiers reach the same outcome on every task once numbers equal their numeric strings.
-- **Do-nothing control.** Scored on the freshly seeded databases, 1 of 647 tasks passes, under both harnesses.
+- **Replay.** The first repeat of every task was replayed with upstream's `MCPClient`, `create_database_from_file`
+  and `VerifierEngine` against fresh gym servers: the public tasks from a run of the 649-task set, the other 468 from
+  the results run above. 1,114 of 1,117 tasks replay (upstream cannot send the three whitespace headers). After
+  normalizing generated IDs and timestamps, 99.8% of the 13,545 tool outputs are identical; the others differ only in
+  generated values (random IDs and the order they sort in, PDF bytes). Upstream's verifiers reach the same outcome on
+  every task once numbers equal their numeric strings.
+- **Do-nothing control.** Scored on the freshly seeded databases, 6 of 1,114 tasks pass, under both harnesses.
 
-## Known issues in the public task set
+## Known issues in the task set
 
 | Issue | Tasks | Here |
 |---|---:|---|
-| A verifier expects a numeric string (`"1"`) from an integer column | 20 | Passable: numbers equal their numeric strings |
-| A context header key has surrounding whitespace, which upstream cannot send | 2 | Passable: the whitespace is stripped |
-| A verifier query fails (unknown column, or the table is on the task's other gym) | 3 | Unpassable; `evaluation_completed: false` |
-| A selected tool is not served by the images | 2 | The task runs without it |
-| The email image's `create_send_as_alias` fails with an internal server error for some inputs | 5 | As upstream |
+| A verifier expects a numeric string (`"1"`) from an integer column | 42 | Passable: numbers equal their numeric strings |
+| A context header key has surrounding whitespace, which upstream cannot send | 3 | Passable: the whitespace is stripped |
+| A verifier query fails, for example on an unknown column or a table of the task's other gym | 7 | Unpassable; `evaluation_completed: false` |
+| A selected tool is not served by the images | 4 | The task runs without it |
+| The task selects no tools | 1 | It gets every tool of its gym servers, as upstream |
+| The email image's `create_send_as_alias` fails with an internal server error for some inputs | 6 | As upstream |

benchmarks/enterpriseops_gym/config.yaml

@@ -26,7 +26,7 @@ enterpriseops_gym_resources_server:
           prepare_script: benchmarks/enterpriseops_gym/prepare.py
           taskset: enterpriseops_gym
           num_repeats: 3
-          license: Apache 2.0
+          license: Creative Commons Attribution-NonCommercial 4.0 International
         - name: example
           type: example
           jsonl_fpath: resources_servers/enterpriseops_gym/data/example.jsonl

benchmarks/enterpriseops_gym/manifest.yaml

@@ -4,11 +4,11 @@ experimental: true
 kind: benchmark
 integration_profile: custom-gym-agent-loop
 domain: agent
-description: ServiceNow EnterpriseOps-Gym oracle-mode tasks across eight enterprise
-  domains, run with the Stirrup agent against live MCP gym servers in per-task sandboxes
-  and scored by SQL checks on the final database state.
+description: ServiceNow EnterpriseOps-Gym, all 1,117 oracle-mode tasks across eight
+  enterprise domains, run with the Stirrup agent against live MCP gym servers in per-task
+  sandboxes and scored by SQL checks on the final database state.
 modality: text
-licensing: Apache-2.0
+licensing: Apache-2.0 AND CC-BY-NC-4.0
 authors:
 - Khushi Bhardwaj
 reward:

benchmarks/enterpriseops_gym/prepare.py

@@ -1,11 +1,11 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
-"""Prepare the EnterpriseOps-Gym benchmark JSONL: the public oracle-mode tasks of all eight domains.
+"""Prepare the EnterpriseOps-Gym benchmark JSONL: all 1,117 oracle-mode tasks of the eight domains.
 
-Downloads ServiceNow's task set and the gym servers' tool schemas, both pinned, and writes one row per task. The row's
-``responses_create_params`` holds the task's system and user prompts and its oracle tools; the other fields are what
-the ``enterpriseops_gym`` resources server reads: the gym servers to start, each with its database dump and context
-headers, the gym server of each tool, and the SQL verifiers.
+Downloads the full task set (a private NVIDIA dataset, so ``hf_token`` needs access to it) and the gym servers' tool
+schemas, both pinned, and writes one row per task. The row's ``responses_create_params`` holds the task's system and
+user prompts and its oracle tools; the other fields are what the ``enterpriseops_gym`` resources server reads: the gym
+servers to start, each with its database dump and context headers, the gym server of each tool, and the SQL verifiers.
 """
 
 from __future__ import annotations
@@ -20,10 +20,11 @@
 BENCHMARK_DIR = Path(__file__).parent
 OUTPUT_FPATH = BENCHMARK_DIR / "data" / "enterpriseops_gym_benchmark.jsonl"
 
-DATASET = "ServiceNow-AI/EnterpriseOps-Gym"
-DATASET_REVISION = "c8e538eae8a6205294f0a86675fefdc1fac408f6"  # pragma: allowlist secret
+# The full set Artificial Analysis evaluates: ServiceNow's public 649 tasks, unchanged, and 468 more.
+DATASET = "nvidia/EnterpriseOps-Gym-Full"
+DATASET_REVISION = "14e2a7e141a763bdf329bea5fc363f302af163a6"  # pragma: allowlist secret
 DOMAINS = ("calendar", "csm", "drive", "email", "hr", "hybrid", "itsm", "teams")
-EXPECTED_TASK_COUNT = 649
+EXPECTED_TASK_COUNT = 1117
 # tools/list of each gym server image, captured from the images the resources server config pins.
 TOOLS_REPO = "nvidia/NeMo-Gym-EnterpriseOps-Assets"
 TOOLS_REVISION = "8918dc64b8575d5ff476e62e1cc3687523ab59c2"  # pragma: allowlist secret
@@ -73,13 +74,16 @@ def convert_task(
 ) -> tuple[dict[str, Any], list[str]]:
     """One benchmark row, and the selected tools that no gym server of the task serves."""
     gym_configs = json.loads(row["gym_servers_config"])
-    allowed = set(row["selected_tools"]) - set(row["restricted_tools"] or [])
+    selected = set(row["selected_tools"] or [])
+    restricted = set(row["restricted_tools"] or [])
     # Upstream offers the selected tools in the gym servers' order, each from the first gym server that serves it.
+    # A task that selects none gets every tool of its gym servers, as upstream filters only by a non-empty selection.
     tool_servers: dict[str, str] = {}
     tools: list[dict[str, Any]] = []
     for gym in gym_configs:
         for tool in tools_by_gym[gym["mcp_server_name"]]:
-            if tool["name"] in allowed and tool["name"] not in tool_servers:
+            offered = (not selected or tool["name"] in selected) and tool["name"] not in restricted
+            if offered and tool["name"] not in tool_servers:
                 tool_servers[tool["name"]] = gym["mcp_server_name"]
                 tools.append(
                     {
@@ -90,7 +94,7 @@ def convert_task(
                         "strict": False,
                     }
                 )
-    missing = sorted(allowed - set(tool_servers))
+    missing = sorted(selected - restricted - set(tool_servers))
     record = {
         "task_id": row["task_id"],
         "domain": row["domain"],

resources_servers/enterpriseops_gym/README.md

@@ -11,8 +11,8 @@ server with the Stirrup agent; see its README for the benchmark and its results.
 1. **`/seed_session`** starts one sandbox per gym server of the task, from the image `gym_images` pins for it, with the
    gym's HTTP server as the sandbox entrypoint and its port declared. Once `/health` answers through
    `sandbox.endpoint(port)`, it seeds a fresh database from the task's SQL dump (`/api/seed-database`). The dumps come
-   from the upstream `gym_dbs.zip`, downloaded once and checked against `seed_archive_sha256`. The first sandbox is
-   lent to the agent (`sandbox_access`), which runs its harness there.
+   from the Hugging Face dataset `seed_dataset`, downloaded once at the pinned `seed_dataset_revision`. The first
+   sandbox is lent to the agent (`sandbox_access`), which runs its harness there.
 2. **`POST /{tool_name}`** serves the task's tools (the oracle tools of the row's `tool_servers`, nothing else). Each
    call goes to the gym that owns the tool as an MCP `tools/call`, with the task database's `x-database-id` and the
    task's context headers (the acting user's token). The gym's text answer is returned. A tool error is a 400 whose
@@ -54,9 +54,9 @@ verifier fixture (`VERIFIER_FIXTURE`, `tests/verifier_cases.jsonl`) scores the s
 built in SQLite, so it needs no gym servers. Two differences from the upstream harness, both deliberate:
 
 - **Every verifier counts.** Upstream keys results by verifier name, so the last of several same-named verifiers
-  overwrites the others (34 public tasks have same-named verifiers with different checks). Here a task passes only when
+  overwrites the others (69 tasks have same-named verifiers with different checks). Here a task passes only when
   all of its verifiers pass, as Artificial Analysis grades EnterpriseOps-Gym-AA.
-- **Numbers equal their numeric strings** in `equals`, `greater_than` and `less_than`. 20 public oracle tasks expect
+- **Numbers equal their numeric strings** in `equals`, `greater_than` and `less_than`. 42 oracle tasks expect
   `"1"` from a `COUNT(*)`, which upstream's strict equality never matches; the same tasks in the dataset's
   distractor-tool configs (`plus_5_tools`, `plus_10_tools`, `plus_15_tools`) expect `1`.
 
@@ -68,7 +68,7 @@ built in SQLite, so it needs no gym servers. Two differences from the upstream h
 | `sandbox_config` | `ttl_s`, `resources`, `env`, `metadata` and `provider_options` of every gym sandbox. |
 | `gym_images` | Per gym server name: `image`, `command` (the entrypoint that starts its HTTP server) and `port`. The defaults pin the upstream images by digest. |
 | `agent_workdir` | Working directory of the sandbox lent to the agent. |
-| `seed_archive_url`, `seed_archive_sha256` | The upstream task database dumps; cached under `seed_archive_dir`, by default `<cache_dir>/enterpriseops_gym`. |
+| `seed_dataset`, `seed_dataset_revision` | The Hugging Face dataset of task database dumps, at a pinned commit, read with the global `hf_token`. Downloaded into the Hugging Face cache at `seed_dataset_dir`, by default `<cache_dir>/enterpriseops_gym`. |
 | `gym_ready_timeout_s`, `gym_request_timeout_s` | Bound on a gym becoming healthy, and on each request to it. |
 | `verify_endpoint_tls` | Verify the TLS certificate of sandbox endpoints; off by default like the OpenSandbox provider's `tls_verify`. |
 
@@ -81,5 +81,6 @@ The server keeps sessions in its process, so it requires `num_workers: 1`.
 ## Licensing
 
 - Code: Apache 2.0.
-- EnterpriseOps-Gym (task data, gym server images, database dumps): Apache 2.0, by ServiceNow.
+- EnterpriseOps-Gym by ServiceNow: the gym server images and the public tasks and database dumps are Apache 2.0; the
+  full task set (`nvidia/EnterpriseOps-Gym-Full`) is CC BY-NC 4.0. The example rows in `data/` are public tasks.
 - Dependencies: `nemo_gym` (Apache 2.0).