[kbhardwaj-nvidia]

What does this PR do?

Adds ServiceNow's EnterpriseOps-Gym as a benchmark. It runs the way Artificial Analysis runs EnterpriseOps-Gym-AA:

• the Stirrup harness, offering each task's oracle tools only;
• a 100-turn cap and 3 repeats;
• strict pass@1 as the headline metric, with the verifier pass rate reported alongside;
• all eight domains.

The gym servers run in per-task sandboxes from Gym's sandbox provider (OpenSandbox), the same way SWE-bench and GDPVal run.

Part of #909.

Related open work:

• Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent #2142 runs operator-managed containers or Apptainer with simple_agent and a turn-logging agent, and keeps upstream's name-collapsed scoring.
• feat(enterpriseops): add external MCP benchmark integration #4944 targets externally hosted MCP services.
• fix(enterpriseops): retain databases until cleanup succeeds #4774 targets Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent #2142's branch.

#2142 and #4944 both also add resources_servers/enterpriseops_gym, so maintainers will need to choose the layout.

resources_servers/enterpriseops_gym (new)

• /seed_session:
  • Starts one sandbox per gym server of the task, from the upstream images pinned by digest.
  • Waits for each server's /health, then seeds a fresh database from the task's dump in upstream's gym_dbs.zip, which is pinned by sha256.
  • Lends the first sandbox to the agent.
• POST /{tool_name}:
  • Forwards each call to the gym that owns the tool, with the task database's ID and the task's context headers, mapped as upstream maps them.
  • A tool error is a 400 that the model sees.
  • An unreachable gym is a 502, which fails the episode so the environment server retries it.
  • A call that reached a gym is never resent.
• /verify:
  • Runs the task's SQL verifiers one at a time, as upstream does. Reward is 1 only when every verifier passes; the response also reports verifier_pass_rate and per-verifier expected and actual values.
  • A query the database rejects fails its verifier. The response then sets evaluation_completed: false and failure_kind: verifier_error, so the row is not read as a hard task.
  • An unreachable gym fails verification for a retry rather than scoring 0.
• Session contract (following the updated add-benchmark rules):
  • num_workers: 1 is enforced and sessions are locked per session.
  • Seeding is idempotent, and a session cannot be re-seeded after close.
  • Every started sandbox is stopped if seeding fails, at close and at shutdown.
• task_data.py defines the row schema. VERIFIER_FIXTURE scores final database states built in SQLite.
• Metrics: pass@1[avg-of-k] of success and of verifier pass rate, overall and per domain.

benchmarks/enterpriseops_gym (new)

• prepare.py writes the 649 public oracle tasks from ServiceNow-AI/EnterpriseOps-Gym@c8e538e. Tool schemas come from the images' tools/list, snapshotted in nvidia/NeMo-Gym-EnterpriseOps-Assets@8918dc6.
• The config pairs the server with stirrup_agent through a single_agent_turn_legacy environment server (taskset enterpriseops_gym).
• Includes the manifest and a README with the measurement contract, results and validation.

responses_api_agents/stirrup_agent

• New code_exec option, default true, which keeps current behavior. The benchmark sets it to false, so the model gets only the task's tools and Stirrup's finish.

Deliberate differences from the upstream harness (documented in the server README):

• Every verifier counts. Upstream keys results by verifier name, so same-named verifiers overwrite each other; 34 public tasks have them. Counting all of them matches AA's strict grading.
• Numbers equal their numeric strings in equals, greater_than and less_than. 20 public tasks expect "1" from a COUNT(*); the same tasks in the dataset's plus_*_tools configs expect 1.
• Header whitespace is stripped. Two tasks have context header keys with surrounding whitespace, which upstream cannot send at all.

Validation

Run against the final diff unless noted:

• pre-commit run --all-files: pass.
• gym env test +entrypoint=resources_servers/enterpriseops_gym +should_validate_data=true: 71 passed; example data validated.
• gym env test +entrypoint=responses_api_agents/stirrup_agent +should_validate_data=true: 169 passed.
  • test_an_agent_that_runs_out_of_wall_time_answers_with_its_partial_run_as_incomplete has a 4 s wall-time budget. It failed in 3 of 5 local suite runs on a heavily loaded shared login node (load average around 80); it passes in CI (5.3 s) and when rerun alone.
  • That test uses the default code_exec: true path, which this PR does not change.
• gym env validate enterpriseops_gym --kind benchmark --sync: pass, with nothing to sync. Datasets: 649 benchmark rows and 5 example rows.
• gym dataset collate "+config_paths=[benchmarks/enterpriseops_gym/config.yaml]" +mode=example_validation: produced data/example_metrics.json. data/example_rollouts.jsonl holds the 5 example tasks run through Stirrup with GLM-5.3-Flash; 3 pass.

Exit checks (add-benchmark Step 7), on the results run below:

Check	Result
Health	quality_summary.json: 1,947 healthy, 0 unhealthy, 0 unobserved; 0 failed rollouts; 0 sandboxes left behind
Evidence	scripts/inspect_harness_conformance.py --bundle rollouts.jsonl --capture-dir model-calls (gym-p0/v1): TE-1 to TE-9 fulfilled for 1,947/1,947 rollouts, exit 0
Score	See Rollouts: on the six domains AA's GLM numbers are comparable on, 39.2% here vs 39.2% at AA
Config	Local gym eval run does not write resolved-config.yaml (only the gym eval submit executors do). The exact command is below.

Parity with the upstream harness, in the spirit of #2142's PARITY.md. The public set has no reference solutions, so 100% oracle runs are not possible; the checks compare against upstream's own code instead.

• Verifier goldens. tests/verifier_golden.json comes from running upstream's VerifierEngine: 12 kinds of /api/sql-runner answer and 660 combinations of actual value, expected value and comparison. The server matches all of them except the 21 that compare a number with a numeric string, which is the intended difference.
• Replay. The first repeat of every task in the results run was replayed into fresh gym servers using only upstream code (create_database_from_file, MCPClient, VerifierEngine), with the recorded tool calls remapped to newly generated IDs.
  • 647 of 649 tasks replay. Upstream's httpx rejects the two whitespace header names.
  • 7,304 of 7,316 tool outputs (99.8%) are identical after normalizing generated IDs and timestamps. The other 12 differ only in generated values: random IDs and the order they sort in, PDF bytes, and random suffixes.
  • Verifiers: 3,420 of 3,486 agree. 65 differ by the numeric-string rule. One differs because its query compares against datetime('now') and the replay ran six hours later.
  • Task outcomes: 647/647 agree counting the numeric-string rule, and 639/647 under upstream's strict equality.
• Do-nothing control. Scoring the freshly seeded databases passes 1 of 647 tasks under both harnesses.

Rollouts

Results run

• Model: GLM-5.3-Flash (nvidia/zai-org/glm-5.3-flash), temperature 1.0 (the lab's recommendation, set on the agent), top_p 0.95.
• 649 tasks × 3 repeats on OpenSandbox at concurrency 64, in 44 minutes, with model-call capture on.
• Command: gym eval run --benchmark enterpriseops_gym --model-type openai_model --model nvidia/zai-org/glm-5.3-flash --model-url https://inference-api.nvidia.com/v1 --model-api-key <key> --split benchmark --concurrency 64 --output <run>/rollouts.jsonl ++enterpriseops_gym_stirrup_agent.responses_api_agents.stirrup_agent.temperature=1.0 ++observability_enabled=true ++model_call_capture_dir=<run>/model-calls.
• The code was this branch. The final head differs from it only in documentation, the datasets' license field and the committed example artifacts.

Domain	Success here	EnterpriseOps-Gym-AA
CSM	30.7	47.3
HR	31.4	34.8
ITSM	37.2	38.1
Email	60.7	37.5
Calendar	36.1	5.8
Teams	51.4	7.0
Drive	54.2	58.1
Hybrid	36.4	24.2
Overall	40.5	33.2

Run details

• Verifier pass rate was 74.2%. Success per repeat was 40.4, 39.8 and 41.3.
• Two earlier full runs, on earlier revisions of this branch with the same settings, scored 41.0 and 41.0. gym eval compare against the last of them gives Δ −0.5 pt, with a 95% CI of [−3.1, +2.1] pt on mean reward.
• Every rollout ended with Stirrup's finish. Rollouts averaged 7.6 model calls, 12.2 tool calls and 2.3k output tokens (1.1k reasoning).
• Exactly 9 rollouts have evaluation_completed: false: the 3 tasks whose verifier SQL is broken, times 3 repeats.

Comparison with AA

• AA scores 1,117 tasks, of which these 649 public tasks are about 60%, so the overall numbers are not directly comparable.
• Weighting the six domains other than Calendar and Teams by the per-domain task counts that AA's scores imply gives 39.2% here and 39.2% at AA.
• On Calendar and Teams, AA reports identical scores (5.8% and 7.0%) for GLM-5.3 and GLM-5.3-Flash, while its other models solve 36–52% there. That floor does not appear here.
• AA reports 11.4 turns and 13.8k output tokens per task for this model.

Compatibility and benchmark impact

• New benchmark and resources server only.
• stirrup_agent: the new code_exec option defaults to true, so existing configs such as GDPVal are unchanged.
• The resources-server config keeps verified: false. The only published reference for this model is AA's 1,117-task score, and this run used GLM-5.3-Flash rather than a frontier model, so baselining is left to maintainers.
• The upstream gym images are published on Docker Hub by a personal account, for amd64 only. They are pinned by digest here.
  • Follow-up: mirror them to an organization registry and map the names with the provider's images.rewrites.
• Known issues in the public task set are listed in the benchmark README:
  • 20 numeric-string verifiers;
  • 2 whitespace headers;
  • 3 tasks with failing verifier SQL;
  • 2 tasks selecting tools no image serves;
  • 5 tasks hitting the email image's create_send_as_alias internal error.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5000-6ead18ef36f4.docs.buildwithfern.com/nemo/gym

[cmunley1]

tested a rollout, worked smoothly!

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40048740820

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40050482626

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40054852036

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40114833043

[cmunley1]

/ok to test d146e81

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40210724713