[mcuevas-nvidia]

Summary

This PR integrates ServiceNow's EnterpriseOps-Gym (EOG) into NeMo Gym: a 649-task benchmark of stateful, multi-step enterprise tool use across 8 domains (calendar, CSM, drive, email, HR, ITSM, teams, and cross-domain hybrid), where an agent operates 512 tools against live MCP servers backed by SQL databases and is scored on final database state. The integration is eval-complete and RL-ready (token-ID capture, fractional reward mode, and a validated GRPO rollout-collection recipe), with no changes to NeMo Gym core.

What's included (46 files)

Component	Path	What it does
Resources server	resources_servers/enterpriseops_gym/	Seeds a per-session database on the external EOG MCP containers, proxies tool calls (catch-all /{tool_name} route, pooled aiohttp, per-session x-database-id), runs verifiers concurrently, cleans up idempotently (TTL janitor + delete-on-verify). Supports replica pools (gym_url_pools), per-domain metrics, per-tool latency capture, and a strict_verifiers fractional-reward mode for RL.
Verifier engine	.../verifier_engine.py	Line-for-line port of EOG's scoring semantics, pinned by golden fixtures generated from the original implementation (33 cases, byte-identical).
Task converter	.../convert_tasks.py + benchmarks/enterpriseops/	Converts the public HF dataset (ServiceNow-AI/EnterpriseOps-Gym) to Responses-API task rows at prepare.py time; benchmark registered with prompt_config: null (pre-baked rows). Dataset files are not committed.
Per-turn telemetry agent	responses_api_agents/turn_logging_agent/	SimpleAgent subclass with an identical loop that records per-turn timestamps, durations, input/output/cached/reasoning tokens, and tool names, attaching turns to the verify response. Generic — not EOG-specific.
Tests	44 total (42 + 2)	Fully offline: a sqlite-backed stub MCP gym (tests/stub_gym.py) plus golden parity fixtures. No containers or network needed to run CI.
Docs	PARITY.md, PERF.md, RLPILOT.md	Full parity evidence, throughput study, and RL rollout-collection pilot (methodology + measured numbers below).

Verification logic

Each task carries verifier_metadata with a list of verifiers. database_state verifiers
run a SQL query against the session's final database state (via the gym containers'
/api/sql-runner), extract a value, and compare it to the expected value using EOG's
comparison semantics; response_check verifiers score the agent's final message with an
LLM judge (defaults to the policy model, temperature pinned to 0.0, matching EOG). The
public oracle split is 100% database_state (3,496/3,496 verifiers), so scoring there is
fully deterministic given a final DB state. Reward = 1.0 iff all (name-collapsed) verifiers
pass, matching the upstream leaderboard; strict_verifiers: true switches to
every-verifier-counts and a fractional strict_pass_rate for RL shaping.

Scoring fidelity (PARITY.md)

The port is validated bug-for-bug against the upstream harness, preserving its quirks (verifier name-collapse, unknown-gym skips, loose comparison semantics) for leaderboard comparability:

• Unit level: golden fixtures generated by running the original EOG engine — byte-identical extraction/comparison behavior.
• Task level: 12/12 identical outcomes and verifier structures on live containers.
• Full split (649 tasks): port 16.4% vs native 16.8% macro (gpt-4.1-mini, temp 0); per-task agreement 90.4%, disagreements symmetric (McNemar exact p = 0.90).
• k=5 × both harnesses (6,480 rollouts): Δ −0.23 ± 0.7 pp, per-task preference exactly 67:67.
• Cross-stack replication (k=5 on dedicated vLLM, Nemotron 3 Nano): formally equivalent within ±2 pp (TOST, α = 0.05).

Performance (PERF.md)

Scale-tested end-to-end at five client concurrencies (c = 8, 16, 32, 64, 128) with a full
649-task pass per level per harness (10 passes, 4×H100 vLLM, identical endpoint): the port
completed 649/649 tasks at every level with zero retries, with success rates flat
across levels. At matched concurrency the port is 1.14–1.71× faster (largest at low
concurrency, nearest the native harness's documented defaults). The gap narrows by design:
both harnesses converge toward the same GPU throughput floor — and reaching it is the key
result. The port saturates the hardware at c=64; the native harness never reaches the
floor in the tested range and needs ~4× the client concurrency for equal throughput.
Net cost: a full-split eval is 1h27m of 4×H100 time (port) vs 4h40m at native's documented
settings — 3.2× GPU-hours. Mechanism: pooled connections, persistent MCP sessions, and
concurrent verifiers keep vLLM's continuous batch fed. (Since the gap is client dead time
relative to GPU service time, it is expected to widen on faster serving hardware, where
saturation demands even more effective concurrency.)

RL readiness (RLPILOT.md)

A config-only pilot (zero code changes) validated GRPO rollout collection end-to-end: 100% token-ID/logprob coverage via return_token_id_information, 15/20 task groups with mixed binary reward at k=8 (mean within-group std 0.341) on a curriculum selected from repeat-run data, plus a measured deployment-sizing guide (sequence-length distribution, memory budget, recommended node shapes).

How to run

# Start the stack (external EOG MCP containers must be running; see resources server README)
ng_run "+config_paths=[resources_servers/enterpriseops_gym/configs/enterpriseops_gym.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  "++enterpriseops_gym.resources_servers.enterpriseops_gym.seed_sql_root=/path/to/EnterpriseOps-Gym"

# Prepare the benchmark data (downloads the public HF dataset) and collect
ng_prepare_benchmark +benchmark_name=enterpriseops
ng_collect_rollouts +agent_name=enterpriseops_gym_simple_agent \
  +input_jsonl_fpath=benchmarks/enterpriseops/data/enterpriseops_oracle_benchmark.jsonl \
  +output_jsonl_fpath=results/enterpriseops.jsonl \
  +responses_create_params.temperature=0.0 +responses_create_params.max_output_tokens=16384

# Tests (offline, no containers needed)
gym env test --resources-server enterpriseops_gym
ng_test +entrypoint=responses_api_agents/turn_logging_agent

Validation on this exact branch state

• 42/42 + 2/2 tests green after rebase onto current main
• ruff check + ruff format --check clean; README environment table regenerated via scripts/update_env_list.py
• Config stack boots via ng_run dry run
• Live smoke on real MCP containers + gateway model: 5/5 rollouts completed across both agents, all verifiers scored, turn telemetry attached
• All commits DCO signed-off

Contribution-guide compliance

Mapped to the environment /
benchmark guides:

• Required files — all present: app.py, configs/*.yaml (valid domain: agent),
  tests/test_app.py (42 tests), data/example.jsonl (5 tasks),
  data/example_rollouts.jsonl (5 pre-generated rollouts against live containers; note
  these samples are CSM-domain, the benchmark's hardest — near-zero rewards on them are
  expected and consistent with the full-split CSM rate of ~4%), requirements.txt,
  README.md with licensing information.
• Reward profiling — run on a closed model (gpt-4.1-mini: 16.4% macro) and an open
  thinking model (Nemotron 3 Nano, reasoning on: 22–25%), i.e. the instruct+thinking
  mixture the guide asks for. Scores sit inside the official leaderboard's published range
  (Qwen3-4B 13.6% … GPT-5-Mini 22.0%) with a coherent domain pattern (email easiest, CSM
  hardest, matching the leaderboard). This benchmark is legitimately hard — no public model
  reaches 30%.
• Variance — k=5 repeat runs on both stacks; mean@5 resolves ±0.7 pp (< 1% as required).
  Calibration guidance for users is included in PARITY.md.
• Failure-case analysis — performed extensively across parity, perf, and RL-pilot runs
  (documented in the three reports).
• Original-repo reproduction — instead of reproducing a leaderboard model's published
  number (leaderboard models weren't available on our serving), we ran the original EOG
  harness side-by-side on identical models, containers, and endpoints: per-task agreement
  with symmetric disagreements (McNemar p = 0.90) and formal TOST equivalence within ±2 pp.
  This isolates harness fidelity even more directly; happy to additionally run a listed
  leaderboard model if reviewers want the published-number check.
• Stacked PRs — per the development-setup guide, happy to restack this as layers
  (resources server + tests → benchmark registration → turn-logging agent → reports) if
  reviewers prefer; presented as one PR first since the layers are tightly coupled by the
  parity evidence.

Design decisions & notes for reviewers

• Hand-rolled MCP client instead of the official SDK (rationale in mcp_client.py docstring): the SDK's transport is httpx-based (banned for async here), EOG needs session-level and per-call x-database-id headers, and half the surface is non-MCP REST (/api/seed-database etc.).
• Upstream EOG quirks intentionally preserved and documented (PARITY.md §1) rather than fixed, to keep leaderboard comparability; strict_verifiers: true opts into every-verifier-counts scoring for RL.
• turn_logging_agent is separable — it's a general-purpose agent; happy to split it into its own PR if preferred.
• verified: false per convention for new resources servers.
• External dependency: the benchmark requires the EOG Docker containers (from the upstream EOG repo) at eval time; unit tests do not.
• Operational findings that affect large runs (MCP container fd leak ~5/task, upstream runner's silent task drops) are documented in PARITY.md §6; we plan to file these upstream against EOG.

Data provenance & licensing

EnterpriseOps-Gym is Apache 2.0 (code) with a public HF dataset (ServiceNow-AI/EnterpriseOps-Gym). This PR commits only: tool-schema snapshots captured from the public EOG containers (7 JSON files, 512 tools), 13 sample tasks derived from the EOG repo's task files, 5 example rollouts generated against live containers, one synthetic hybrid task hand-authored for tests (written with LLM assistance against live container schemas — disclosed per the synthetic-data guideline), and golden verifier fixtures generated by running the EOG engine. The full benchmark dataset is downloaded at prepare.py time and gitignored.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

can these long files be pulled from somewhere like huggingface instead of committed?

[cmunley1]

/claude review

[claude]

SHIP WITH CARE — no BLOCKERs. The verifier port and reward aggregation are the highest-risk surface here and they're the most carefully done part of the PR: EOG parity quirks (name-collapse, unknown-gym skips, loose comparisons) are documented and pinned by golden fixtures, strict_verifiers cleanly separates leaderboard-parity reward from RL reward, verify is idempotent and deletes DBs in a finally, the MCP client correctly routes all async HTTP through nemo_gym.server_utils.request() (no httpx, no ray.get()), and per-call x-database-id isolation is the right call over the SDK's session-level headers. No core changes, tests are fully offline, deps declared. Good work.

Two non-blocking findings, both inline:

1. RISK (turn_logging_agent/app.py): the agent hardcodes bare /v1/responses where simple_agent uses url_path_for_request/url_path_for_run, dropping the /ng-rollout/<id> capture-correlation prefix. No-op when observability is off, so eval/reward and inline return_token_id_information are unaffected — but this agent's whole purpose is RL telemetry, which is where capture gets enabled. The docstring even claims step-for-step parity; this is the divergence.

2. RISK (app.py verify): all([]) is True, so a fully empty or all-skipped verifier set scores reward 1.0. A gym_name/pool misconfig silently converts every task to a false pass — inflates eval macro-rates and feeds spurious positive reward into GRPO. Guard the strict path (bool(strict_passed) and all(...)) and at least log.warning on num_verifiers_scored == 0.

Neither blocks merge; #2 is the one I'd want resolved before this feeds a training run.

[mcuevas-nvidia]

Good question. Quick context on what these are: the data/tools/*.json files are snapshots of the tool schemas advertised by EnterpriseOps-Gym's MCP containers (512 tools across 7 domains). They can't come from HuggingFace as-is because the upstream HF dataset only contains tasks; the schemas exist only inside the running Docker containers, and they're needed at prepare.py time to build task rows. Pinned copies are also what guarantees byte-level prompt parity with the upstream harness (see PARITY.md). I agree they're bulky. Two options: 1. Host them on HF (a small companion dataset, or the existing dataset repo if ServiceNow will take them, which would need upstreaming) and have prepare.py download them, keeping only a checksum in-repo. Downside: prepare gains a network dependency for data that changes only when upstream ships new containers. 2. Keep them committed as the reproducibility anchor. They're versioned exactly like golden test fixtures. My mild preference is (2) for reproducibility, but (1) works if repo size is the priority. From: Christian Munley ***@***.***> Date: Monday, July 27, 2026 at 10:13 AM To: NVIDIA-NeMo/Gym ***@***.***> Cc: Marc Cuevas ***@***.***>; Author ***@***.***> Subject: Re: [NVIDIA-NeMo/Gym] Add EnterpriseOps-Gym benchmark: resources server, benchmark registration, and per-turn telemetry agent (PR #2142) @cmunley1 commented on this pull request.

________________________________ In resources_servers/enterpriseops_gym/data/tools/calendar.json<#2142 (comment)>:

@@ -0,0 +1,3436 @@

+{ can these long files be pulled from somewhere like huggingface instead of committed? — Reply to this email directly, view it on GitHub<#2142?email_source=notifications&email_token=B55WDQREOH2VD5KOXZ7ALYD5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#pullrequestreview-4789624130>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/B55WDQVTNXPZ77HYPHRFJJD5G6ERPAVCNFSNUABGKJSXA33TNF2G64TZHMYTANBUGU2TIOBUGA5US43TOVSTWNBZG4ZTENRWHEYTTILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/B55WDQVOGQBHPOBD2ZQZR735G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/B55WDQR4JTS3JH3KU26HLPL5G6ERPA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTINZYHE3DENBRGMYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.

[zyzhou5]

/ok to test a25f0e4

[cmunley1]

This forks a July SimpleAgent and misses later fixes on main: a reasoning-only turn doesn't end the loop here, so it runs to max_steps (#3536). SimpleAgent's trajectory capture already records per-turn and per-tool timing. If cached/reasoning token sums are missing, can we add them there and drop this agent.

[cmunley1]

This replaces the cookie jar instead of merging it (#4449).

[cmunley1]

I would suggest to make core provider change its own PR. endpoint() also returns loopback for any port without checking it was declared.

[cmunley1]

Rejecting --net/--netns-path whenever ports are declared means services can never be network-isolated. We block egress by default, so this probably needs another way e.g. a netns with port forwarding.

[cmunley1]

Images come from a personal Docker Hub account. Please mirror them to an org-owned registry (keeping the digests).

[cmunley1]

Would suggest using something other than the policy as the judge

[cmunley1]

The proxy doesn't enforce restricted_tools / selected_tools, so a model can call any tool the gym exposes.

[cmunley1]

If one gym's seed fails, are the databases already seeded for the other never cleaned up?

[d-molinari]

Some potentially useful cleanup hardening to be considered:

#4774