Skip to content

feat(enterpriseops): add external MCP benchmark integration #4944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
d-molinari wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(enterpriseops): add external MCP benchmark integration #4944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/config/.secrets.baseline
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,8 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Deepsearchqa | knowledge | Grade DeepSearchQA single-answer and set-answer responses | Evaluate multi-step web research answers | - | - | - | <a href='resources_servers/deepsearchqa/configs/deepsearchqa.yaml'>deepsearchqa.yaml</a> | - |
| Denovo Swe | | - | - | - | - | - | <a href='resources_servers/denovo_swe/configs/denovo_swe_example.yaml'>denovo_swe_example.yaml</a> | - |
| Denovo Swe | | - | - | ✓ | - | Creative Commons Attribution 4.0 International | <a href='resources_servers/denovo_swe/configs/denovo_swe_opencode.yaml'>denovo_swe_opencode.yaml</a> | - |
| Enterpriseops Gym | agent | EnterpriseOps-Gym benchmark with per-turn telemetry logging (timings, token usage incl. cache hits, tool latencies) | Improve stateful multi-step enterprise planning and tool use | - | - | - | <a href='resources_servers/enterpriseops_gym/configs/enterpriseops_gym_turnlog.yaml'>enterpriseops_gym_turnlog.yaml</a> | - |
| Enterpriseops Gym | agent | EnterpriseOps-Gym enterprise tool-use benchmark (external MCP gym servers + SQL/judge verifiers) | Improve stateful multi-step enterprise planning and tool use | - | - | - | <a href='resources_servers/enterpriseops_gym/configs/enterpriseops_gym.yaml'>enterpriseops_gym.yaml</a> | - |
| Equivalence Llm Judge | agent | Short bash command generation questions with LLM-as-a-judge | Improve foundational bash and IF capabilities | ✓ | ✓ | GNU General Public License v3.0 | <a href='resources_servers/equivalence_llm_judge/configs/nl2bash-equivalency.yaml'>nl2bash-equivalency.yaml</a> | - |
| Equivalence Llm Judge | knowledge | Short answer questions with LLM-as-a-judge | Improve knowledge-related benchmarks like GPQA / HLE | - | - | - | <a href='resources_servers/equivalence_llm_judge/configs/equivalence_llm_judge.yaml'>equivalence_llm_judge.yaml</a> | - |
| Equivalence Rule | knowledge | Question - Answering with rule-based reward | Improve retrieval and counting capabilities | - | - | - | <a href='resources_servers/equivalence_rule/configs/lc.yaml'>lc.yaml</a> | - |
Expand Down
37 changes: 37 additions & 0 deletions benchmarks/enterpriseops/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# EnterpriseOps-Gym Benchmark

Pinned oracle-mode public split of [ServiceNow EnterpriseOps-Gym](https://github.com/ServiceNow/EnterpriseOps-Gym)
(Apache 2.0): 649 stateful enterprise tool-use tasks across 8 domains (Calendar, CSM, Drive,
Email, HR, ITSM, Teams, Hybrid), graded by SQL verifiers on final database state. See
`resources_servers/enterpriseops_gym/README.md` for architecture and parity notes.

## Prerequisites

1. The seven pinned MCP gym containers running and the pinned EOG checkout with
`gym_dbs.zip` unzipped (`seed_sql_root` config). Exact pins and commands are in
`resources_servers/enterpriseops_gym/RUNTIME.md`.
2. Hub egress at prepare time (`huggingface.co`). Preparation pins dataset revision
`c8e538eae8a6205294f0a86675fefdc1fac408f6`, validates exact per-domain counts, and
atomically writes the combined file. `NEMO_GYM_EOG_LOCAL_TASKS` is accepted only when
it points to a complete eight-domain fallback, so a partial canonical file cannot be
produced silently.

## Usage

```bash
gym eval prepare --benchmark enterpriseops

gym env start --benchmark enterpriseops --model-type openai_model \
--model-url <OpenAI-compatible base URL> --model-api-key <API key> --model <model> \
"++enterpriseops_benchmark_resources_server.resources_servers.enterpriseops_gym.seed_sql_root=<abs path to EOG checkout>"

gym eval run --no-serve --benchmark enterpriseops --model-type openai_model \
--model-url <OpenAI-compatible base URL> --model-api-key <API key> --model <model> \
--agent enterpriseops_benchmark_simple_agent \
--input benchmarks/enterpriseops/data/enterpriseops_oracle_benchmark.jsonl \
--output results/enterpriseops_oracle.jsonl
```

Scoring: `reward` = EOG leaderboard parity (all name-collapsed verifiers pass). Aggregate
metrics include per-domain `{domain}/success_rate`, `{domain}/verifier_pass_rate`, and
`macro_success_rate` (mean over domains, the leaderboard "Avg").
2 changes: 2 additions & 0 deletions benchmarks/enterpriseops/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
24 changes: 24 additions & 0 deletions benchmarks/enterpriseops/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Chain to the EnterpriseOps-Gym resources server + agent config
config_paths:
- resources_servers/enterpriseops_gym/configs/enterpriseops_gym.yaml

# `_inherit_from` keeps this benchmark config isolated from the generic config above.
enterpriseops_benchmark_resources_server:
_inherit_from: enterpriseops_gym

enterpriseops_benchmark_simple_agent:
_inherit_from: enterpriseops_gym_simple_agent
responses_api_agents:
simple_agent:
resources_server:
name: enterpriseops_benchmark_resources_server
datasets:
- name: enterpriseops
type: benchmark
jsonl_fpath: benchmarks/enterpriseops/data/enterpriseops_oracle_benchmark.jsonl
prepare_script: benchmarks/enterpriseops/prepare.py
# Rows are fully materialized by the converter (system+user input, tools baked in),
# so no prompt templating applies.
prompt_config: null
# EOG leaderboard reports single-run task success; raise for mean@k.
num_repeats: 1
1 change: 1 addition & 0 deletions benchmarks/enterpriseops/data/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.jsonl
Loading
Loading