[adil-a]

Problem

1. A benchmark's agent is inferred from config position. Whichever agent block declares the benchmark dataset becomes the benchmark's agent. Nothing enforces that this inference is well-defined, and a benchmark's score depends on the harness — the resolution rules deserve to be explicit and validated.
2. The inference breaks on real configs. bigcodebench's manifest pulls in the base environment config, so its merged config has two agents on the same resources server; which one "declared the dataset" is an accident of merge order.
3. gym env init still generates the old layout. New environments are born with datasets: under the agent block — every scaffold run creates more configs the migration has to fix later.

Change

1. Benchmark agent resolution becomes an explicit, validated rule set (with an optional agent: key as the disambiguator for genuinely ambiguous configs — unambiguous configs, which is all of them today, need no pin).

# Before — the agent is whichever block holds the dataset (nothing says so):
aime24_math_with_judge_simple_agent:
  responses_api_agents:
    simple_agent:
      datasets:
      - name: aime24
        type: benchmark
        jsonl_fpath: benchmarks/aime24/data/aime24_benchmark.jsonl
        num_repeats: 32

# After — the dataset lives on the resources server; the harness resolves from the unique
# agent → resources-server edge (an `agent:` key on the dataset can override when ambiguous):
aime24_math_with_judge:
  resources_servers:
    math_with_judge:
      datasets:
      - name: aime24
        type: benchmark
        jsonl_fpath: benchmarks/aime24/data/aime24_benchmark.jsonl
        num_repeats: 32

How a benchmark finds its agent now, first hit wins:

Rule	Covers
1. The dataset's agent: key	the escape hatch for ambiguous configs (none exist in-repo today)
2. The agent block that declares the dataset	every manifest that declares the dataset inside an agent block — unchanged behavior. Note this rule is unambiguous when it applies — the bug was never single-declarer inference, it was the ambiguous case below.
3. The dataset sits on a resources server with no pin, and exactly one agent's config contains resources_server: {name: <that server>} → that agent	RS-declared datasets without a pin
4. Same, but zero or 2+ agents reference that server → hard error telling you to add agent:	the bigcodebench case — previously resolved silently by merge order

2. gym env init generates the decoupled layout. Scaffolded configs declare datasets: on the resources server block; scaffolded benchmark datasets need no agent: pin (the scaffold has exactly one agent referencing the resources server); the generated data .gitignore covers the per-declaration prepared files from #2717. One exception: when scaffolding reuses a verifier from another config file, the dataset stays on the agent block (we don't know the other file's internal keys, so a partial override can't be merged safely — still fully supported).

3. Manifest validation accepts the new layout. gym env validate previously required exactly one dataset-bearing agent; it now equally accepts one dataset-bearing resources server, resolving the agent by the same rules as above.

What does NOT change

• Every existing benchmark manifest resolves to the same agent as before (rule 2 is today's behavior, unchanged).
• Full-pipeline replay over all 225 runnable configs in the repo: no observable change beyond the diffs already enumerated for feat(data): collate stamps task_source; datasets declarable on resources servers #2717.
• No repo configs are edited here — scaffolding and resolution rules only. The migration is the next PR.

Where agent_ref is headed

Same timeline as #2717: this stack removes agent_ref from data files. This PR ensures nothing new is ever generated in the old format; the next PR migrates what exists; the cleanup PR removes the field from the data contract (it remains only in run outputs, as a record of which agent actually ran each row).

Tests

90 passed across benchmark/scaffold suites. New tests pin all four resolution rules and the scaffolded config layout (datasets on the RS block, none on the agent block).

Relationship to other PRs

Part 4 of the dataset↔agent decoupling work (RFC: gym-dataset-preparation, Open Question 1), now an individual PR targeting main (#2710/#2713/#2717 merged). The repo-wide migration in #2732 builds on this PR's resolver and should merge after it.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[adil-a]

/ok to test babadb8

[adil-a]

/ok to test babadb8

[adil-a]

/ok to test d94c88e

[adil-a]

/ok to test bc44b78

[adil-a]

/ok to test 0e751be

[ananthsub]

take this config:

shared_rs:
  datasets:
    - agent: agent_b

agent_a:
  resources_server: shared_rs

agent_b:
  resources_server: some_other_rs

This line selects agent_b because the explicit pin wins without validating the relationship. Collation later writes only task_source: shared_rs, so runtime follows the actual resources-server edge and selects agent_a.

benchmark listing/preparation → agent_b
actual rollout                → agent_a

we should reject the pin unless the named agent references the declaring resources server

[adil-a]

Confirmed and fixed in fcce242. Resolution now goes through a single shared resolver (resolve_dataset_agent in global_config.py), which rejects any pin that doesn't name the declaring agent itself or an agent referencing the declaring resources server — so this config is now a hard error at discovery, preparation, validation, and dispatch alike. Test: test_rs_declared_pin_must_reference_the_declaring_rs.

[ananthsub]

Suppose two benchmark declarations both select shared_agent:

benchmark_a → shared_agent
benchmark_b → shared_agent

The assignments become:

benchmarks_dict["shared_agent"] = benchmark_a
benchmarks_dict["shared_agent"] = benchmark_b

where benchmark_b overrides benchmark_a silently

[adil-a]

Confirmed and fixed in fcce242. benchmarks_dict is now keyed by the declaring instance (unique by construction; each instance is limited to one benchmark dataset) instead of the resolved agent, so both declarations are prepared. Test: test_two_declarations_resolving_to_one_agent_both_prepare.

[ananthsub]

Consider this configuration:

shared_rs:
  datasets:
    - agent: agent_b

agent_a:
  resources_server: shared_rs

agent_b:
  resources_server: shared_rs

Discovery honors the pin and selects agent_b. Collation then emits only:

{"task_source": "shared_rs"}

At rollout time, resolve_task_sources() asks which agents reference shared_rs. It finds both agent_a and agent_b, no longer has the pin, and rejects the row as ambiguous.

We should propagate the pin into the runtime routing, equivalent to:

agent_map:
  shared_rs: agent_b

along with the discovery -> collate -> rollout test for this case

[adil-a]

Confirmed and fixed in fcce242. Rather than materializing an implicit agent_map, resolve_task_sources() now resolves through the same shared resolver, reading the agent: pin back off the declaring instance in the merged config — so this config routes to agent_b end to end (an explicit +agent_map still overrides). Conflicting pins across one instance's datasets are a hard error, since rows carry only task_source. Discovery → collate → rollout test added: test_pin_survives_discovery_collate_and_rollout (real collate stamping, real dispatch resolution).

[adil-a]

/ok to test

[copy-pr-bot]

/ok to test

@adil-a, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[adil-a]

/ok to test bf8f5fa

[adil-a]

/ok to test 151f1ed

[adil-a]

@marta-sd closing the loop on this one, late. You were right on both counts.

bigcodebench's two-agent closure is an artifact: its benchmark config defines its own agent instead of _inherit_from-ing the base agent like the other benchmarks do. There is no real use for two agents there.

So the pin is going away: #3209 removes the agent: key entirely. An ambiguous config (several agents on one resources server, dataset on the server) is a load-time error that points to agent_map, fan_out, or declaring the dataset on the agent. A leftover agent: key is also an error, not silently ignored. Nothing in the repo used the key — #2732 already dropped every pin, and the scaffold never emitted one — so behavior is unchanged for every config on main.

The bigcodebench _inherit_from cleanup is a small separate follow-up.