[ananthsub]

What changed and why

Stacked on #4790. This is the last of three changes that let every resources, agent and environment server either run with num_workers > 1 or refuse it at startup with a reason. #4884, now merged, added the refusal method, and #4790 gave every server that is safe with several workers its module-level app.

The servers below keep state that session routing cannot reach, or own a resource that one process must hold. Each now overrides multiple_workers_unsupported_reason(), and run_webserver refuses num_workers > 1 with that reason before uvicorn starts, and before a dry run returns.

Server	Refuses	Reason
aviary (all five entrypoints)	always	Environments are kept in memory by the env_id in the request body, and the aviary agent sends step, verify and close without the session cookie.
toolsandbox	always	Episodes are kept in memory by the env_id in the request body, and toolsandbox_agent sends step, close and verify without the session cookie.
critpt	always	verify calls are pooled in memory into batches submitted together; the batches, submission counters and API key rotation must be shared by every request.
speed_bench	with metrics_source: prometheus	The speculative decoding window starts from a metrics snapshot in memory, so each worker would measure from a different start.
osworld	always	Docker host capacity, startup cleanup of orphaned containers and the sessions.json state file are each owned by one process.
longmt_eval	always	Each worker builds its own Ray GPU actor pool, so several workers multiply the GPU actors and can fail to schedule them.
wmt_translation	with compute_comet	Each worker builds its own Ray GPU actor pool for COMET, so several workers multiply the GPU actors and can fail to schedule them.
gdpval	always	A seed claims the persisted deliverables directory of its task and repeat in memory, keyed by the task ID and rollout ID in the request body, so sessions on different workers could write the same directory.
terminal_bench_4	always	Seeds are deduplicated in memory by the client_session_id and rollout_id in the request body, which session routing does not route.
vlm_eval_kit	always	MMBench scoring waits in memory for every row of a group, keyed by the group in the request body, so a group split across workers never completes.
Legal Agent Bench Harbor bridge	always	Its entrypoint lives outside the harbor_agent directory, so the module path uvicorn workers import does not resolve.
voicebench	always	Legacy scoring draws every rule-scored row from one random stream shared by the whole server.
enterpriseops_gym	always	EnterpriseOps-Gym sessions are process-local.
partner_if mock tools	always	Mock-tool episodes live in the server's memory.
proof_judge	with zero_reward_incorrect_groups	It waits in memory for every rollout of a problem, keyed by the problem text.
genrm_compare	with cohort verification (fixed_baseline or num_rollouts_per_prompt > 1)	It collects a prompt's rollouts in memory.
ns_tools	with the HTTP PythonTool	Each worker starts a sidecar on the fixed python_tool_port.
terminus_2_sandboxed_agent	without a resources_server	Its episode sessions are process-local.
agentic_if	always	Its sessions hold their sandbox connections in one process.

speed_bench, wmt_translation, proof_judge, genrm_compare and ns_tools get the module-level app too, because they are safe in their other configurations: speed_bench's default response metrics source scores each request on its own, and wmt_translation without COMET, as the IN22 profile runs it, keeps nothing between requests. proof_judge's debug log takes a file lock around each append, as proof_genselect and proof_verification do since #4790, and still writes the record on a filesystem without file locks.

terminal_bench_4, osworld, genrm_compare, gdpval, voicebench, enterpriseops_gym, agentic_if, the partner_if mock tools and terminus_2_sandboxed_agent already refused several workers, in model_post_init, in a config validator or with num_workers: Literal[1]. Those checks now go through the method with the same conditions, so they all fail the same way and a dry run reports them. A review of the current code suggests that enterpriseops_gym, the partner_if mock tools and terminus_2_sandboxed_agent keep their per-session state where routing reaches it, so their owners may be able to lift these refusals; this pull request keeps their behavior.

run_webserver asks for the reason after it builds the server, and building a server runs model_post_init. gdpval, voicebench, the partner_if mock tools and speed_bench do setup there: installing LibreOffice, downloading NLTK data, loading a tool backend, or taking a metrics snapshot. So each also checks the refusal at the start of model_post_init, which keeps the fail-fast behavior their own checks had: a server with several workers fails on the worker count, not on that setup.

A unit test reads every server entrypoint without importing it: each server directory's app.py and each module a config names, anywhere in the repository. It fails when a server has neither the module-level app nor an override of the method, on its class or on a parent server class in the repository, and the message says which to add. The only exemptions are the Gymnasium base class module and the local SEC search library, which configs name but which start no server. The resources server guide now mentions the test.

Issue

No separate issue: this follows #3905 and completes its multi-worker support for every resources, agent and environment server.

Validation

• pytest -n 16 tests/unit_tests in a virtual environment synced with the extras the unit-test workflow installs (uv sync --extra dev --extra ray --extra sandbox --extra openshell --extra mlflow --extra wandb), on main at 390c954. Every failure also fails on main at the same commit in that environment, apart from one docker_pty test that fails under parallel load and passes on its own.
• The repository-wide test fails on the parent commit, naming exactly the servers this change makes refuse, and passes here.
• Each changed server's own tests, in the server's own virtual environment (the one gym env test creates): aviary, toolsandbox, critpt, speed_bench, longmt_eval, wmt_translation, osworld, gdpval, terminal_bench_4, vlm_eval_kit, voicebench, enterpriseops_gym, partner_if, terminus_2_sandboxed_agent, agentic_if, legal_agent_bench, proof_judge, genrm_compare and ns_tools. They include a refusal test for each server whose refusal depends on its config, and one for vlm_eval_kit. wmt_translation's venv needs its current requirements, which add the IN22 normalizer.
• NEMO_GYM_MULTIWORKER_E2E=1 pytest tests/e2e/session_routing.
• pre-commit run --files <changed files>.

Rollout evidence

N/A: the change is a startup check and a debug log lock; no verifier or agent logic changes.

Compatibility

• A config that sets num_workers > 1 on a refusing server now fails at startup with the reason, instead of failing in its own way or starting and misbehaving.
• genrm_compare in its stateless mode (rollout_cohort with num_rollouts_per_prompt == 1), speed_bench with its default response metrics source, wmt_translation without COMET, and proof_judge and ns_tools in their other configurations accept num_workers > 1.
• gdpval refuses num_workers > 1 in run_webserver instead of in model_post_init.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4885-0ca5d400dfd4.docs.buildwithfern.com/nemo/gym

[yaoyu-33]

Non-blocking: could we keep the refusal before dependency setup? With num_workers=2, GDPVal now calls ensure_libreoffice() before the shared guard; VoiceBench's IFEval path similarly calls _ensure_nltk(). This can install dependencies unnecessarily or mask the worker-count error with a setup failure. Calling the shared guard early in these constructors would preserve the previous fail-fast behavior.

[ananthsub]

Fixed. gdpval, voicebench, the partner_if mock tools and speed_bench now call refuse_unsupported_num_workers(self) at the start of model_post_init, before LibreOffice, the NLTK data, the tool backend or the startup metrics snapshot. run_webserver still checks it once the server is built, so nothing else changes.

The other refusing servers build only in-memory objects or read small files in model_post_init, so their worker-count error already comes first.

The gdpval and voicebench tests now fail if LibreOffice or the NLTK data is set up before the refusal, and both fail without the early check.