[laszkiewiczp]

Problem

SciCode's verify() dispatches each sub-step to a ray.remote worker that runs subprocess.run([sys.executable, "-c", program]). The worker's sys.executable is whatever Python the Ray cluster runs — which is not guaranteed to be the SciCode resources server's venv, the only one with SciCode's scientific deps (h5py, scipy, numpy, sympy, matplotlib).

In a split deployment — vLLM + the Ray head in one container, the resources server joining that cluster via ray_head_node_address=auto — the sub-step lands on a worker whose interpreter lacks those deps, so every generated program dies at import h5py and mean/reward is stuck at 0 even though the model's answers are fine.

Fix

Execute the sub-step in the resources server's own process via the thread-pool executor instead of shipping it to a Ray worker:

result = await loop.run_in_executor(None, run_substep, program, self.config.timeout_secs)

The resources server process already runs the SciCode venv, so run_substep's subprocess.run([sys.executable, ...]) now uses the interpreter that has the deps. Concurrency is unchanged — still bounded by the existing num_processes semaphore, and subprocess.run releases the GIL. The now-unused run_substep_remote/import ray are removed and the verify() tests retarget run_substep.

Validation

Ran the full SciCode benchmark end-to-end on a split (vLLM + separate resources) deployment. Before this change every sub-step failed at import h5py and reward was 0; after, sub-steps execute in the resources-server venv and subtask_accuracy lands in line with the NeMo-Skills SciCode reference. Unit tests updated and passing.

Alternative considered

Keeping the Ray task but pinning its interpreter (or a runtime_env) to the resources-server venv also works, but relies on that venv path being valid on every Ray worker (fragile across nodes/containers). Running in-process removes the ambient-interpreter assumption entirely; SciCode sub-steps are short, timeout-bounded subprocesses, so single-node execution is sufficient.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.