[Refactor] Bound GPU test parallelism and reap strays

[huydhn]

Second of three. Two ways the test run misbehaves in a job container, both independent of where the job runs.

• Jobs hung to the 120-minute timeout after the script had already exited 0. run_with_env_secrets.py drains the step's stdout until EOF, and EOF waits on every process holding the write end. A ps dump named the holder: Xvfb, started by the test setup and outliving pytest. Under linux_job_v2 the surrounding docker run reaped it. Now the script dumps survivors and reaps them.
• -n auto gives 16 xdist workers on the GPU pods. 16 concurrent CUDA contexts on one A10G leave no room: a test spawning its own CUDA subprocess dies in cuDevicePrimaryCtxRetain with CUDA_ERROR_OUT_OF_MEMORY, and ordinary Triton tests fail to allocate. Which tests lose varies per run, so cap GPU concurrency at 4. Shard 3 goes ~650s → ~1130s, well inside the timeout. CPU keeps its measured 24.

test_dreamer_v3_checkpoint_resume_processes is also deselected from the xdist runs and added to the serial shard — the existing quarantine is path-based, and moving all of test_dreamer_v3.py would cost ~1300s of the shard's ~2950s.

Stacked on #4531, below #4530.

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4532

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (6 Unrelated Failures)

As of commit 133b21d with merge base 3231aea ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Examples Tests on Linux / tests (3.12, 13.0, 1) / linux-job (gh) (trunk failure)
  AttributeError: module 'ray.internal' has no attribute 'free'
• Examples Tests on Linux / tests (3.12, 13.0, 2) / linux-job (gh) (trunk failure)
  AttributeError: module 'ray.internal' has no attribute 'free'
• Unit-tests on Linux / tests-cpu (3.10, collectors) / linux-job (gh) (trunk failure)
  test/test_collectors.py::TestCollectorStats::test_ray_collector_stats
• Unit-tests on Linux / tests-cpu (3.13, collectors) / linux-job (gh) (trunk failure)
  test/test_collectors.py::TestCollectorStats::test_ray_collector_stats
• Unit-tests on Linux / tests-gpu-distributed (3.12, 13.0) / linux-job (gh) (trunk failure)
  test/test_distributed.py::TestRayCollector::test_ray_collector_policy_constructor
• Unit-tests on Linux / tests-stable-gpu-distributed (3.12, 13.0) / linux-job (gh) (trunk failure)
  test/test_distributed.py::TestRayCollector::test_ray_collector_policy_constructor

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[huydhn]

@pytorchbot drci