[NVBug 6668005] Verify tau2 banking_alltools startup no longer exceeds spin-up timeout #4513

Queue rank 25/48 · Difficulty 3/5 · Status on main: partially_fixed

NVBug: https://nvbugspro.nvidia.com/bug/6668005 · Priority: P0-Must have · ARB: Ewa Dobrowolska
GitHub: PR #3681 (uv lock timeout), PR #3735 (bounded spin-up wait)

Summary

On an H100x8 node, gym env start for tau2 banking_alltools sat at 2/3 servers ready for ~5 min waiting on tau2_banking_knowledge_alltools_agent, whose venv is built from a git-branch dep with a 5m30s cold uv resolve (Resolved 174 packages in 5m 30s). In the captured run the resolve finished and the log reached 3/3 servers ready; ~79s later every worker died with Failed to connect to GCS within 60 seconds — a teardown symptom not yet root-caused, not a proven spin-up-timeout failure.

Reproduction

On an H100 node, clone Gym (main), set an OpenAI key, run fresh (no cached venv): gym env start --config benchmarks/tau2/configs/banking_alltools.yaml. Observe the cold uv resolve of tau2[knowledge] @ git+.../tau2-bench@bxyu/nemo_gym_stable.

Root cause

The tau2 agent pins a moving git branch (responses_api_agents/tau2/requirements.txt). A branch ref is not content-addressable, so uv cold-resolves 174 packages every fresh run (~5m30s), near uv's 300s cache-lock wait. The later GCS failure occurred after servers were ready and remains unexplained.

Status on main (partially_fixed)

Spin-up infra was hardened after this bug (observed 2026-08-20):

• nemo_gym/global_config.py:1657-1663 adds uv_lock_timeout_seconds (default 1800s); comment names the "cold install larger than uv's 300s" failure (fix(cli): launch servers with their venv interpreter and harden spin-up #3681).
• nemo_gym/cli/env.py:745-768 bounds the wait with server_spinup_timeout_seconds (default 600s) and raises an actionable timeout naming the stalled server (fix(cli): bound Gym server startup wait #3735).
  Residual: the dep is still an unpinned branch (cold resolve ~5m+, may approach the 600s default); the GCS symptom is unaddressed.

Proposed fix

1. Re-verify on main; if needed raise ++server_spinup_timeout_seconds=1200.
2. Pin the git dep to an immutable commit SHA in requirements.txt, source.py, README; add a unit test asserting a 40-char SHA.
3. Consider a pre-warmed venv in the QA harness.

Validation

The GCS teardown is unreproduced, so an H100 re-verify is mandatory before close; the deps pin alone does not prove the fix. Run gym env start banking_alltools.yaml on H100 and confirm 3/3 ready with no GCS death. Note testcases/env/server_lifecycle.py is a generic mcqa smoke that never exercises tau2/banking_alltools or the uv-lock path and will not validate this bug.

Triage

Difficulty 3/5 — spin-up hardening merged; residual is verify-on-main plus a deps pin, but the unexplained GCS death and GPU-verify gate keep it medium. Needs GPU/model: yes.

Reviewer notes

Verifier confirmed partially_fixed and raised difficulty 2→3. The draft overstated a timeout teardown — the log reached 3/3 ready, then GCS failed ~79s later (unexplained); server_lifecycle.py is a generic mcqa smoke that cannot validate this path; an H100 re-verify is mandatory before close since the GCS death is unreproduced.