[svcnemo-autobot]

Background

Make managed installs reproducible for AUT-2574 while preserving custom components and parent-environment compatibility. Lock-update automation is excluded.

What changed

• Add managed component/preparation locks and locked sync, retaining custom setup and parent compatibility resolution.

• Current externally updated head: 02151a64.

• The observed description records repairs to Stirrup's removed-file consumer and the Slurm export tests' pip-install assumptions. Independent source and validation reconciliation remains pending.

• Preserve Pool's confirmed timeout and model-call evidence when trajectory capture fails. Keep unreadable-trajectory failures fatal after completed inference, and cover both trajectory states in the existing regression test.

• Retain upstream driver-install failure reporting around locked sync, clearing the failure trap only after sync succeeds. Update the remote-reader assertion and venv activation fixture to preserve their existing behavior with locked installs.

• Retain tomlkit alongside upstream's OpenTelemetry dev dependency and default tiktoken dependency. Refresh the affected managed locks while preserving existing component package versions.

Details

flowchart LR
  L[Project and lock] --> S[Locked sync]
  S --> E[Component environment]
  P[Parent requirements] --> C[Compatibility check]
  E --> C
  C --> R[Compatibility resolution when needed]
  U[Custom component] --> V[Custom setup]

Loading

The requester authorized landing this PR through normal repository requirements and addressing Kajal's remaining requests in a follow-up PR. No check bypass is requested.

Tested

• Signed merge: deb1f9d64b38b6ee700dae5ae007ff8682a9a505, joining the previous PR head 558eeb9ae8962212e2e9a5c9640026c28d06c80f with upstream main at 3a9b9eb981ab7a3ea0ebd517cbd4fca1d586768b. Signature and DCO sign-off verified.
• pre-commit run --all-files: passed, including Ruff, formatting, guardwords and import-linter. git diff --cached --check passed.
• python scripts/component_locks.py --jobs 4: all 267 projects passed. Existing component package versions were preserved; the root lock adopts upstream's tiktoken==0.14.0.
• The following focused validation passed 661 tests:

pytest -q -o addopts='' --tb=short \
  tests/unit_tests/test_slurm_script.py \
  tests/unit_tests/test_orchestration_exports.py \
  tests/unit_tests/test_remote_managed_recovery.py \
  tests/unit_tests/test_remote_reader_transport.py \
  tests/unit_tests/test_reader_cleanup.py \
  tests/unit_tests/test_reader_login_fallback.py \
  tests/unit_tests/test_reader_srun.py \
  tests/unit_tests/test_reader_progress.py \
  tests/unit_tests/test_allocation_cut_recovery.py \
  tests/unit_tests/test_component_dependencies.py \
  tests/unit_tests/test_import_linter_coverage.py \
  tests/unit_tests/test_locked_env.py \
  tests/unit_tests/test_speed_bench_preparation.py \
  tests/unit_tests/orchestration/test_orchestration_otel.py

• A final git fetch origin main confirmed the merged main commit remains 3a9b9eb981ab7a3ea0ebd517cbd4fca1d586768b; git merge-base --is-ancestor origin/main HEAD passed.
• Previous-head CI: all five requested required checks and Pool server shard 4 passed on 558eeb9ae8962212e2e9a5c9640026c28d06c80f in run 41814672225. That run retains a separate non-blocking Kilocode conformance failure. These are not verdicts for this merge commit; CI has not been rerun for the latest head.
• Model rollouts: N/A for this merge integration. The broader integration and review items listed above remain open.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ko3n1g]

/ok to test e4269bd

[ko3n1g]

/ok to test e4269bd

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4837-558eeb9ae896.docs.buildwithfern.com/nemo/gym

[ko3n1g]

/ok to test dc605e1

[ko3n1g]

/ok to test 1dd1950

[kajalj22]

Thanks for the follow-up. Most of what I raised last time is fixed (osworld dependencies, the codec torch check, mirror sync, benchmark prepare, macOS locking, the freshness workflow, cvdp pip). I re-checked the current head (1dd1950) and found four things I'd like addressed. Severity: P1 = should be fixed before merge, P2 = real but non-blocking. I may be missing context, so please push back if I've misread anything.

P1 (fix before merge)

1. [P1] Wheel installs break the molopt and OSWorld installers. setup.py replaces every packaged uv.lock with a 54-byte placeholder. chemcotbench/setup_molopt.py:76,98 and osworld_agent/install_optional_runtime_deps.sh run uv sync --locked directly on those files. In a wheel built from this head, chemcotbench/molopt/uv.lock and osworld_agent/runtime/uv.lock are placeholders, and uv sync --locked on one fails with Failed to parse uv.lock. Please route these through _locked_env/_wheel_project, or ship these two locks unchanged.

2. [P1] The transformers>=5.17.0 constraint overrides COMET's own limit. unbabel-comet 2.2.7 requires transformers<5.0 and huggingface-hub<1.0. The wmt_translation and longmt_eval locks resolve transformers 5.22.1 through the override. I have not tested COMET on transformers 5, so this is a risk rather than a confirmed break. Could we either run the COMET tests on it, or exempt these units from the floor?

3. [P1] Lock freshness is not a required check. component-locks.yml is path-filtered and is not in Nemo_CICD_Test's needs. The required pytest tests check that locks exist and match the policy, but not that they are up to date. A stale lock can still merge. Could you make this check required, or call it from a job Nemo_CICD_Test depends on?

P2 (non-blocking)

4. [P2] uv 0.13.4 is now the minimum, but CONTRIBUTING still says 0.11.21. CONTRIBUTING.md:100 and the comments in full-test-suite.yml:35-36 say 0.11.21 or newer. With uv 0.11.28 every managed project fails with Required uv version >=0.13.4 does not match the running version 0.11.28. CI is pinned to 0.13.4, so this only affects contributors. Please update the docs and add a short migration note.

[ko3n1g]

/resolve-conflict

[ko3n1g]

/ok to test 982c264

[ko3n1g]

/ok to test 97d9b77

[ko3n1g]

/ok to test 02151a6

[ko3n1g]

/ok to test 558eeb9

[copy-pr-bot]

/ok to test 558eeb9

@ko3n1g, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[ko3n1g]

/ok to test 558eeb9

[OliviaViessmann]

not touching core code from the bionemo side. Approving.

[ko3n1g]

/ok to test deb1f9d

[qiaochuz-nv]

Hi @ko3n1g
#4837 breaks vLLM startup in responses_api_models/local_vllm_model on main.

The new local_vllm_model/pyproject.toml lists "pycountry; sys_platform == 'never'" under
[tool.uv] override-dependencies, so the lock never installs pycountry. vLLM needs it:
transformers 5.22.1 MistralCommonBackend -> pydantic_extra_types.language_code, which raises
without pycountry, and vllm serve exits 1 during startup.

Repro on main d56eeb5 (still broken after #5604):
responses_api_models/local_vllm_model/.venv/bin/python -c "import pydantic_extra_types.language_code"
-> RuntimeError: The language_code module requires "pycountry" to be installed.
responses_api_models/local_vllm_model/.venv/bin/python -c "from transformers import MistralCommonBackend"
-> ModuleNotFoundError: Could not import module 'MistralCommonBackend'

The parent 3a9b9eb (mistral_common 1.13.0, transformers 5.21.0, pycountry installed) starts fine.
Impact: anything that serves a model through local_vllm_model fails at startup; our QA GPU lane
went 7/7 red in ~15 s on d56eeb5.
Likely fix: drop the pycountry override for local_vllm_model. The same override appears in other
component pyprojects (e.g. benchmarks/birdbench); worth checking any that load vLLM or transformers'
Mistral backend.