chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade by svcnemo-autobot · Pull Request #5997 · NVIDIA/Megatron-LM

svcnemo-autobot · 2026-07-23T17:29:45Z

Background

Transformer Engine 2.18 exposed functional regressions and ARM64 CI builders need bounded compiler concurrency.

What changed

Upgrade Transformer Engine and required NCCL/build dependencies.
Handle stateless grouped-TE extra state and trusted legacy test fixtures.
Refresh the deterministic Nemotron QAD golden.
Set MAX_JOBS=8 only for ARM64 TE builds.

Details

The compiler limit is exported only when uname -m is aarch64; UV_CONCURRENT_BUILDS is not overridden, and x86 builds are unchanged. The branch includes current main.

Tested

Formatting, lint, DCO, installation, wheel, static-inference, MoE resume, and Nemotron QAD gates passed before the ARM-only amendment.
Fresh exact ARM64 CI Dockerfile build running for this head.

…and uv.lock Updated the version of transformer-engine from 2.17.0+2e559f06 to 2.18.0+e7c550c5 in both pyproject.toml and uv.lock files, ensuring consistency across dependencies. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Replaced "nvidia-mathdx" with "nvidia-cudnn-frontend>=1.25.0" in pyproject.toml and added "nvidia-cudnn-frontend" to uv.lock to ensure compatibility with the latest requirements. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Modified the NCCL version in Dockerfile.ci.dev and install_nccl.sh to 2.30.7-1+cuda13.3 to align with the requirements of the transformer-engine 2.18's NCCL EP build. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

copy-pr-bot · 2026-07-23T17:29:48Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-07-23T17:29:55Z

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

Add the oncall reviewer (optional reviewer)
Add required review teams based on your changes

See the contribution guide for more details.

svcnemo-autobot · 2026-07-23T17:35:10Z

/ok to test 01e452d

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Adjusted numerical values in the golden value JSON files for both gpt3_7b_tp1_pp4_memory_speed and gpt3_7b_tp4_pp1_memory_speed test cases. These changes include minor corrections to memory allocation, iteration times, and other performance metrics to ensure accuracy in functional tests. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

svcnemo-autobot · 2026-07-25T00:04:41Z

/ok to test c6f76db

svcnvidia-nemo-ci · 2026-07-25T16:06:50Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30164969177

…A#5997) Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com> Co-authored-by: Ajay Balasa <abalasa@nvidia.com>

Repairs for failures found by local validation of main slice ab20a60 merged into dev f80139c: - tests/unit_tests/inference: give the prefix-caching _StubEngine the cuda_graph_all_prefills attribute the merged DynamicInferenceEngine reads. - megatron/core/optimizer/layer_wise_optimizer.py: tolerate a pg_collection without a tp group (getattr(..., 'tp', None)). - tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover kernel correlation ids without FunctionEvent.linked_correlation_id, which the dev image's torch does not provide. - megatron/core/parallel_state.py: size dynamic data/context-parallel groups against the replicate DP-CP world (with_gtp_remat=False). - gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}: refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss and num-zeros match the values main recorded for this change. - megatron/core/inference/contexts/dynamic_context.py: compute Mamba intermediate-state offsets for every prefill chunk, as main does, so the last block boundary of a multi-chunk prompt is cached (required by main dfe9d04 NVIDIA#5952). - tests/unit_tests/test_training.py: give the dev-only training_log fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967) made training_log read when computing the global batch size. - megatron/training/training.py: read args.freeze_all_layers with getattr(..., False) in the config-container model-builder path that main ff6b92a (NVIDIA#5926) added, as the same change already does in _forward_backward_grad_context; dev-only tests pass minimal args. - megatron/core/parallel_state.py: with weight rematerialization off (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing groups of identical membership for the expert GTP weight-remat group and the EGTP-merged expert tp-ep-pp group instead of creating duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them unconditionally; every NCCL group keeps a watchdog thread until process exit, and the extra groups exhausted the thread limit in the transformer unit-test bucket. - tests/functional_tests golden values: drop non-finite iteration-time samples from the 10 golden-value files this sync changes and set start_step/end_step/step_interval to the remaining steps, the layout main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values. The H100 files lose only step 1, which lies outside the iteration-time steady window; the two gpt3_7b memory_speed GB200 files keep their even steps 2-24 at step_interval 2. - tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the read-only GitLab credential to /root/.netrc with `>>` instead of `| tee -a`, so the token is not echoed to stdout and CI logs. The recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file otherwise identical to main. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Repairs for failures found by local validation of main slice ab20a60 merged into dev f80139c: - tests/unit_tests/inference: give the prefix-caching _StubEngine the cuda_graph_all_prefills attribute the merged DynamicInferenceEngine reads. - megatron/core/optimizer/layer_wise_optimizer.py: tolerate a pg_collection without a tp group (getattr(..., 'tp', None)). - tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover kernel correlation ids without FunctionEvent.linked_correlation_id, which the dev image's torch does not provide. - megatron/core/parallel_state.py: size dynamic data/context-parallel groups against the replicate DP-CP world (with_gtp_remat=False). - gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}: refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss and num-zeros match the values main recorded for this change. - megatron/core/inference/contexts/dynamic_context.py: compute Mamba intermediate-state offsets for every prefill chunk, as main does, so the last block boundary of a multi-chunk prompt is cached (required by main dfe9d04 NVIDIA#5952). - tests/unit_tests/test_training.py: give the dev-only training_log fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967) made training_log read when computing the global batch size. - megatron/training/training.py: read args.freeze_all_layers with getattr(..., False) in the config-container model-builder path that main ff6b92a (NVIDIA#5926) added, as the same change already does in _forward_backward_grad_context; dev-only tests pass minimal args. - megatron/core/parallel_state.py: with weight rematerialization off (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing groups of identical membership for the expert GTP weight-remat group and the EGTP-merged expert tp-ep-pp group instead of creating duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them unconditionally; every NCCL group keeps a watchdog thread until process exit, and the extra groups exhausted the thread limit in the transformer unit-test bucket. - tests/functional_tests golden values: drop non-finite iteration-time samples from the 10 golden-value files this sync changes and set start_step/end_step/step_interval to the remaining steps, the layout main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values. The H100 files lose only step 1, which lies outside the iteration-time steady window; the two gpt3_7b memory_speed GB200 files keep their even steps 2-24 at step_interval 2. - tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the read-only GitLab credential to /root/.netrc with `>>` instead of `| tee -a`, so the token is not echoed to stdout and CI logs. The recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file otherwise identical to main. - tests/test_utils/python_scripts/generate_jet_trigger_job.py and .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction (build_test_script, --enable-error-extraction and the four stage flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped them without a recorded reason, so this merge kept dev's copies while importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four files now match main; 01.build.yml keeps dev's base-image pin. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Repairs for failures found by local validation of main slice ab20a60 merged into dev f80139c: - tests/unit_tests/inference: give the prefix-caching _StubEngine the cuda_graph_all_prefills attribute the merged DynamicInferenceEngine reads. - megatron/core/optimizer/layer_wise_optimizer.py: tolerate a pg_collection without a tp group (getattr(..., 'tp', None)). - tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover kernel correlation ids without FunctionEvent.linked_correlation_id, which the dev image's torch does not provide. - megatron/core/parallel_state.py: size dynamic data/context-parallel groups against the replicate DP-CP world (with_gtp_remat=False). - gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}: refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss and num-zeros match the values main recorded for this change. - megatron/core/inference/contexts/dynamic_context.py: compute Mamba intermediate-state offsets for every prefill chunk, as main does, so the last block boundary of a multi-chunk prompt is cached (required by main dfe9d04 NVIDIA#5952). - tests/unit_tests/test_training.py: give the dev-only training_log fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967) made training_log read when computing the global batch size. - megatron/training/training.py: read args.freeze_all_layers with getattr(..., False) in the config-container model-builder path that main ff6b92a (NVIDIA#5926) added, as the same change already does in _forward_backward_grad_context; dev-only tests pass minimal args. - megatron/core/parallel_state.py: with weight rematerialization off (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing groups of identical membership for the expert GTP weight-remat group and the EGTP-merged expert tp-ep-pp group instead of creating duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them unconditionally; every NCCL group keeps a watchdog thread until process exit, and the extra groups exhausted the thread limit in the transformer unit-test bucket. - tests/functional_tests golden values: drop non-finite iteration-time samples from the 10 golden-value files this sync changes and set start_step/end_step/step_interval to the remaining steps, the layout main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values. The H100 files lose only step 1, which lies outside the iteration-time steady window; the two gpt3_7b memory_speed GB200 files keep their even steps 2-24 at step_interval 2. - tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the read-only GitLab credential to /root/.netrc with `>>` instead of `| tee -a`, so the token is not echoed to stdout and CI logs. The recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file otherwise identical to main. - tests/test_utils/python_scripts/generate_jet_trigger_job.py and .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction (build_test_script, --enable-error-extraction and the four stage flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped them without a recorded reason, so this merge kept dev's copies while importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four files now match main; 01.build.yml keeps dev's base-image pin. - moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_{overlap_grad_reduce_param_gather_groupedGEMM,ddp_average_in_collective}_1node and moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node: refresh the dev GB200 golden values after main cd4afff (NVIDIA#5916). This merge imports the grad-norm/num-zeros fix but kept dev's pre-fix GB200 goldens, so lm loss (exact) and num-zeros failed on dgx_gb200. The values come from the dgx_gb200 golden-value artifacts of CICD run 37688059691 on the previous head 29cf2c4. lm loss, num-zeros and both memory metrics are bit-identical to the goldens the reverted dev sync 5ccdc55 (NVIDIA#7094) regenerated. iteration-time drops its step-1 NaN (start_step 2), as above. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Repairs for failures found by local validation of main slice ab20a60 merged into dev f80139c: - tests/unit_tests/inference: give the prefix-caching _StubEngine the cuda_graph_all_prefills attribute the merged DynamicInferenceEngine reads. - megatron/core/optimizer/layer_wise_optimizer.py: tolerate a pg_collection without a tp group (getattr(..., 'tp', None)). - tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover kernel correlation ids without FunctionEvent.linked_correlation_id, which the dev image's torch does not provide. - megatron/core/parallel_state.py: size dynamic data/context-parallel groups against the replicate DP-CP world (with_gtp_remat=False). - gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}: refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss and num-zeros match the values main recorded for this change. - megatron/core/inference/contexts/dynamic_context.py: compute Mamba intermediate-state offsets for every prefill chunk, as main does, so the last block boundary of a multi-chunk prompt is cached (required by main dfe9d04 NVIDIA#5952). - tests/unit_tests/test_training.py: give the dev-only training_log fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967) made training_log read when computing the global batch size. - megatron/training/training.py: read args.freeze_all_layers with getattr(..., False) in the config-container model-builder path that main ff6b92a (NVIDIA#5926) added, as the same change already does in _forward_backward_grad_context; dev-only tests pass minimal args. - megatron/core/parallel_state.py: with weight rematerialization off (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing groups of identical membership for the expert GTP weight-remat group and the EGTP-merged expert tp-ep-pp group instead of creating duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them unconditionally; every NCCL group keeps a watchdog thread until process exit, and the extra groups exhausted the thread limit in the transformer unit-test bucket. - tests/functional_tests golden values: drop non-finite iteration-time samples from the 10 golden-value files this sync changes and set start_step/end_step/step_interval to the remaining steps, the layout main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values. The H100 files lose only step 1, which lies outside the iteration-time steady window; the two gpt3_7b memory_speed GB200 files keep their even steps 2-24 at step_interval 2. - tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the read-only GitLab credential to /root/.netrc with `>>` instead of `| tee -a`, so the token is not echoed to stdout and CI logs. The recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file otherwise identical to main. - tests/test_utils/python_scripts/generate_jet_trigger_job.py and .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction (build_test_script, --enable-error-extraction and the four stage flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped them without a recorded reason, so this merge kept dev's copies while importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four files now match main; 01.build.yml keeps dev's base-image pin. - moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_{overlap_grad_reduce_param_gather_groupedGEMM,ddp_average_in_collective}_1node and moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node: refresh the dev GB200 golden values after main cd4afff (NVIDIA#5916). This merge imports the grad-norm/num-zeros fix but kept dev's pre-fix GB200 goldens, so lm loss (exact) and num-zeros failed on dgx_gb200. The values come from the dgx_gb200 golden-value artifacts of CICD run 37688059691 on the previous head 29cf2c4. lm loss, num-zeros and both memory metrics are bit-identical to the goldens the reverted dev sync 5ccdc55 (NVIDIA#7094) regenerated. iteration-time drops its step-1 NaN (start_step 2), as above. - megatron/core/models/gpt/fine_grained_callables.py: forward config.moe_ncclep_zero_copy from the GPT schedule-plan TransformerLayerNode to ScheduleNode, the contract main NVIDIA#5735 added, so free_input never empties NCCL-EP symmetric-memory inputs. megatron/core/transformer/transformer_config.py: reject moe_ncclep_zero_copy with dev's ep_overlap_use_scheduled_tensor_release (NVIDIA#7062), whose scheduled release would own and empty those persistent shared buffers. Addresses review 5470751474 (finding 1). - megatron/core/optimizer/__init__.py and emerging_optimizers.py: with GTP_remat (main NVIDIA#4967), tag fused QKV weights from their TP-local rows before GTP sharding (param._unsharded_shape) instead of the local shard, which silently disabled dev's split-QKV Muon (NVIDIA#7085); rebuild the GTP_remat row shards and strip their alignment padding before the projection/per-head/global QKV split, then restore this rank's shard. Mirrors main's later fix (NVIDIA#7117), which is outside this slice. Addresses review 5470751474 (finding 2). Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

balasaajay and others added 7 commits July 22, 2026 10:12

chore: Update dependencies in pyproject.toml and uv.lock

af88420

Replaced "nvidia-mathdx" with "nvidia-cudnn-frontend>=1.25.0" in pyproject.toml and added "nvidia-cudnn-frontend" to uv.lock to ensure compatibility with the latest requirements. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

chore: Update NCCL version to 2.30.7 for CUDA 13.3 compatibility

eccffd1

Modified the NCCL version in Dockerfile.ci.dev and install_nccl.sh to 2.30.7-1+cuda13.3 to align with the requirements of the transformer-engine 2.18's NCCL EP build. Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

allow trusted TE extra-state pickle loading

f1c8a6c

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Merge remote-tracking branch 'upstream/main' into fix/te-2-18-ci

0629c60

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

fix(checkpointing): handle stateless grouped TE extra state

07b59ed

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

test(te): trust fixtures and refresh QAD golden

01e452d

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

svcnemo-autobot requested review from a team as code owners July 23, 2026 17:29

svcnemo-autobot requested a review from balasaajay July 23, 2026 17:29

svcnvidia-nemo-ci marked this pull request as draft July 23, 2026 17:29

svcnemo-autobot added Run functional tests labels Jul 23, 2026

copy-pr-bot Bot temporarily deployed to public July 23, 2026 17:35 Inactive

copy-pr-bot Bot temporarily deployed to test July 23, 2026 17:36 Inactive

copy-pr-bot Bot temporarily deployed to public July 23, 2026 17:38 Inactive

copy-pr-bot Bot temporarily deployed to public July 23, 2026 17:39 Inactive

copy-pr-bot Bot temporarily deployed to public July 23, 2026 17:40 Inactive

copy-pr-bot Bot temporarily deployed to public July 23, 2026 17:50 Inactive

balasaajay marked this pull request as ready for review July 23, 2026 21:16

balasaajay approved these changes Jul 23, 2026

View reviewed changes

svcnvidia-nemo-ci added the Final Review PR is in the "final review" stage label Jul 23, 2026

github-actions Bot added the complexity: low label Jul 23, 2026

Phlip79 approved these changes Jul 23, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public July 24, 2026 20:00 Inactive

copy-pr-bot Bot temporarily deployed to public July 24, 2026 20:01 Inactive

copy-pr-bot Bot temporarily deployed to public July 24, 2026 20:10 Inactive

svcnemo-autobot and others added 4 commits July 24, 2026 23:33

Merge remote-tracking branch 'upstream/main' into fix/te-2-18-ci

6df37a8

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Merge remote-tracking branch 'upstream/main' into fix/te-2-18-ci

a9538b6

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

test(golden): remove non-finite GB200 timings

c6f76db

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

copy-pr-bot Bot temporarily deployed to public July 25, 2026 00:05 Inactive

copy-pr-bot Bot temporarily deployed to test July 25, 2026 00:05 Inactive

copy-pr-bot Bot temporarily deployed to public July 25, 2026 00:08 Inactive

copy-pr-bot Bot temporarily deployed to public July 25, 2026 00:18 Inactive

balasaajay added this pull request to the merge queue Jul 25, 2026

Merged via the queue into NVIDIA:main with commit 7e6a045 Jul 25, 2026
181 checks passed

cuichenx mentioned this pull request Aug 14, 2026

fix(checkpoint): replicate stateless grouped extra state #6509

Closed

4 of 6 tasks

cuichenx mentioned this pull request Aug 21, 2026

feat(recipes): modernize DeepSeek V4 library configs NVIDIA-NeMo/Megatron-Bridge#5554

Merged

1 task done

Navigation Menu

Repository navigation

chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade - #5997

chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade#5997
balasaajay merged 15 commits into
NVIDIA:mainfrom
svcnemo-autobot:fix/te-2-18-ci

svcnemo-autobot commented Jul 23, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 25, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 25, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

svcnemo-autobot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

What changed

Details

Tested

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 25, 2026

Uh oh!

svcnvidia-nemo-ci commented Jul 25, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

svcnemo-autobot commented Jul 23, 2026 •

edited

Loading