Skip to content

ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow - #5957

Merged
balasaajay merged 3 commits into
NVIDIA:mainfrom
balasaajay:nemo-triage-integ-4
Jul 23, 2026
Merged

ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow#5957
balasaajay merged 3 commits into
NVIDIA:mainfrom
balasaajay:nemo-triage-integ-4

Conversation

@balasaajay

@balasaajay balasaajay commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

  • Consolidates functional and smoke results into the existing notification and Linear report artifacts, avoiding duplicate smoke-only notifications.
  • Assigns stable smoke-test identities such as recipe@smoke-gb200, allowing smoke failures to be reconciled against Linear.
  • Adds threaded Slack follow-ups containing categorized failure details and Linear created/updated/closed and assignee summaries.
  • Clean up release test type notifications

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

@balasaajay
Enhance CI triage configuration and testing
aed2285 
- Updated `.gitlab/nemo-ci-triage.yml` to include `channel_id_env` and `reconcile_proposal` for the `megatron_lm` module.
- Modified `06.triage.yml` to add detailed follow-up logic for Slack notifications based on thread timestamps.
- Updated `Dockerfile.linting` to reflect the latest `NEMO_CI_TRIAGE_COMMIT`.
- Added new test for retry exhaustion alerts in `test_ci_triage.py`, ensuring only release tests trigger Slack alerts.
- Enhanced `linear_ci.py` to support smoke test prefix handling in pipeline names.

These changes improve the integration of Slack notifications and enhance the testing framework for better reliability.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
Refactor functional test configuration and update notification logic
343f455 
- Commented out the `functional:smoke_notify` job in `.gitlab/stages/04.functional-tests.yml` to streamline the CI pipeline.
- Updated tests in `test_ci_triage.py` to reflect the removal of `functional:smoke_notify`, ensuring assertions are accurate.
- Modified `notify.py` to support multiple job prefixes for functional tests, enhancing flexibility in job handling.

These changes improve the clarity of the CI configuration and ensure that the testing framework remains robust and up-to-date.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
balasaajay requested a review from a team as a code owner July 22, 2026 05:17
@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft July 22, 2026 05:17
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@copy-pr-bot

copy-pr-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@balasaajay balasaajay changed the title ci: Integrate latest features from triage ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow Jul 22, 2026
@balasaajay
balasaajay marked this pull request as ready for review July 22, 2026 07:32
ko3n1g
ko3n1g reviewed Jul 22, 2026
View reviewed changes
Comment thread .gitlab/stages/04.functional-tests.yml Resolved
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Jul 22, 2026
@balasaajay
Remove commented-out functional:smoke_notify job from CI configuration
c6d7840 
- Deleted the `functional:smoke_notify` job from `.gitlab/stages/04.functional-tests.yml` to simplify the CI pipeline.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
@balasaajay
balasaajay added this pull request to the merge queue Jul 22, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29957183522

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29966718379

@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/29972922692

Merged via the queue into NVIDIA:main with commit b842f59 Jul 23, 2026
90 checks passed
@balasaajay
balasaajay deleted the nemo-triage-integ-4 branch July 23, 2026 05:39
shjwudp pushed a commit to shjwudp/Megatron-LM that referenced this pull request Jul 30, 2026
@balasaajay @shjwudp
ci: Integrates the latest config-driven nemo-ci-triage Slack and Line…
ae84082 
…ar workflow (NVIDIA#5957)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
@balasaajay @terminator123
ci: Integrates the latest config-driven nemo-ci-triage Slack and Line…
2d1d9a5 
…ar workflow (NVIDIA#5957)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
svcnvidia-nemo-ci pushed a commit to dimapihtar/Megatron-LM that referenced this pull request Aug 4, 2026
@balasaajay @dimapihtar
ci: Integrates the latest config-driven nemo-ci-triage Slack and Line…
2b45ab9 
…ar workflow (NVIDIA#5957)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
nemo-autobot-origin Bot pushed a commit to svcnemo-autobot/Megatron-LM that referenced this pull request Oct 7, 2026
@svcnemo-autobot
fix: post-sync corrections for nightly sync 2026-10-02
29cf2c4 
Repairs for failures found by local validation of main slice ab20a60
merged into dev f80139c:

- tests/unit_tests/inference: give the prefix-caching _StubEngine the
  cuda_graph_all_prefills attribute the merged DynamicInferenceEngine
  reads.
- megatron/core/optimizer/layer_wise_optimizer.py: tolerate a
  pg_collection without a tp group (getattr(..., 'tp', None)).
- tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover
  kernel correlation ids without FunctionEvent.linked_correlation_id,
  which the dev image's torch does not provide.
- megatron/core/parallel_state.py: size dynamic data/context-parallel
  groups against the replicate DP-CP world (with_gtp_remat=False).
- gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}:
  refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss
  and num-zeros match the values main recorded for this change.
- megatron/core/inference/contexts/dynamic_context.py: compute Mamba
  intermediate-state offsets for every prefill chunk, as main does, so
  the last block boundary of a multi-chunk prompt is cached (required by
  main dfe9d04 NVIDIA#5952).
- tests/unit_tests/test_training.py: give the dev-only training_log
  fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967)
  made training_log read when computing the global batch size.
- megatron/training/training.py: read args.freeze_all_layers with
  getattr(..., False) in the config-container model-builder path that
  main ff6b92a (NVIDIA#5926) added, as the same change already does in
  _forward_backward_grad_context; dev-only tests pass minimal args.
- megatron/core/parallel_state.py: with weight rematerialization off
  (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing
  groups of identical membership for the expert GTP weight-remat group
  and the EGTP-merged expert tp-ep-pp group instead of creating
  duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them
  unconditionally; every NCCL group keeps a watchdog thread until
  process exit, and the extra groups exhausted the thread limit in the
  transformer unit-test bucket.
- tests/functional_tests golden values: drop non-finite iteration-time
  samples from the 10 golden-value files this sync changes and set
  start_step/end_step/step_interval to the remaining steps, the layout
  main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py
  from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values.
  The H100 files lose only step 1, which lies outside the iteration-time
  steady window; the two gpt3_7b memory_speed GB200 files keep their even
  steps 2-24 at step_interval 2.
- tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the
  read-only GitLab credential to /root/.netrc with `>>` instead of
  `| tee -a`, so the token is not echoed to stdout and CI logs. The
  recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file
  otherwise identical to main.
- tests/test_utils/python_scripts/generate_jet_trigger_job.py and
  .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction
  (build_test_script, --enable-error-extraction and the four stage
  flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped
  them without a recorded reason, so this merge kept dev's copies while
  importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml
  (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four
  files now match main; 01.build.yml keeps dev's base-image pin.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot svcnemo-autobot mentioned this pull request Oct 7, 2026
Merged
nemo-autobot-origin Bot pushed a commit to svcnemo-autobot/Megatron-LM that referenced this pull request Oct 8, 2026
@svcnemo-autobot
fix: post-sync corrections for nightly sync 2026-10-02
d92c81e 
Repairs for failures found by local validation of main slice ab20a60
merged into dev f80139c:

- tests/unit_tests/inference: give the prefix-caching _StubEngine the
  cuda_graph_all_prefills attribute the merged DynamicInferenceEngine
  reads.
- megatron/core/optimizer/layer_wise_optimizer.py: tolerate a
  pg_collection without a tp group (getattr(..., 'tp', None)).
- tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover
  kernel correlation ids without FunctionEvent.linked_correlation_id,
  which the dev image's torch does not provide.
- megatron/core/parallel_state.py: size dynamic data/context-parallel
  groups against the replicate DP-CP world (with_gtp_remat=False).
- gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}:
  refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss
  and num-zeros match the values main recorded for this change.
- megatron/core/inference/contexts/dynamic_context.py: compute Mamba
  intermediate-state offsets for every prefill chunk, as main does, so
  the last block boundary of a multi-chunk prompt is cached (required by
  main dfe9d04 NVIDIA#5952).
- tests/unit_tests/test_training.py: give the dev-only training_log
  fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967)
  made training_log read when computing the global batch size.
- megatron/training/training.py: read args.freeze_all_layers with
  getattr(..., False) in the config-container model-builder path that
  main ff6b92a (NVIDIA#5926) added, as the same change already does in
  _forward_backward_grad_context; dev-only tests pass minimal args.
- megatron/core/parallel_state.py: with weight rematerialization off
  (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing
  groups of identical membership for the expert GTP weight-remat group
  and the EGTP-merged expert tp-ep-pp group instead of creating
  duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them
  unconditionally; every NCCL group keeps a watchdog thread until
  process exit, and the extra groups exhausted the thread limit in the
  transformer unit-test bucket.
- tests/functional_tests golden values: drop non-finite iteration-time
  samples from the 10 golden-value files this sync changes and set
  start_step/end_step/step_interval to the remaining steps, the layout
  main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py
  from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values.
  The H100 files lose only step 1, which lies outside the iteration-time
  steady window; the two gpt3_7b memory_speed GB200 files keep their even
  steps 2-24 at step_interval 2.
- tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the
  read-only GitLab credential to /root/.netrc with `>>` instead of
  `| tee -a`, so the token is not echoed to stdout and CI logs. The
  recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file
  otherwise identical to main.
- tests/test_utils/python_scripts/generate_jet_trigger_job.py and
  .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction
  (build_test_script, --enable-error-extraction and the four stage
  flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped
  them without a recorded reason, so this merge kept dev's copies while
  importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml
  (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four
  files now match main; 01.build.yml keeps dev's base-image pin.
- moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_{overlap_grad_reduce_param_gather_groupedGEMM,ddp_average_in_collective}_1node
  and moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node:
  refresh the dev GB200 golden values after main cd4afff (NVIDIA#5916). This
  merge imports the grad-norm/num-zeros fix but kept dev's pre-fix GB200
  goldens, so lm loss (exact) and num-zeros failed on dgx_gb200. The values
  come from the dgx_gb200 golden-value artifacts of CICD run 37688059691 on
  the previous head 29cf2c4. lm loss, num-zeros and both memory metrics are
  bit-identical to the goldens the reverted dev sync 5ccdc55 (NVIDIA#7094)
  regenerated. iteration-time drops its step-1 NaN (start_step 2), as above.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
nemo-autobot-origin Bot pushed a commit to svcnemo-autobot/Megatron-LM that referenced this pull request Oct 9, 2026
@svcnemo-autobot
fix: post-sync corrections for nightly sync 2026-10-02
be3eaaf 
Repairs for failures found by local validation of main slice ab20a60
merged into dev f80139c:

- tests/unit_tests/inference: give the prefix-caching _StubEngine the
  cuda_graph_all_prefills attribute the merged DynamicInferenceEngine
  reads.
- megatron/core/optimizer/layer_wise_optimizer.py: tolerate a
  pg_collection without a tp group (getattr(..., 'tp', None)).
- tests/unit_tests/distributed/mfsdp_v2/profiler_utils.py: recover
  kernel correlation ids without FunctionEvent.linked_correlation_id,
  which the dev image's torch does not provide.
- megatron/core/parallel_state.py: size dynamic data/context-parallel
  groups against the replicate DP-CP world (with_gtp_remat=False).
- gpt3_mcore_te_tp2_pp2_ep4_mhc_overlap_attn_cudagraph{,_nosplit}:
  refresh dev H100 golden values after main cd4afff (NVIDIA#5916); lm loss
  and num-zeros match the values main recorded for this change.
- megatron/core/inference/contexts/dynamic_context.py: compute Mamba
  intermediate-state offsets for every prefill chunk, as main does, so
  the last block boundary of a multi-chunk prompt is cached (required by
  main dfe9d04 NVIDIA#5952).
- tests/unit_tests/test_training.py: give the dev-only training_log
  fixtures gtp_weight_remat_size=1, the argument main c5ff22b (NVIDIA#4967)
  made training_log read when computing the global batch size.
- megatron/training/training.py: read args.freeze_all_layers with
  getattr(..., False) in the config-container model-builder path that
  main ff6b92a (NVIDIA#5926) added, as the same change already does in
  _forward_backward_grad_context; dev-only tests pass minimal args.
- megatron/core/parallel_state.py: with weight rematerialization off
  (gtp_remat_size == expert_gtp_remat_size == 1), reuse the existing
  groups of identical membership for the expert GTP weight-remat group
  and the EGTP-merged expert tp-ep-pp group instead of creating
  duplicate NCCL process groups. main c5ff22b (NVIDIA#4967) creates them
  unconditionally; every NCCL group keeps a watchdog thread until
  process exit, and the extra groups exhausted the thread limit in the
  transformer unit-test bucket.
- tests/functional_tests golden values: drop non-finite iteration-time
  samples from the 10 golden-value files this sync changes and set
  start_step/end_step/step_interval to the remaining steps, the layout
  main uses (cd4afff NVIDIA#5916, 7e6a045 NVIDIA#5997). tools/check_golden_values.py
  from main bb5647a (NVIDIA#5989) rejects NaN/inf in changed golden values.
  The H100 files lose only step 1, which lies outside the iteration-time
  steady window; the two gpt3_7b memory_speed GB200 files keep their even
  steps 2-24 at step_interval 2.
- tests/test_utils/recipes/gb200/gpt-dynamic-inference.yaml: append the
  read-only GitLab credential to /root/.netrc with `>>` instead of
  `| tee -a`, so the token is not echoed to stdout and CI logs. The
  recipe arrives in dev with main 354b5d2 (NVIDIA#6104); this keeps the file
  otherwise identical to main.
- tests/test_utils/python_scripts/generate_jet_trigger_job.py and
  .gitlab/stages/0{2,3,4}*.yml: restore main's opt-in error extraction
  (build_test_script, --enable-error-extraction and the four stage
  flags from main 06bb399, NVIDIA#5831). The earlier dev sync NVIDIA#7055 dropped
  them without a recorded reason, so this merge kept dev's copies while
  importing main's test_ci_triage.py, linear_ci.py and 06.triage.yml
  (e443bd3 NVIDIA#5881, b842f59 NVIDIA#5957), which depend on them. All four
  files now match main; 01.build.yml keeps dev's base-image pin.
- moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_{overlap_grad_reduce_param_gather_groupedGEMM,ddp_average_in_collective}_1node
  and moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node:
  refresh the dev GB200 golden values after main cd4afff (NVIDIA#5916). This
  merge imports the grad-norm/num-zeros fix but kept dev's pre-fix GB200
  goldens, so lm loss (exact) and num-zeros failed on dgx_gb200. The values
  come from the dgx_gb200 golden-value artifacts of CICD run 37688059691 on
  the previous head 29cf2c4. lm loss, num-zeros and both memory metrics are
  bit-identical to the goldens the reverted dev sync 5ccdc55 (NVIDIA#7094)
  regenerated. iteration-time drops its step-1 NaN (start_step 2), as above.
- megatron/core/models/gpt/fine_grained_callables.py: forward
  config.moe_ncclep_zero_copy from the GPT schedule-plan
  TransformerLayerNode to ScheduleNode, the contract main NVIDIA#5735 added,
  so free_input never empties NCCL-EP symmetric-memory inputs.
  megatron/core/transformer/transformer_config.py: reject
  moe_ncclep_zero_copy with dev's ep_overlap_use_scheduled_tensor_release
  (NVIDIA#7062), whose scheduled release would own and empty those persistent
  shared buffers. Addresses review 5470751474 (finding 1).
- megatron/core/optimizer/__init__.py and emerging_optimizers.py: with
  GTP_remat (main NVIDIA#4967), tag fused QKV weights from their TP-local rows
  before GTP sharding (param._unsharded_shape) instead of the local
  shard, which silently disabled dev's split-QKV Muon (NVIDIA#7085); rebuild
  the GTP_remat row shards and strip their alignment padding before the
  projection/per-head/global QKV split, then restore this rank's shard.
  Mirrors main's later fix (NVIDIA#7117), which is outside this slice.
  Addresses review 5470751474 (finding 2).

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

This branch was previously deployed

2 inactive deployments
public — c6d7840f Deployed Jul 22, 2026 by copy-pr-bot[bot] via finalize / publish-docs #4267
test — c6d7840f Deployed Jul 22, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #17324
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ko3n1g ko3n1g ko3n1g approved these changes

Assignees

No one assigned

Labels

Approved All necessary approvals have been made complexity: medium

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@balasaajay @svcnvidia-nemo-ci @ko3n1g