Repository navigation
ci: group CPU test selection before the GPU matrix - #8381
Merged
balasaajay merged 5 commits intoOct 23, 2026
Merged
Conversation
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Contributor
Author
|
/ok to test 74d3a44 |
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Contributor
Author
|
/ok to test bcaef0c |
svcnvidia-nemo-ci
left a comment
svcnvidia-nemo-ci
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Reviewed all four changed files and the affected cache, GPU plan-execution, and coverage paths. No actionable issues found within the light-review scope. Tests were not executed in this read-only review.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
ko3n1g
approved these changes
Oct 23, 2026
balasaajay
enabled auto-merge
October 23, 2026 18:04
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/40007943358 |
asolergi-nv
added a commit
to asolergi-nv/Megatron-LM
that referenced
this pull request
Oct 26, 2026
Five commits landed on main since the last merge (0a2f6ee); one of them conflicts with this branch, NVIDIA#7047 "[DCP V2 3/4] Integrate dynamic CP with standard train and eval schedules" (ea99fdf). The other four (NVIDIA#8381 CI test selection, NVIDIA#8269 PR template, NVIDIA#6502 checkpoint conversion integrity, NVIDIA#7780 gtp scratch lifetime) do not touch this branch's files. Conflicts, all resolved in this branch's structure: - megatron/core/datasets/gpt_dataset.py: imports (warnings for the new deprecation warning; no `field`, token_dtype_code is a property here). NVIDIA#7047's warn-and-rewrite of the deprecated hybrid_context_parallel alias landed in validate_without_tokenizer(), which asserts only: it keeps the warning and the "not both" error and does not rewrite the alias; the SBHD exclusion counts the alias. - pretrain_gpt.py, pretrain_hybrid.py: NVIDIA#7047 edited the legacy factory and rank predicate this branch replaced with argument_utils and megatron.training.datasets.providers; this branch's imports and the removals stay, NVIDIA#7047's get_batch changes (pg_collection, dynamic_cp) merge as they are. - tools/prepare_cache.py: same; the shared factory maps the new field. The renamed field through the data section: dynamic_context_parallel joins CONTAINER_WRITTEN_FIELDS next to the deprecated alias; the container's cross-section check reads either name against model.transformer's dynamic_context_parallel and finalize() writes the canonical field and clears the alias; the standalone factory forwards both from the arguments, which validate_args already normalizes (NVIDIA#7047). The ownership, section, provider and Core config tests follow, with cases for the alias; the frozen legacy factories the provider tests use as their oracle take the same one-line rename NVIDIA#7047 gave the originals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: asolergi-nv <asolergibert@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #8019 lists every unit-test bucket twice in the Actions UI, once for CPU selection and once for GPU execution. Group PR test selection into one CPU job per platform named
unit tests selection, then expand only the GPU job as a matrix of buckets that need tests.Keep the existing H100 and GB200 recipe parser jobs and pass their complete bucket lists into the reusable unit-test workflow. Full-suite runs skip selection and execute all parsed buckets. Python handles the bucket loop and matrix generation; a small inline workflow adapter calls GitHub's cache and artifact SDKs.
The existing selector, cache format, per-bucket plan artifacts, and full-suite policy are unchanged. Coverage continues to use the original parser outputs, and the GB200 container image remains an output of its parser job. GPU jobs start after their platform's selection finishes; an unexpected selection or artifact-upload failure stops that platform's GPU dispatch.
Validation:
tests/test_utils/test_testmon_workflow.py, including grouped selection, full-mode execution with skipped selection, and coverage validation.git diff --checkpassed._unit-tests.ymlpasses Actionlint.cicd-main.ymlretains two existing missing-scriptinput errors in the functional-test jobs, also reproduced on the base commit.Follow-up to #8019. Example of the current duplicate rows: https://github.com/NVIDIA/Megatron-LM/actions/runs/39505667208.