[Connor-XY]

Depends on #8049, which contains #6293. Please review the diff from the merge of #8049 into main: scoped diff.

Why. Megatron Core's fallbacks to the global process groups emit ProcessGroupFallbackWarning (#8049), but a run cannot prove that it never reaches one:

• A bare simplefilter("error", ...) misses a fallback that already warned, because each fallback warns once per process.
• python -W and PYTHONWARNINGS cannot name the class. Python resolves them before torch can be imported.
• suppress_duplicate_logs_off_rank0 ignores FutureWarning off rank 0, so only rank 0 reports fallbacks, although ranks can fall back differently, for example per pipeline stage. Its filters were prepended, so they also beat an error filter installed earlier. Rank 0 then raised while the other ranks went on and waited at their next collective.
• Without an initialized grid, resolve_process_groups(None, ...) returns None groups, and torch.distributed treats None as WORLD. The warning did not say so.

What.

• megatron/rank_log_setup.py: the rank-0 suppression appends its filters, so earlier filters keep precedence. ProcessGroupFallbackWarning passes both its filters and its showwarning override. The module stays torch-free: it matches the class by module and qualified name through an ABC __subclasshook__.
• enable_strict_process_group_fallbacks() in megatron.core.process_groups_config puts an error filter for the class at the front of the filter list and forgets which fallbacks already warned. Every later fallback then raises on the rank that reaches it. Other warnings are unaffected, and warnings.catch_warnings() scopes it. Megatron-Bridge and NeMo-RL can call it too.
• --strict-process-group-fallbacks makes initialize_megatron call it right after logging setup, before distributed initialization, seeding and model construction.
• Without a grid, resolve_process_groups still builds the collection from the accessors. It does not raise, because code that works without a grid on main moves onto the resolver during the deprecation window: the FP8/FP4 contexts, the batch-invariant MoE unpermute, and callers that check the groups themselves. The warning now names the groups that are None and says that torch.distributed treats them as WORLD. It is recorded apart from the initialized-grid warning. An accessor that needs a grid, such as the one for dp, still raises. On an initialized grid nothing changes.
• skills/mcore-testing/SKILL.md documents the unit-test opt-in, pytest -W error::megatron.core.process_groups_config.ProcessGroupFallbackWarning. pytest accepts the dotted class path.

No functional test enables the flag in this PR. #8777 turns it on in five functional tests and depends on this PR.

Tests.

• test_rank_log_setup.py (CPU): fallback warnings show on ranks 0 and 1, also inside a library's catch_warnings; error filters for four categories, installed before and after the suppression; a pytest filterwarnings marker.
• test_process_group_fallback.py:
  • the strict helper raises on every fallback, including one that already warned, and on nothing else;
  • it takes precedence over earlier filters, and raises on every rank after the rank-0 suppression, in one process and across all ranks;
  • the no-grid fallback names its None groups, keeps its own once-per-owner record, and still raises for dp.
• New tests/unit_tests/training/test_strict_process_group_fallbacks.py: the flag parses, is applied after logging setup, and makes a fallback raise during distributed initialization and without a grid.

In the integration run (main 77391e7 with all 93 #6307 PRs and the 6 main-based gap fixes merged, against main itself), every test in these files passes on 4 GPUs: test_process_group_fallback.py passes 34 tests (the file is not on main), test_rank_log_setup.py passes 16 vs 5 on main, and training/test_strict_process_group_fallbacks.py passes 10 (a new file). None of these files shows a regression against main.

Earlier measurements on the code before the sync with main 77391e7: on 4 GPUs, an earlier version of this branch on #8049's previous head showed no regressions, and 7 of the 8 training-flag cases failed without the change and passed with it. The eighth runs without the flag and passes on both. The branch then ran on 4 CPU gloo ranks: no regressions against #8049's head, and the 36 new tests pass. Only the final state was tested; the intermediate commits were not run on their own. With the other process-group changes from #6307 applied:

• the full unit test suite shows no regressions against main;
• unit-test reruns under -W error::megatron.core.process_groups_config.ProcessGroupFallbackWarning pass;
• 19 of 20 end-to-end training smoke configurations run under --strict-process-group-fallbacks with no fallback warning on any rank. The 20th needs Transformer Engine 2.19 or later and was skipped.

Compatibility.

• The flag is off by default.
• The rank-0 suppression now appends its filters. An earlier error filter (user -W, strict mode, pytest) therefore raises on every rank, not only on rank 0, and ProcessGroupFallbackWarning shows on every rank. A launch that installs default or always filters for these categories before the suppression now wins over it, and only the showwarning override hides those warnings. On a standard launch the same warnings stay hidden off rank 0.
• rank_log_setup.py matches the class by module and qualified name. Renaming or moving ProcessGroupFallbackWarning requires updating it; a test catches a mismatch.
• Add a dedicated fallback warning and process-group collection helpers #8049's resolver has no production callers yet. Callers that later move onto it and asserted on main without a grid get None groups and the warning instead: param_is_not_tensor_parallel_duplicate, model_parallel_cuda_manual_seed, ExtendedRMSNorm.sharded_state_dict, MegatronModule.sharded_state_dict, build_inference_pg_collection and CudaGraphManager. Strict mode turns each of these into an error.
• Strict mode works through the filter list. A filter installed later that matches the class, such as simplefilter("ignore") or a library's catch_warnings() with simplefilter("always"), takes precedence, and the fallback then only warns. Frameworks should call the helper after their own warning setup. The warning is an Exception, so a broad except Exception around a fallback would swallow the strict error.
• --yaml-cfg runs build their args from the YAML file alone, so those files must carry strict_process_group_fallbacks, as they already carry batch_invariant_mode.
• Whichever of this PR and Seed the RNG from the model's process-group collection #8031 lands second combines the imports in megatron/training/initialize.py into from megatron.core.process_groups_config import (ProcessGroupCollection, enable_strict_process_group_fallbacks).
• Needs determinism-exempt while Remove global process-group reads from Megatron Core #6293 is unmerged: the diff against main includes Remove global process-group reads from Megatron Core #6293's dot_product_attention.py and dsa.py. This PR's own diff touches no kernel file. The SKILL.md edit leaves skills/mcore-testing/skill.oms.sig stale, as earlier edits to that file did.

The branch was synced with main 77391e7 on 2026-11-01 by a merge commit that brings in #8049's updated head (no rebase, no force-push), so existing commits and review links stay valid. The merge had no conflicts. The process-group allowlist records the reads main added since this PR was written.

Part of #6307.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[Connor-XY]

Superseded by #8895, which carries this change as commit 3611587 under the same title, with "Originally #8733." in its message. Closing so review happens in one place. The branch stays on the fork.