[Connor-XY]

Depends on #8054 and #8058 (which contains #8049 and #6293). Please review the diff from the merge of their heads: scoped diff. The merge commit resolves the conflicts between #8058 and #8054 in text_generation_controller.py and the allowlist; this PR's own commit touches neither file.

Why. InferenceConfig.resolve_pg_collection is where inference picks its process groups. It checked the collections it took from the model or built from parallel_state, but returned a configured collection as given, and that is what to_inference_config and the inference shards pass. An unset field reads as None, which torch.distributed treats as WORLD. For a collection without mp, DynamicInferenceEngine.start_listening_to_data_parallel_coordinator broadcast global rank 0's request address over the world and called barrier() on the world:

• That hangs, or synchronizes unrelated ranks, when inference runs on part of the world, for example RL training and inference in one job.
• When it does not hang, the MP workers of every replica subscribe to global rank 0's publisher.

A None dp turns the coordinator address broadcast into a world broadcast and lets every replica start its own coordinator. A None tp or pp makes every rank an MP coordinator. The disaggregated coordinator setup also reads cp and expt_tp, and the Dynamo engine service reads expt_dp, which the warned fallback never built.

What.

• resolve_pg_collection checks whichever collection it returns. The collection must set tp, pp, dp, mp, ep, cp, expt_tp and expt_dp, and tp, pp, dp and mp must be process groups, not None. Otherwise it raises ValueError before any component uses the collection, and stores nothing.
• ep, cp, expt_tp and expt_dp may still be None for an axis that is off.
• A model's collection is used only when it sets all eight groups and tp_ep, and the fallback builds all eight and tp_ep, as Resolve inference process groups at one warned boundary #8058 requires. A configured collection may omit tp_ep.
• Use and validate the inference process groups instead of the global grid #8054's tests for unset groups built collections with only tp and pp and expected the wrapper's and the controller's own messages, which the config now reports first. test_rejects_unset_group now removes, or sets to None, one of tp, pp, dp and mp in a full collection and expects the config's message. The two controller tests for an unset dp are dropped, because the config rejects such a collection before a controller exists and the dp cases of test_rejects_unset_group cover it.

Tests. test_inference_process_group_fallbacks.py adds cases for:

• each communication group absent or None, and absent axis groups;
• non-member groups;
• model collections: complete, partial, without tp_ep, and with mp=None;
• the eight-group fallback;
• a dynamic context that rejects a collection without mp while global reads raise.

On 4 GPUs against its base, with no regressions: models/test_t5_model.py: 26 passed here vs 26 on the base; test_process_group_fallback.py: 19 vs 19; training/test_dataset_providers.py: 103 vs 103; transformer/test_rope.py: 8 vs 8. TestT5Model::test_constructor fails on both sides because it needs 8 ranks; the integration run, which merges all #6307 PRs onto main 77391e7, passes it on 8 GPUs (two nodes). These runs did not include this PR's two inference test files, and the new tests were not run without this change. In the integration run, this PR's test files show no regressions against main: test_inference_process_group_fallbacks.py passes 27 of 27 and test_inference_process_groups.py passes 12 of 12 on 4 GPUs. Neither file exists on main. Before the sync, the branch ran on 4 CPU gloo ranks: test_inference_process_groups.py passed 12 of 12, including the 4 new test_rejects_unset_group cases. test_inference_process_group_fallbacks.py needs flash_attn and does not import on CPU.

Compatibility.

• Collections from use_mpu_process_groups(), build_inference_pg_collection and Megatron-Bridge set all eight groups, so the standard grid is unchanged.
• A hand-built configured collection that is partial, or that has tp, pp, dp or mp set to None, now raises instead of silently using world groups. This applies on every inference path, including static and direct mode, which never run the coordinator collectives. NeMo-RL builds InferenceConfig without a collection, so the new raise for configured collections does not reach it.
• A model whose own collection sets only some of the eight groups now takes the warned global fallback instead of its own groups. No in-repo producer builds such a collection. Should that case raise instead?
• Use and validate the inference process groups instead of the global grid #8054 allows dp=None when the sampling-seed offset is off. This PR requires dp on every path, so Use and validate the inference process groups instead of the global grid #8054's wrapper tp/pp check and its controller dp=None check can no longer be reached through the config. They are left in place.
• Not done, by design: deriving mp from tp x pp (the config raises instead), and the engine and coordinator communicators that deliberately span the world.
• Resolve the remaining inference process-group fallbacks with warnings #8751 updates a stub collection in its inference tests to pass this check.
• Needs determinism-exempt while Remove global process-group reads from Megatron Core #6293 is unmerged, because the kernel coverage check diffs against main. This PR's own diff touches no kernel file.

Synced with main 77391e7 on 2026-11-01 by a merge commit that lists each conflict and its resolution, so existing commits and review links stay valid.

Part of #6307.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[Connor-XY]

Superseded by #8901, which carries this change as commit 059fac6 under the same title, with "Originally #8736." in its message. Closing so review happens in one place. The branch stays on the fork.