[Connor-XY]

Why. RealQuantTransformerLayer._report_quantize_tensor_info calls torch.distributed.barrier() with no group before and after it logs. It runs once per layer during model construction, when RealQuantTransformerLayer.verbose is True. Every rank enters the two barriers once for each layer it builds.

• With uneven pipeline stages, ranks enter different numbers of WORLD barriers. One stage's barriers then pair with another stage's later collectives, or the job hangs.
• A model whose grid does not span every rank, such as one MIMO module, hangs on its first layer. The other ranks never enter.
• Only global rank 0 writes the report, so the barriers order nothing.

The path is dormant in this repo. verbose is a class attribute, and no flag sets it.

What.

• Remove both barriers from megatron/core/post_training/modelopt/layers.py.
• Log each line through log_single_rank. No collective replaces the barriers, so no group has to match the ranks that build the layer.
• The report still prints on rank 0 by default. It now follows set_default_log_ranks, and it no longer needs torch.distributed to be initialized.

Tests.

• New test_real_quant_report_logs_without_collectives in tests/unit_tests/post_training/test_modelopt_module_spec.py. It patches barrier, monitored_barrier, all_reduce, all_gather and broadcast to raise, then calls the report on a stub layer. On a log rank it expects one line with the original and the quantized dtype. On any other rank it expects none.
• On 4 CPU ranks with Gloo, the file gives 7 passed and 9 errors on this branch. Main with only the new test added gives 6 passed, 1 failed (the new test, on the patched barrier) and the same 9 errors.
• The 9 errors are the GPU-only model tests (TestModelOptGPTModel, TestModelOptMLAMoE, TestModelOptLlama4MoE, TestModelOptHybridModel), which fail in setup without CUDA. They build no real-quant layer.
• On 4 GPUs the file gives 16 passed on this branch and 15 passed on main. The extra pass is the new test; the 9 model tests pass on both.
• No real-quant model with verbose=True was built on GPUs.
• tools/check_process_group_usage.py reports the same 3 pre-existing entries before and after.

Compatibility.

• Without verbose, nothing changes.
• With verbose, the report no longer synchronizes ranks. It logs the same lines, on the ranks that set_default_log_ranks names (rank 0 unless changed).

Part of #6307.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[Connor-XY]

Superseded by #8892, which carries this change as commit 4d2036b under the same title, with "Originally #8817." in its message. Closing so review happens in one place. The branch stays on the fork.