[Connor-XY]

Depends on #7991, #8730, #8768 and #8772. Please review the diff from their merge: scoped diff.

• Warn on the last unwarned core reads and once per omitted collection #8772 moves the last two unwarned core reads behind their warnings, so the new tree-wide test passes with an empty UNWARNED_READS. Through Warn on the last unwarned core reads and once per omitted collection #8772, this branch contains Make the process-group checks enforce what the deprecation notes claim #8770, which adds the warned and unwarned classification and the guard extensions this PR builds on, and Count core calls of every deprecated parallel_state function #8766, Deprecate the parallel_state accessors of the global grid #8765, Deprecate use_mpu_process_groups outside core and close its silent paths #8767, Add a public process-group registry to parallel_state #8747, Run tools/checkpoint on registered process groups without rank overrides #8762, Seed the RNG from the model's process-group collection #8031, Add a dedicated fallback warning and process-group collection helpers #8049 and Resolve the schedules' process groups through one warned fallback #8055.
• docs: document the process-group CI check and building a ProcessGroupCollection #7991 writes the deprecation guide, docs/developer/parallel-state-deprecation.md, that this PR edits.
• The stricter guard accepts the silenced bootstrap block in megatron/training/initialize.py only in the form Read the bootstrap process groups from the registry #8768 gives it. Without Read the bootstrap process groups from the registry #8768, the guard's in-tree test fails.
• The docs and the review rules cite test_global_memory_buffer_imports.py from Take the global memory buffer from megatron.core.utils #8730.
• The converter test calls parallel_state.get_registered_process_groups() from Add a public process-group registry to parallel_state #8747 and model_parallel_cuda_manual_seed(pg_collection=...) from Seed the RNG from the model's process-group collection #8031, and fails at run time without them.
• The merge of Read the bootstrap process groups from the registry #8768 keeps docs: document the process-group CI check and building a ProcessGroupCollection #7991's deprecation paragraph in the Gloo accessor docstrings of parallel_state.py. The other merges are clean.

This PR takes over #8774, which ran the checkpoint-converter functional test on its registered groups, and #8776, which aligned the agent and review rules with these checks.

Why.

• Nothing that blocks a merge caught a new unwarned read of the global grid in Megatron Core. The ratchet step is report-only until [TRACKING] Deprecate parallel_state: explicit process groups in megatron/core #6307 decides on new fallbacks, TestAllowlist checks only the allowlist file, and calls from Megatron Core never emit ParallelStateDeprecationWarning, so strict mode stays silent too.
• The checker resolved only calls made through imports. An assigned module alias (grid = parallel_state), getattr() with a constant name, a reference that is not called, a module looked up through importlib or sys.modules, and any use of parallel_state's private globals all passed it.
• The guard for code outside Megatron Core accepted any catch_warnings() block that ignores ParallelStateDeprecationWarning as bootstrap code, so a read wrapped that way passed silently. It never looked at private attributes, which bypass the deprecation warning altogether. It skipped all of tests/, functional tests included.
• The checker tests, the guard and the removal-version test read the source as text. Testmon never maps a source change to them, so a selective unit-test run skips them.
• The functional checkpoint-converter test (tests/functional_tests/test_cases/common/ckpt_converter) read the global grid through 11 deprecated parallel_state calls. It also called get_model, get_forward_backward_func, the pipeline schedule, the RNG seeding and the tensor-parallel gather without process groups, so each of them fell back to the global grid. Since save train state for checkpoints #7516 its save fails with a TypeError, on main too, because it passes num_floating_point_operations_so_far=None. The recipe runs the test only in its broken scopes, so nothing noticed.
• The process-group section of AGENTS.md (which CLAUDE.md links to) predates the deprecation of the parallel_state accessors. It allows "migration fallbacks with an explicit comment" and says not to apply the rules to megatron/training. The pr-review rubric carries its own copy of that text, and docs/developer/contribute.md calls the process-group job report-only. An agent or reviewer following them would add or pass a commented, silent read of the global grid, skip the training code and functional tests that the guard covers, accept a silent catch_warnings() block or a private write that CI rejects, and assume that nothing blocks an unwarned core read.

What. The first six commits tighten the checker and the guard.

• The checker (tools/check_process_group_usage.py) counts every reference to a counted function, called or not. It follows:
  • assigned aliases within its scoped bindings, including names declared global or nonlocal, walrus targets in comprehensions, either side of a conditional expression and the operands of and and or;
  • self, cls and class-attribute aliases, as the guard does;
  • getattr(), setattr() and delattr() with a constant name;
  • importlib.import_module() with a positional or keyword name, and relative names with their package;
  • sys.modules lookups and star imports, megatron.core's included.
• A new private kind counts reads, writes and deletions of parallel_state's private attributes. They are never warned. The deprecated _set_global_memory_buffer() stays a deprecated function. Each alias binding is resolved once per pass, so long reassignment chains cannot stall CI.
• The tree has none of the new forms. The per-line results over megatron/core are identical to the previous checker's, and the allowlist does not change.
• The guard (tests/unit_tests/test_parallel_state_accessor_uses.py) flags every use of a private parallel_state attribute, including through setattr() and delattr(), and follows the same new alias and lookup forms.
• The guard accepts a silenced block only after the code has warned on its own: a statement that is only a call of warnings.warn(..., ParallelStateDeprecationWarning) or warn_global_process_group_fallback(), before the with, in the block that holds it or in a block that contains that block. Only a spelling of warnings.warn that the file's imports bind counts, and the category name must match exactly, so logger.warn() does not. The lazy initialization in megatron/training/initialize.py is the only such block in the tree.
• New TestTree in test_check_process_group_usage.py runs the checker's classification over megatron/core. It fails on a counted read that is neither warned nor guarded, apart from the reads in UNWARNED_READS (none now), and on any private use. It also reports stale UNWARNED_READS entries, including those of a moved or deleted file. It runs in the blocking "Test the process-group usage checker" step and in the unit-test bucket.
• tests/unit_tests/testmon_mandatory_tests.yaml maps the folders these three tests scan, and each top-level script, to the three test files, for the H100 buckets. A new test in tests/test_utils/test_unit_testmon_mandatory.py checks that the mapping covers every scanned folder and top-level script.
• The guide, docs/api-backwards-compatibility-check.md and the workflow comments describe the new failures, the private reads, and the exact bootstrap rule. The guide's guard paragraph is the reference statement of that rule: which filter forms the guard accepts (simplefilter or filterwarnings with "ignore" and the category, in the block's body), which it flags (catch_warnings(action=..., category=...), a blanket ignore, an ignore of FutureWarning), and that a function or lambda defined in the block gets no cover. The docs also say that test_global_memory_buffer_imports.py rejects memory-buffer aliases from parallel_state, fallback or not.

The next four commits move the checkpoint-converter functional test onto its registered groups.

• One commit saves with num_floating_point_operations_so_far=0, the count a run starts from. It is independent of the rest and applies to main on its own. Would maintainers rather take it as a separate PR?

• Each pipeline keeps the collection that initialize_model_parallel() registers for its grid, read with parallel_state.get_registered_process_groups(), and passes it or its groups to every call that needs them:

  • ranks and stages from pg.tp, pg.pp, is_pp_first_stage() and is_pp_last_stage(), and the first rank of pg.tp as the broadcast source;
  • get_model(pg_collection=...) and model_parallel_cuda_manual_seed(123, pg_collection=...);
  • get_forward_backward_func(pp_size=pg.pp.size(), vp_size=None), and the schedule with pg_collection and a P2PCommunicator on pg.pp;
  • the embedding all-reduce over pg.embd, and gather_from_tensor_model_parallel_region(group=pg.tp).

  The ranks, stage checks, broadcast source, RNG seeds and the order of the collectives stay the same. The helpers that read the groups become instance methods.

• The guard also scans tests/functional_tests. Functional tests run Megatron end to end like the entry points the guard already covers, and only the unit tests use the global grid on purpose. The comment above SCANNED_FOLDERS names every folder the guard skips and why. tests/functional_tests joins the Testmon mandatory mapping, so a change there selects the parallel_state source checks in a selective unit run.

• The guide lists tests/functional_tests/ among the guarded folders. The guard docstring and docs/api-backwards-compatibility-check.md say that the unit tests, not all tests, are skipped.

The last four commits align the agent and review rules. They change docs and agent guidance only.

• AGENTS.md states the current rules:
  • Code takes its groups from the owning model's collection or an explicit group from its caller. New code requires its groups.
  • What is deprecated, in which releases, and what is not: bootstrap, teardown and the registry stay.
  • Compatibility fallbacks go only through the warned helpers, dated 0.21 and 0.23 by default. A comment does not make a read a fallback.
  • What the checker counts and allows in megatron/core, and that its tests fail CI on a read that is neither warned nor guarded.
  • What the guard covers outside megatron/core (the rest of megatron/, examples/, tools/, tasks/, scripts/, docs/, tests/functional_tests/ and the top-level scripts) and what it allows.
  • What the pytest filters turn into errors, how to find fallbacks with strict mode, and what reviewers flag.
• skills/pr-review/references/light.md and strict.md state the same rules. The strict rubric gives 0.21 and 0.23 as defaults: a fallback that first warns in a later release passes its own dates and is not a finding for them.
• The guide's "Reviewing?" paragraph no longer lists bootstrap code and warned fallbacks as allowed in megatron/core, and covers the code outside it.
• docs/developer/contribute.md says which reads in megatron/core fail the process-group job. Only its allowlist ratchet is report-only now.
• Every text states the rules the same way and points to the guide's guard paragraph for the details:
  • the bootstrap rule: a statement that is only a call of warnings.warn(..., ParallelStateDeprecationWarning) or warn_global_process_group_fallback() comes before the with, in the block that holds it or in a block that contains that block;
  • the filter the guard accepts: a body call of warnings.simplefilter("ignore", ParallelStateDeprecationWarning) or the matching filterwarnings();
  • private parallel_state attributes are never allowed. The memory buffer comes from megatron.core.utils, since test_global_memory_buffer_imports.py rejects its aliases from parallel_state even in a fallback;
  • only the unit tests may use the global grid.

Tests.

• New self-tests for each read form, the guard's bootstrap and private rules, the stale-entry report and the Testmon mapping. test_many_reassignments_of_a_name_resolve_quickly fails if resolving 60 reassignments takes more than 10 s.
• Before the sync, on 4 CPU ranks with Gloo, test_check_process_group_usage.py, test_parallel_state_accessor_uses.py, test_removal_versions.py and test_unit_testmon_mandatory.py: 93 passed on the base and 116 on this branch, with no failures. Without the converter change, the guard lists the converter test's 11 deprecated calls.
• The checker on this branch and on the base, before the sync: "OK: no new uses of global parallel_state (101 grandfathered: 57 guarded, 44 warned, 0 unwarned)".
• The bootstrap rule, the accepted filter forms and the memory-buffer rule in the docs were checked by running the guard, the checker and test_global_memory_buffer_imports.py on small examples of each form.
• A CPU run of the converter harness before the sync, with Gloo and the local transformer implementation, without the conversion step, which needs CUDA, gives bitwise-identical outputs before and after the change on five TP and PP grids. It raises no deprecation or fallback warning after the change, against 9 to 13 per rank before.

On 4 GPUs against its base, no test regresses:

• test_check_process_group_usage.py: 70 passed here vs 52 on the base (202 vs 130 subtests). The 18 new tests pass.
• test_parallel_state_accessor_uses.py: 11 passed here vs 7 on the base (93 vs 49 subtests). The 4 new tests pass.
• test_unit_testmon_mandatory.py: 28 passed here vs 27 on the base. The new test passes.
• Four files this PR does not touch, test_parallel_state_deprecation.py, test_example_process_groups.py, transformer/test_owner_linear_pg_collection.py and inference/high_level_api/test_apis.py, pass 185 tests on each side.
• tests/functional_tests/test_cases/common/ckpt_converter/__main__.py errors at pytest collection on both sides, because it is a functional-test entry point that the recipe runs with torchrun -m.

In the integration run (main 77391e7 with all 93 #6307 PRs and 6 related fixes merged, against main), this PR's test files in tests/unit_tests show no regressions against main. On 4 GPUs there, test_check_process_group_usage.py passes 70 tests against 29 on main. test_parallel_state_accessor_uses.py, which main does not have, passes 11.

The converter test ran on GPUs only before the 2026-11-01 sync, on the stacked version of these commits, where they were three PRs:

• With all the other process-group changes from [TRACKING] Deprecate parallel_state: explicit process groups in megatron/core #6307 applied, the converter's TP2 PP2 round trip passes on 4 GPUs, with no ParallelStateDeprecationWarning or ProcessGroupFallbackWarning.
• On that tree the full converter test module on 8 GPUs (two nodes) passes all 7 GPT pipelines: t8p1 to t1p8, t4p2 to t2p4, t2p4 to t4p2, t1p8 to t8p1, TE to local, local to local and local to TE. The output MSE against the source model is 0 to 1.7e-07 after loading, against 6.9e-03 to 2.3e-02 after the test re-initializes the weights at random. No log has a deprecation or fallback warning.
• On main with only the save fix, the round trip still fails in the conversion with model_provider() missing 1 required positional argument: 'model_builder'. Run tools/checkpoint on registered process groups without rank overrides #8762 fixes that loader bug.

Compatibility.

• No change to Megatron Core or the training code: the checker, the guard, tests, the CI workflow, docs and agent guidance. The allowlist is unchanged, and the guard passes on the tree.
• Newly blocking: the "Test the process-group usage checker" step fails on a counted read in megatron/core that is neither warned nor guarded, and on any private parallel_state use there. The ratchet step stays report-only, and a new warned or guarded read still passes this test. How many fallbacks Megatron Core may add stays with [TRACKING] Deprecate parallel_state: explicit process groups in megatron/core #6307.
• TestTree adds about 2.3 s to that step and to the unit-test bucket.
• The guard now reports deprecated parallel_state calls in functional tests. Nothing else under tests/functional_tests calls one.
• A core call of resolve_gtp_remat_group(..., owner=None) still counts as guarded, although it does not warn. Its three callers warn before the call. Should a follow-up classify these calls by the warning before them, like the other reads?
• Neither tool follows a module passed as an argument or a parameter default, returned, stored in a container, or bound by a for or with target; an import in a class body reached through self; getattr() with a name built at run time; vars(parallel_state)[...], parallel_state.__dict__[...], __import__() or sys.modules.get(). An attribute alias such as self.grid counts in every class of its file.
• A new top-level script needs a line in testmon_mandatory_tests.yaml. test_unit_testmon_mandatory.py checks that, but no GitHub workflow runs tests/test_utils. The mapping does not list test_inference_entry_point_groups.py, whose static checks a selective run therefore skips. Should a follow-up add it?
• The recipe tests/test_utils/recipes/h100/ckpt_converter.yaml stays in its broken scopes (mr-broken, mr-github-broken). Should it be re-enabled, given the GPU runs above? The recipe also checks out a reference tree for its backward-compatibility step, which these runs did not cover.
• Not exercised in the converter test: the LLaVA pipeline (commented out in test_all_pipelines) and the MoE pipelines (marked to-do). vp_size=None is deliberate: the old fallback read the grid's virtual-pipeline size, which Utils.initialize_model_parallel leaves None. The harness passes the registered collection object itself, with all its fields set.
• AGENTS.md says that under pytest pyproject.toml turns the deprecation warning into an error for this repository's modules. That describes Make the process-group checks enforce what the deprecation notes claim #8770's error filter. If maintainers drop the filter, this sentence needs the matching edit.
• The skills/pr-review files may need review from their code owners.
• Not touched, and already on main: the adjust_batch_dimensions_for_expert_parallelism docstring, which says that ep_group=None uses the global grid, and the @deprecated helpers in moe_utils.py that are past their removal versions.
• determinism-exempt is needed only while the PRs below this one are unlanded. CI diffs against main, and the base contains registered kernel files. This PR's own diff passes the coverage check.

The branch was synced with main 77391e7 on 2026-11-01 by a merge of its updated bases, with no rebase or force-push, so existing commits and review links stay valid. That merge had no conflicts.

Later on 2026-11-01 the branch also merged main bfab98f. Its #8471 moves FullyShardedDataParallelV1 and FullyShardedDataParallelV2 from mcore_fsdp_adapter.py into mcore_adapter_v1.py and mcore_adapter_v2.py, and keeps mcore_fsdp_adapter.py as the factory that re-exports them. The merge carries its bases' edits to those classes where the code moved; this PR's own diff does not change. On 4 GPUs against its new base, the Megatron-FSDP adapter tests (7 files) show no regressions: 80 passed here, 80 on the base. The two CUDA-graph end-to-end cases of test_mcore_fully_sharded_data_parallel.py fail on both sides, as they do on main.

Part of #6307.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Connor-XY]

Superseded by #8910, which carries this change as commit c3724a3 under the same title, with "Originally #8773." in its message. Closing so review happens in one place. The branch stays on the fork.