[wangx700]

Add sparse HCCL weight synchronization between Megatron training and vLLM-Ascend rollout, plus an end-to-end comparison against a freshly loaded checkpoint.

Interface flow

Pause generation → start update → Bridge export or shard diff → ordered trainer-rank gather → payload preparation → send_weight_chunk → HCCL broadcast and checkpoint loader → finish update → resume generation.

The first synchronization sends complete checkpoint coordinates and values. Later synchronizations send only changed positions. Both use Bridge conversion and unpacked sparse patches. Inference uses PP=1; the six-NPU configuration trains with TP2/PP2/EP2 and rolls out with TP2/PP1/EP2.

Implementation

• Simplify tests/test_sparse_hccl.py from 22 functions / 37 parameterized cases to 19 functions / 35 cases. Merge sender/non-sender seeding and parallel-config rejection cases, share updater setup, and remove redundant one-record batching and optimization-specific assertions. Preserve the four-rank full-export reconstruction, 12 publication/failure combinations and 20 gather combinations. The non-sender seed case now drains a nonempty export iterator; seed reconstruction also checks zero values against a NaN-initialized destination.

• Align the dependency sparse engine with vLLM-Ascend PR18915 at 841dcfc1c: checkpoint loader views and validation live inside the sparse engine. No sparse changes remain in routed_experts.py. Keep the older pinned dependency's supported HCCL initialization API instead of importing the newer hccl_common module.

• Synchronize trainer failure state after every nonempty gather queue flush, before any rank enters its next gather. Preserve the original sender exception on rank zero and propagate a synchronized failure on peers. Existing recovery distinguishes validation failures before transmission from potentially partial sends, which keep generation paused. Add 12 distributed regression cases covering immediate/residual batches, gathered/replicated rows, success, validation failures and sender failures.

• Document the sparse export prerequisite: Megatron-Bridge is absent from the standard NPU installation. The historical A5 run used v0.3.0 at 21b02e0cfb2f8ff907e0d8baee0c5205876c6812. Its Transformers <5 requirement differs from the measured A5 Transformers 5.17.0 stack; that combination needs environment-specific validation.

• Group shared incremental synchronization helpers under vime/backends/megatron_utils/delta_sync/: hf_weight_iterator_bridge.py, megatron_delta_export.py and sparse_gather.py. Keep the transport-specific updater in update_weight/update_weight_from_sparse_hccl.py. Update its imports and test imports without changing helper implementations or updater/test execution logic. McoreParamExport directly stores gather routing; delta_spec.py is removed.

• Remove sparse updater metrics, per-phase timers, profiling contexts and success-statistics logs. Keep the shared pop_metrics() interface returning an empty dictionary.

• Remove the unused verification flag, shape/placement bookkeeping, redundant slot cache and ignored iterator arguments.

• Use one bounded batched diff implementation for both batch size 1 and larger batches. Keep bitwise comparison, snapshot refresh and Bridge conversion.

• Remove added dependency profiling and unused loader extension hooks and their dedicated tests. The existing checkpoint loader and Ascend layout adaptation remain.

• Keep numeric payload validation, inference PP/PCP checks and recovery after failures. Ordered indices retain linear duplicate checking; unordered indices retain the sorting fallback.

• Keep sparse additions focused on backend registration, HCCL communicator cleanup and checkpoint loader views, with relevant unit tests. Preserve all pre-existing Ascend compatibility patch sections from the PR base, including IPC, packed transfer, worker/model runner, KV-cache, disk reload, device and convolution fixes; these are baseline behavior and are not sparse HCCL changes. Retain the baseline full HCCL trainer. Remove only the newly added standalone example, dependency-side five-NPU end-to-end test and its registration helpers. The VIME end-to-end inference comparison remains.

• Keep the standard six-NPU launcher and enable sparse synchronization using --update-weight-mode delta and --update-weight-transport sparse_hccl.

End-to-end inference comparison

tests/test_qwen3_30B_A3B_sparse_hccl_npu.py now:

1. Runs actual rollout/training and a full seed followed by nonempty sparse updates.
2. Exports the final frozen training model through a complete Bridge conversion into a local HF safetensors checkpoint, independently of sparse patches and snapshots.
3. Captures greedy inference after sparse synchronization using a configurable baseline prompt and saves the exact prompt token IDs and sampling parameters.
4. Checks that repeated inference on the updated engine is exactly repeatable.
5. Stops that engine and starts a fresh serving process on the same two NPUs with the same engine options, loading the exported checkpoint directly from disk.
6. Requires identical generated token IDs and every generated-token logprob, with no tolerance. Missing/non-finite logprobs and any mismatch fail the job.

Use a fresh result directory with room for a complete BF16 checkpoint (about 61 GB). Artifacts include baseline.json, sparse-inference.json, checkpoint-inference.json and comparison.json. The comparison records per-field equality and maximum absolute logprob error; strict success requires zero error. The test observes successful patch publication only to require nonempty updates; it does not collect timing metrics. A one-prompt match only establishes consistency for that input.

Runtime prerequisites

Sparse export requires Megatron-Bridge. The standard NPU installation does not include it. The validated fixed version is v0.3.0 at 21b02e0cfb2f8ff907e0d8baee0c5205876c6812:

python -m pip install --no-deps \
  'git+https://github.com/NVIDIA-NeMo/Megatron-Bridge.git@21b02e0cfb2f8ff907e0d8baee0c5205876c6812'
python -c 'from megatron.bridge import AutoBridge'

--no-deps preserves the installed NPU training/serving stack. Bridge v0.3.0 declares Transformers <5; the historical A5 validation used Transformers 5.17.0. That measured combination does not establish general dependency compatibility. Bridge must be provided by the training environment.

Current measured unpacked baseline

Qwen3-30B-A3B, A5, six NPUs: training TP2/PP2/EP2 and inference TP2/PP1/EP2. Both historical jobs completed successfully on 2026-10-27.

Update	Full HCCL unpacked (s)	Sparse HCCL (s)	Sparse scalar update events	Fraction of complete seed	Sparse payload (MiB)
Seed	63.7	100.8	30532122624	100%	174706.21
v2	52.4	31.0	42302697	0.1386%	242.06
v3	52.8	36.9	62543882	0.2048%	357.88
v4	52.5	37.4	69395035	0.2273%	397.08

Table note: seconds are complete update_weights wall-clock calls, including preparation, transfer and inference application. Full transmits all weights each round; sparse sends the listed changed-element events. The denominator is 30532122624 seed elements. Repeated coordinates across versions count again. Payload includes int32 indices and BF16 values, excluding protocol overhead. Steady averages from these rounded Timer values are approximately 52.57 s and 35.10 s; sparse saves about 33.2% against unpacked full. These measurements precede this revision and do not establish a speedup from the new validation optimization.

Validation

Current head: 156f18fd85868ad55471a48f6559f36dcb3fea88.

Real trainer-side regression on 192.168.13.188

Container cjy-vime-npu-1023, isolated directory /home/w00899129/pr492-18915-align-188.VupM0o, NPUs 8,9,10,11. Installed Bridge v0.3.0 at 21b02e0cfb2f8ff907e0d8baee0c5205876c6812 into an isolated target without dependencies; Megatron source was pinned at 1dcf0dafa884ad52ffb243625717a3471643e087.

python -m pytest -q -s tests/test_sparse_hccl.py
NPU-SPARSE-PUBLISH PASS: 12 cases
NPU-SPARSE-GATHER PASS: 20 cases
SPARSE VS FULL PASS: tensors=45 updates=1975 bitwise_equal=True
37 passed, 33 warnings in 105.33s

All 37 tests passed, exit code 0, with no skips. The distributed test used actual training TP2/PP2/EP2/ETP1, real Bridge conversion and real HCCL gathering. Sparse reconstruction of all 45 HF tensors was bitwise identical to an independent full export, including BF16/FP32 and PP placeholders. The 12 failure/publication cases ran on four actual trainer ranks with controlled export/sender failures. The 20 gather cases cover empty/unequal/noncontiguous/split/reused buffers. These subcases belong to the 37 tests and are not additional pytest test counts. 105.33 seconds is complete pytest wall time, not weight-update latency.

The 37-test trainer result above is historical, from the original 71fa4d4f snapshot. Subsequent changes move the three unchanged helper modules into megatron_utils/delta_sync/ and simplify the test suite to 19 functions / 35 parameterized cases. The revised suite has not been rerun on remote hardware. Local pre-commit and parsing passed; all five merged unsupported parallel configurations were checked against the actual connection guard and rejected before setup. Historical results are not presented as a pass of this revised suite.

Real receiver regression on the pinned serving pair

Container cjy-vllm-acend-1019, same isolated directory. Applied both VIME dependency patches to clean exports of vLLM e6bfe03ad73a3330cb427885aa90d97a12e1c704 and vLLM-Ascend fd815467c221ee600137f6bdd53fe354d5e7c999:

• Sparse engine and routed-expert UT: 24 passed, exit code 0.
• PR18915's unchanged synthetic checkpoint E2E: 2 passed, no skips, exit code 0, 188.37 seconds total pytest wall time. TP2/DP1/EP2 and TP1/DP2/EP2 each used one trainer plus two inference NPUs (8,9,10). Every worker's complete parameter hashes and greedy generated token IDs matched native dense loading. Each case applied 37 patch records, including a repeated coordinate update. This is a small synthetic model, not pretrained-model accuracy or performance validation.
• Native/CANN extensions were reused from the installed container, not rebuilt. The first E2E attempt on 1,4,5 failed during worker initialization after other jobs occupied the selected memory; that failure log is retained separately.
• Patched engine and tests were byte-verified against the local patch. verified-results.json records the receiver results and input hashes. The earlier two-rank publication probe also passed all 12 cases; the full four-rank test above supersedes that probe for trainer publication coverage.

Raw logs: trainer-bridge.log, pinned-ut-final.log, pinned-e2e-free.log, their exit-code files, source manifests and both receiver validation.json files under the isolated directory above.

Repository checks

• Full pre-commit run --all-files --show-diff-on-failure passed after fixing two isort imports. The final source matches the checked files; the production implementation is unchanged by the formatting fix.

• NPU CI steps retain the environment update and test launch commands; automatic Bridge installation has been removed.

• The combined Ascend patch passed actual clean-index application on fd815467c; all 13 affected Python files parse. All 12 pre-existing compatibility patch sections remain unchanged from the preceding functional head, including IPC and worker fixes.

• All PR commits contain sign-offs; GitHub DCO passed after the historical sign-off and email cleanup. Added missing sign-offs to 23 historical commits (16 wangx700 and 7 A5 sync). With the author's confirmation, the 7 A5 sync commits retain their author name and use the existing valid email wangxin700@huawei.com instead of a5-sync@local. Author dates and every code tree remain unchanged by the metadata rewrite.

Remaining acceptance limits

• The current complete six-NPU VIME training/rollout job with a full seed, subsequent deltas and a fresh-checkpoint token/logprob comparison has not been rerun. The four-rank trainer regression and three-NPU receiver E2E establish their respective behaviors, not that complete combined job. Historical 30B measurements above predate this revision.
• Sparse jobs require Bridge to be provided by their runtime environment. Bridge v0.3.0 declares Transformers <5, so the tested NPU combination does not establish general dependency compatibility.
• New GitHub checks are triggered for the CI cleanup commit. Full local pre-commit passed; manual validation does not establish that all repository merge gates have passed.

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #35496290 | 📁 Comparing 14e6ca8 against latest (d7e40fe)

  🔍 Preview build  

69 files changed · + 10 added · ± 38 modified · - 21 deleted

+ Added

• examples/deepseek-r1.html
• examples/glm4-9B.html
• examples/glm4.7-30B-A3B.html
• examples/glm4.7-355B-A32B.html
• examples/glm5.2-744B-A40B.html
• examples/qwen3-30B-A3B.html
• examples/qwen3-4B.html
• get_started/NPU.html
• _examples_synced/retool/README.html
• _examples_synced/search-r1/README.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 28 more...

- Deleted

• library.html
• advanced/parallelism-memory.html
• advanced/policy-mismatch.html
• advanced/rl-systems.html
• advanced/rollout-scheduling.html
• advanced/straw.html
• blogs/index.html
• blogs/introducing_slime.html
• blogs/release_v0.1.0.html
• developer_guide/index.html
• and 11 more...

[gemini-code-assist]

Code Review

This pull request introduces a sparse HCCL weight transfer engine for Ascend NPUs, enabling checkpoint-coordinate sparse weight updates during RLHF. It adds the necessary engine classes, validation logic, profiling diagnostics, and comprehensive unit and end-to-end tests. The feedback highlights a potential runtime error when checking NPU formats on CPU parameters, suggests a more compatible way to retrieve the current NPU stream, and identifies a parameter naming bug in the Qwen3 MoE example script.

[gemini-code-assist]

Calling torch_npu.get_npu_format on parameters without checking if they are on NPU can cause runtime errors (e.g., when parameters are on CPU during testing or initialization). Additionally, torch_npu might not be imported globally in this file. It is safer to import torch_npu locally and only check the NPU format for parameters that are actually on an NPU device.

    @staticmethod
    def validate_checkpoint_weight_loader_view(layer):
        parameters = [getattr(layer, name, None) for name in ("w13_weight", "w2_weight")]
        if any(parameter is None for parameter in parameters):
            raise NotImplementedError("Checkpoint sparse updates require unsplit ND expert weights")
        import torch_npu
        for parameter in parameters:
            if parameter.device.type == "npu":
                if torch_npu.get_npu_format(parameter) not in (0, 2):
                    raise NotImplementedError("Checkpoint sparse updates require ND expert weights")

[gemini-code-assist]

Calling torch.npu.current_stream() without the device argument is more standard and compatible across different torch_npu versions. Since we are already inside the with torch.npu.device(device): context manager, the current device is already set, and current_stream() will automatically return the stream for that device.

            stream = torch.npu.current_stream()

[gemini-code-assist]

In standard Hugging Face / Transformers implementations of Qwen3 MoE, the parameter name is gate_up_proj.weight (with the .weight suffix), similar to gate_proj.weight. Accessing gate_up_proj directly without the .weight suffix will raise an AttributeError on real models.

                fused = model.get_parameter(f"{prefix}.gate_up_proj.weight")