Repository navigation
feat(mooncake): enable Ascend pinned receive and loader-side H2D - #1071
Open
curnane-lab wants to merge 1 commit into
Open
curnane-lab wants to merge 1 commit into
curnane-lab wants to merge 1 commit into
Conversation
The pinned receive pool and its device copy path were CUDA-only: on NPU the trainer received pageable host tensors and paid a blocking H2D per micro-step on the training thread (~5 ms per 40 MiB sample on 910B3, measured through MooncakeFeatureStore over TCP). - consumer_device() reports the rank's NPU for pinned consumers when the trainer device is npu (LOCAL_RANK mirrored, as for CUDA), so pooled reads finish their H2D in the loader workers instead of the training thread. Explicit CPU trainers stay on the host. - The pinned pool allocates through torch_npu's pinned host allocator when Ascend is available, and still degrades to pageable host buffers on accelerator-less hosts. - copy_stream() and the slot-copy path dispatch to the device type's stream module (torch.cuda / torch.npu). Slot release still waits for copy completion before reuse, and NPU outputs record the consumer stream when the tensor supports it. - Prefetched NPU batches record a ready event after transform/collate. Consumption waits on that event and records the consumer stream so later worker operations and allocator lifetime are handed off safely. This is an enablement/correctness change for the default pinned path, not a demonstrated throughput win. Measured on 910B3: - Microbenchmark (40 MiB DFlash samples, TCP): pinned host H2D 17.3 GiB/s vs 11.5 GiB/s pageable; get(npu) ~15 ms vs ~16.5 ms pageable serially. - Training A/B (managed-local, 8 steps, same seed, main+N1 vs +this change): per-step losses bit-identical, steady-state step time and train_compute_time_s unchanged within noise; data_wait_time_s is already zero after step 1 on both paths, i.e. the existing loader prefetch already hides receive latency at this scale. No end-to-end speedup is claimed; the benefit is removing the trainer-thread H2D for large-payload or slower-transport deployments. Hardware-verified: exact values with NPU delivery and host integer features, A/B immediate slot-reuse safety, release -> KeyError, and a delayed-collate handoff from a loader worker to a separate NPU consumer stream. Dispatch and handoff tests use device stand-ins and run without torch_npu; real-backend tests mirror the CUDA receive suite. Validation: 95 passed, 7 skipped, 1 xfailed with backend autoload disabled; 8 passed on 910B3. black/isort checks pass.
curnane-lab
force-pushed
the
perf/mooncake-ascend-receive
branch
from
November 3, 2026 06:38
ace83e9 to
d58e652
Compare
curnane-lab
marked this pull request as ready for review
November 4, 2026 02:12
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pinned receive pool and its device copy path were CUDA-only. On NPU, the trainer received pageable host tensors and paid a blocking H2D copy on the training thread. This PR enables Ascend pinned receive and loader-side H2D while preserving the existing CPU and CUDA behavior.
This is an enablement/correctness change, not a demonstrated throughput win.
Modifications
consumer_device()reports the rank's NPU for pinned consumers when the trainer device is NPU (LOCAL_RANKis mirrored, as for CUDA), so pooled reads finish H2D in loader workers. Explicit CPU trainers stay on the host.copy_stream()and the slot-copy path dispatch to the device type's stream module (torch.cuda/torch.npu). Slot release waits for copy completion before reuse, and NPU outputs record the consumer stream when supported.FeatureDataLoaderrecords an NPU ready event after transform/collate. Consumption performswait_event()andrecord_stream()on the consumer stream, covering both the execution dependency and allocator lifetime after collate.Validation
Validated PR head:
d58e652.Clean PR head:
KeyError, and delayed-collate handoff to a separate NPU consumer stream.Supplementary test-only validation:
d58e652+ this test-only patch, SHA-256d1f14c08b62ca3e59cb13f2113dfe9b684e61600004a761df6acf79ef1aff99d, adds two cases to the PR's eight NPU tests and passed 10/10 on 910B3. This is not a clean-PR-head test count.Benchmark & Profiling
train_compute_time_swere unchanged within noise;data_wait_time_swas already zero after step 1 on both paths. Existing loader prefetch already hid receive latency at this scale, so no end-to-end speedup is claimed.d58e652used_FakeMooncakeStorewith a real NPU, not TCP/RDMA. For an 8.015625 MiB payload, it measured onlyget(npu:1) + torch.npu.synchronize(npu:1), with 10 warmups and 100 alternating-order iterations per mode. Median latency was 1.936 ms pageable vs. 2.220 ms pinned (0.872x pageable/pinned ratio). This microbenchmark shows no receive speedup at this size; raw samples and metadata are archived.Validation archive and reproduction notes.
Checklist