Skip to content

feat(mooncake): enable Ascend pinned receive and loader-side H2D - #1071

Open
curnane-lab wants to merge 1 commit into
sgl-project:mainfrom
curnane-lab:perf/mooncake-ascend-receive
Open

curnane-lab wants to merge 1 commit into
sgl-project:mainfrom
curnane-lab:perf/mooncake-ascend-receive

Conversation

@curnane-lab

@curnane-lab curnane-lab commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

The pinned receive pool and its device copy path were CUDA-only. On NPU, the trainer received pageable host tensors and paid a blocking H2D copy on the training thread. This PR enables Ascend pinned receive and loader-side H2D while preserving the existing CPU and CUDA behavior.

This is an enablement/correctness change, not a demonstrated throughput win.

Modifications

  • consumer_device() reports the rank's NPU for pinned consumers when the trainer device is NPU (LOCAL_RANK is mirrored, as for CUDA), so pooled reads finish H2D in loader workers. Explicit CPU trainers stay on the host.
  • The pinned pool allocates through torch_npu's pinned host allocator when Ascend is available and still degrades to pageable host buffers on accelerator-less hosts.
  • copy_stream() and the slot-copy path dispatch to the device type's stream module (torch.cuda / torch.npu). Slot release waits for copy completion before reuse, and NPU outputs record the consumer stream when supported.
  • FeatureDataLoader records an NPU ready event after transform/collate. Consumption performs wait_event() and record_stream() on the consumer stream, covering both the execution dependency and allocator lifetime after collate.

Validation

Validated PR head: d58e652.

Clean PR head:

  • 95 passed, 7 skipped, 1 xfailed with backend autoload disabled.
  • 8 passed on Ascend 910B3.
  • black and isort checks passed.
  • Hardware checks covered exact NPU-delivered values with host integer features, A/B immediate slot-reuse safety, release followed by KeyError, and delayed-collate handoff to a separate NPU consumer stream.

Supplementary test-only validation:

  • d58e652 + this test-only patch, SHA-256 d1f14c08b62ca3e59cb13f2113dfe9b684e61600004a761df6acf79ef1aff99d, adds two cases to the PR's eight NPU tests and passed 10/10 on 910B3. This is not a clean-PR-head test count.
  • The positive handoff tests passed, but earlier negative controls also passed on pre-fix code; they do not demonstrate race-detection capability.

Benchmark & Profiling

  • Training A/B (managed-local, 8 steps, same seed, main+N1 vs. this change): per-step losses were bit-identical. Steady-state step time and train_compute_time_s were unchanged within noise; data_wait_time_s was already zero after step 1 on both paths. Existing loader prefetch already hid receive latency at this scale, so no end-to-end speedup is claimed.
  • A mock-backend receive/copy microbenchmark on clean d58e652 used _FakeMooncakeStore with a real NPU, not TCP/RDMA. For an 8.015625 MiB payload, it measured only get(npu:1) + torch.npu.synchronize(npu:1), with 10 warmups and 100 alternating-order iterations per mode. Median latency was 1.936 ms pageable vs. 2.220 ms pinned (0.872x pageable/pinned ratio). This microbenchmark shows no receive speedup at this size; raw samples and metadata are archived.

Validation archive and reproduction notes.

Checklist

The pinned receive pool and its device copy path were CUDA-only: on NPU
the trainer received pageable host tensors and paid a blocking H2D per
micro-step on the training thread (~5 ms per 40 MiB sample on 910B3,
measured through MooncakeFeatureStore over TCP).

- consumer_device() reports the rank's NPU for pinned consumers when the
  trainer device is npu (LOCAL_RANK mirrored, as for CUDA), so pooled
  reads finish their H2D in the loader workers instead of the training
  thread. Explicit CPU trainers stay on the host.
- The pinned pool allocates through torch_npu's pinned host allocator
  when Ascend is available, and still degrades to pageable host buffers
  on accelerator-less hosts.
- copy_stream() and the slot-copy path dispatch to the device type's
  stream module (torch.cuda / torch.npu). Slot release still waits for
  copy completion before reuse, and NPU outputs record the consumer
  stream when the tensor supports it.
- Prefetched NPU batches record a ready event after transform/collate.
  Consumption waits on that event and records the consumer stream so
  later worker operations and allocator lifetime are handed off safely.

This is an enablement/correctness change for the default pinned path,
not a demonstrated throughput win. Measured on 910B3:

- Microbenchmark (40 MiB DFlash samples, TCP): pinned host H2D
  17.3 GiB/s vs 11.5 GiB/s pageable; get(npu) ~15 ms vs ~16.5 ms
  pageable serially.
- Training A/B (managed-local, 8 steps, same seed, main+N1 vs +this
  change): per-step losses bit-identical, steady-state step time and
  train_compute_time_s unchanged within noise; data_wait_time_s is
  already zero after step 1 on both paths, i.e. the existing loader
  prefetch already hides receive latency at this scale. No end-to-end
  speedup is claimed; the benefit is removing the trainer-thread H2D
  for large-payload or slower-transport deployments.

Hardware-verified: exact values with NPU delivery and host integer
features, A/B immediate slot-reuse safety, release -> KeyError, and a
delayed-collate handoff from a loader worker to a separate NPU consumer
stream. Dispatch and handoff tests use device stand-ins and run without
torch_npu; real-backend tests mirror the CUDA receive suite.

Validation: 95 passed, 7 skipped, 1 xfailed with backend autoload
disabled; 8 passed on 910B3. black/isort checks pass.
@curnane-lab
curnane-lab force-pushed the perf/mooncake-ascend-receive branch from ace83e9 to d58e652 Compare November 3, 2026 06:38
@curnane-lab
curnane-lab marked this pull request as ready for review November 4, 2026 02:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant