Skip to content

feat(data_plane): add Mooncake GDR support - #3501

Open
zyzhou5 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
zyzhou5:zezhou/tq-mooncake-gdr
Open

feat(data_plane): add Mooncake GDR support#3501
zyzhou5 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
zyzhou5:zezhou/tq-mooncake-gdr

Conversation

@zyzhou5

@zyzhou5 zyzhou5 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the minimal NeMo-RL configuration plumbing needed to enable TransferQueue v0.1.9's Mooncake GPUDirect RDMA (GDR) path. TransferQueue v0.1.9 is already pinned on current main by #3423.

  • adds optional data_plane.use_gdr; omitting it defaults to false, so existing recipes are unchanged;
  • passes protocol: rdma and use_gdr: true to Mooncake when GDR is enabled;
  • leaves RDMA device selection local to each Mooncake client unless explicitly configured;
  • requires CUDA to be initialized before a GPU worker attaches its GDR-enabled TQ client, avoiding TQ's CPU fallback;
  • retains NeMo-RL's existing transfer/decode paths and TQ's default staging-buffer size; and
  • documents how users select TCP, CPU RDMA, or GPU RDMA.

Persistent queued Mooncake objects remain in CPU RAM. GDR changes the CUDA client's transfer path by using a registered GPU staging buffer; it does not turn the persistent queue into GPU storage.

User configuration

data_plane:
  enabled: true
  impl: transfer_queue
  backend: mooncake_cpu
  use_gdr: true

Relationship to #2935

This PR does not stack on or require #2935. The TQ dependency update is already on main via #3423. The runtime-environment cleanup proposed by #2935 is separate and is not included here.

Relationship to #3837

To use GDR on the put side instead of falling back to CPU RDMA, we need #3837 to get merged since TQ must be initialized inside the generation worker after CUDA is initialized so it selects GDR for PUTs.

@zyzhou5
zyzhou5 requested review from a team as code owners August 5, 2026 18:14
@copy-pr-bot

copy-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@zyzhou5
zyzhou5 marked this pull request as draft August 5, 2026 18:14
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Aug 5, 2026
@zyzhou5
zyzhou5 force-pushed the zezhou/tq-mooncake-gdr branch from 8fae9a1 to 6e39a2b Compare August 7, 2026 01:57
@zyzhou5 zyzhou5 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Aug 7, 2026
@zyzhou5
zyzhou5 marked this pull request as ready for review August 7, 2026 22:27
@zyzhou5
zyzhou5 force-pushed the zezhou/tq-mooncake-gdr branch from 6e39a2b to d2edefb Compare August 7, 2026 22:37
@zyzhou5

zyzhou5 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d2edefb

@zyzhou5
zyzhou5 force-pushed the zezhou/tq-mooncake-gdr branch from d2edefb to 31f9117 Compare August 17, 2026 21:13
@zyzhou5
zyzhou5 requested a review from a team as a code owner August 17, 2026 21:13
@zyzhou5

zyzhou5 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 31f9117

@github-actions github-actions Bot added the CI Relating to CI label Aug 17, 2026
Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>
@zyzhou5
zyzhou5 force-pushed the zezhou/tq-mooncake-gdr branch from 31f9117 to 5bdc8e7 Compare August 26, 2026 17:42
@github-actions github-actions Bot removed the CI Relating to CI label Aug 26, 2026
@zyzhou5

zyzhou5 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5bdc8e7

Signed-off-by: Zeyu Zhou <zezhou@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant