feat(token-capture): carry media through Megatron capture rollouts by lauradang · Pull Request #4332 · NVIDIA-NeMo/RL

lauradang · 2026-09-29T16:42:25Z

What does this PR do ?

Builds on #4191 (vLLM VLM media capture) and #4129 (MInf ledger capture), both now on main, which is this PR's base. This supersedes lauradang#4, which targeted a fork copy of #4191's pre-rebase branch.

This PR adds media capture on the Megatron Inference generation backend, on top of #4191's shared media contract: the same TQ media columns, validate_media_tensors, batched fetch_media, and finalizer.

Token capture on the Megatron backend was text-only. Receipt rows carried no pixel_values, so a VLM capture run trained image-blind, and Gym refused Megatron capture requests with image or video parts outright.

Expanded-prefix splicing. Megatron-LM now owns multimodal prefix stitching ([inference] Fix multimodal prefix-stitching. NVIDIA/Megatron-LM#7598). When earlier turns carried media, the chat endpoint writes _prefix_media_count into offload_params. TQMegatronPromptPreparer splices the exact expanded staged chain into the compact render, checks the endpoint's media count against what the chain staged, and answers with _prefix_expanded_token_count (the chain's length). The engine splits that prefix back off and expands only the media placeholders after it. There is one token space end to end: the staged token_ids_delta is both what the trainer reads and what the next turn splices.
EOS sets. The endpoint now ships the model's EOS ids as a list. splice_prefix_tokens accepts a set, counts any member as a turn boundary, and keeps the EOS the model actually emitted, so the spliced prompt still starts with the model's exact tokens (Gym verifies that prefix). Callers that pass one id (vLLM, TRT-LLM, Dynamo) behave exactly as before.
Media on the call row, as per-call deltas. Every chat request carries the whole conversation, so the engine hands the stager media_tensors for every image in the prompt. The preparer records how many media items the parent chain already staged (offload_params["ng_capture_minf"]["media_prev_count"], counted from the parents' small media columns, never pixels). The preparer also forwards the (height, width) of each of those items (media_prev_sizes); the stager compares the leading items of the engine's media_tensors against them and fails the call as capture_failed on any difference, so a later turn that resends an earlier image at a different size cannot leave the trainer with the earlier turn's pixels. Placeholder-token checks stay vLLM-only, since Megatron Inference reports no spans. The stager then cuts the engine's packed tensors at that boundary (slice_media_tensors, now next to the stager since the vLLM path no longer uses it) and stages the rest on the same staging key as extra columns (MEDIA_STAGING_FIELDS), the way routed experts ride the row.
Finalizer. RolloutReassembler walks the terminal chain, checks each call's media columns against that call's staged geometry (rejecting the group with media_chain_identity, media_chain_incompatible, or invalid_media_columns), concatenates the deltas in chain order as it does for tokens, and publishes the engine's packed patches as pixel_values with imgs_sizes and num_frames. The Megatron-Bridge Omni model accepts the [total_patches, C*P*P] layout directly, so training projects exactly the pixels the policy generated against.
Mixed text/media batches fail loudly. TQ answers a batch fetch with only the columns every requested key wrote, so one key without pixel_values used to strip media from the whole batch and the policy trained image-blind with no error. The TQ adapter's get_samples now raises KeyError when a requested column is missing (matching LocalDataPlaneClient.get_samples), and under capture_media the finalizer drops a group in which no valid rollout carried media. This applies on both backends: a vLLM capture run on a dataset that mixes text-only and media prompts now loses its all-text groups too, where before this PR they were published without a pixel_values column. The drop shrinks that step (counted in dropped_prompt_groups); it is not replaced, even under on_dropped_prompt="replace". The adapter check is shared, so it also changes behaviour outside Megatron capture: non-capture single-controller VLM runs on a mixed text/media dataset (or mixed image/video keys) now fail at the train fetch instead of silently training without media, and vLLM capture gets the same check.
Setup. Registers the media columns for multimodal capture runs on either backend (drops feat(capture): capture vLLM VLM media alongside token deltas #4191's vLLM-only media guard). The pinned megatron-core carries the prefix-stitching keys the preparer imports (see Dependencies). grpo.deduplicate_multimodal_data is rejected under capture.
Sampling defaults. Gym chat requests omit top_k, and newer Megatron-LM fills omitted sampling fields from the model's generation_config.json (e.g. Qwen3's top_k=20), which would sample off-policy. The Megatron worker now passes the policy's temperature / top_p / top_k as the HTTP server defaults. This fix is independent of media and also applies to text capture.

docs/design-docs/token-capture-ledger.md gains a "Multimodal rollouts on Megatron Inference" section.

Worked example

A two-turn VLM rollout under capture, with toy values: the media placeholder is token 99, EOS is 2, one 4x4 image expands to three media tokens and to four packed patches of 12 features. Turn 1 sends text plus one image. Turn 2 sends the whole history plus a second image; the chat template re-tokenizes the turn-1 answer as 13 although the model generated 12, which is what the splice corrects. Each box is a process; the label says whose code runs in it.

sequenceDiagram
    box Gym server (Gym code)
        participant G as Agent + capture middleware<br/>+ MegatronWorkerCaptureHandler
    end
    box Megatron HTTP endpoint, in RL's Megatron worker (Megatron code)
        participant EP as chat_completions.py
    end
    box Engine TP0/PP0 (Megatron loop, RL+Gym hooks)
        participant P as TQMegatronPromptPreparer
        participant S as TQMegatronTokenStager<br/>+ Gym adapter
    end
    box Engine, all ranks (Megatron)
        participant E as add_request / _build_vlm_request
    end
    participant TQ as TransferQueue
    participant L as Gym ledger
    box RL actors (RL code)
        participant F as RolloutReassembler
        participant T as Trainer
    end

    Note over G,L: Turn 1 - text + image1
    G->>EP: chat req, offload_params.ng_capture = {c1, text, prev_len 0}
    EP->>P: compact [80,99,81] + image bytes (via DP coordinator)
    P->>E: text mode, pass through, broadcast
    E->>E: decode img1 to imgs[1,4,12], expand to [80,99,99,99,81], gen [12,2]
    E->>S: payload {prompt/gen ids, media_tensors (image1)}
    S->>TQ: r0/c1 - token delta [80,99,99,99,81,12,2], media cols (image1)
    S->>EP: ng_commit_coords {r0/c1, cum_len 7} via engine reply
    EP->>G: "a cat" + coords
    G->>L: CallRecord c1

    Note over G,L: Turn 2 - history + image2
    G->>L: resolve_parent finds c1
    G->>EP: ng_capture = {c2, token_in, prev_len 7, staging_chain [r0/c1]}
    EP->>P: compact [80,99,81,13,2,20,99,21], template_prefix [80,99,81,13,2], eos [2], _prefix_media_count 1
    P->>TQ: fetch_prefix_chains([r0/c1])
    TQ->>P: expanded [80,99,99,99,81,12,2], media_count 1
    P->>E: splice to [80,99,99,99,81,12,2,20,99,21], _prefix_expanded_token_count 7, required_prefix = expanded, media_prev_count 1
    E->>E: keep prefix[:7], expand suffix [20,99,21] with image2 only, prompt [80,99,99,99,81,12,2,20,99,99,99,21], gen [30,2]
    E->>S: payload {prompt/gen ids, media_tensors (image1 + image2)}
    S->>S: Gym checks prompt[:7] == expanded chain, delta [20,99,99,99,21,30,2], slice media at 1 (image2 only)
    S->>TQ: r0/c2 token delta + media cols (image2)
    S->>EP: coords {r0/c2, prev_len 7, cum_len 14}
    EP->>G: answer + coords
    G->>L: CallRecord c2 (parent c1)

    Note over F,T: Rollout end
    F->>L: manifest to receipt {terminal c2, [c1, c2]}
    F->>TQ: fetch rows for 14 expanded tokens, fetch_media for r0/c1 and r0/c2, concat to imgs[1,8,12]
    F->>F: imgs_sizes cols == extras OK, pixel_values[8,12]
    F->>TQ: publish canonical row, clear r0/c1 and r0/c2
    T->>TQ: read input_ids + pixel_values, six 99s match six embeddings

Why the prefix length matters: if the engine were handed [80,99,99,99,81,12,2,20,99,21] without _prefix_expanded_token_count, it would treat all four 99s as placeholders for the request's two images and reject the request on the placeholder count. With the count, it expands only the 99 after position 7, and image 1's embeddings fill the three already-expanded slots.

Dependencies

Megatron-LM: [inference] Fix multimodal prefix-stitching. NVIDIA/Megatron-LM#7598 (expanded-prefix stitching, which also drops OffloadedRequestPayload.compact_prompt_token_ids and sends EOS ids as a list). It reaches RL through the Megatron-Bridge bump to ec835530e (ci: Bump MBridge to ec835530efeff71b55ec015f36bcc0c33b8b52b7. #4363, merged into this branch), which nests Megatron-LM b4d72b79c. test_offloaded_payload_exposes_multimodal_capture_fields pins the payload field and stitching keys on that pin, and the preparer tests import the real keys instead of patching them in.
Gym: pinned to Gym main 2c10237cc, which includes feat(token-capture): admit multimodal Megatron capture requests Gym#3787 (admit multimodal Megatron capture requests) and requires ray[default]>=2.58.0; RL's lock is already on ray 2.58.0 from upstream main, and uv.lock is regenerated for the new Gym metadata (no resolved versions change). test_megatron_capture_handler_admits_media_requests sends image and video parts through the Megatron capture handler, so a re-pin to a Gym without fix(sc): apply reward scaling before advantage estimation #3787 fails RL CI.

Testing

Local (macOS, in-process TransferQueue simple backend, Gym @ a2ef85b0e), after merging #4191 at 691b9359b:

test_openai_server_utils.py, test_setup.py, test_captured_media.py, test_multimodal_dict.py: 221 passed
test_tq_token_sink.py + test_rollout_reassembler.py (-m nemo_gym, megatron-core inference_request stubbed because megatron-core cannot install on macOS): 113 passed, including the vLLM/Megatron worked-example parity test and the expanded-prefix preparer cases
test_megatron_token_capture_hosting.py and the mcore tests in test_megatron_generation_parse.py (including the new sampling-defaults test) need real megatron-core; CI only

End-to-end Nemotron Omni video run (earlier compact-chain revision)

This run used the earlier revision of this PR, which spliced a compact chain against a Megatron-LM fork. The expanded-prefix revision needs a fresh run once RL's Megatron-Bridge pin carries NVIDIA/Megatron-LM#7598. The same bump also changes Nemotron Omni prompt rendering, so compare against a new baseline.

Completed a 42-step, 8-node × 4-GPU VSTAT GRPO run in 19m14s with this PR stack:

RL: 79fa6856b5e40a456c8d932f434a490c9682e296
Gym: fafecf7c71ef63d027bb2197895e93c9effced4a
Megatron-LM: 0aa6cce27f75c2a4cf733b8eeb0e8180dc451707
Megatron-Bridge: 8e4ccc34b67a79bca73cb76242479f9090856ff4

Run: PR stack (jtg9a7rp) · cye baseline (o4ed2p8n) · W&B overlay

The reward curves track closely: Pearson correlation 0.9169, mean absolute error 0.0486; mean reward over all 42 steps was 0.1692 for this PR stack vs. 0.1592 for the baseline. Steps 23–42 averaged 0.2229 vs. 0.2281.

The chart uses nemo_rl/step on the x-axis (not W&B's synthetic STEP). Duplicate history entries from continuation were reduced to the final value for each nemo_rl/step.

Important effective configuration:

policy:
  model_name: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
  is_vlm: true
  megatron_cfg:
    freeze_vision_model: false
    freeze_vision_projection: false
  generation:
    backend: megatron
    mcore_generation_config:
      megatron_inference_wrapper: megatron.core.inference.model_inference_wrappers.multimodal.nemotron_omni_inference_wrapper.NemotronOmniInferenceWrapper
      parsers: [nemotron-v3-reasoning, qwen3-coder-tool]
      video_num_frames: 16
      video_temporal_patch_size: 2
      video_target_num_patches: 1024

data:
  train:
    data_path: /opt/nemo-rl/workspace/datasets/vstat-8n4g/train-gym.jsonl
  default:
    dataset_name: NemoGymDataset
    processor: nemo_gym_data_processor
    num_frames: 16
    video_sampling_style: nemotron_vl
    video_temporal_patch_size: 2
    video_target_num_patches: 1024
    video_maintain_aspect_ratio: true

grpo:
  deduplicate_multimodal_data: false

data_plane:
  enabled: true
  impl: transfer_queue
  backend: simple

token_capture:
  enabled: true

checkpointing:
  enabled: true
  save_period: 10
  save_data_plane: true

The dataset rows contain real local video paths. The step-40 native TQ checkpoint contains pixel_values, imgs_sizes, and num_frames (plus their row-shape metadata) in replay_buffer_metadata.pt, and the same media columns in the TQ storage-unit pickle. A continuation read those columns from TQ and reached Nemotron Omni's _encode_images(images, ..., num_frames) path during policy logprob computation.

🤖 Generated with Claude Code

copy-pr-bot · 2026-09-29T16:42:30Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-29T16:44:12Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 0f7dd4c (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of pranav/vlm_capture_support branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-29T23:27:36Z

✅ Submodule Fast-Forward Check Results

Check based on commit: f5d3810 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of pranav/vlm_capture_support branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-30T01:37:39Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 7745197 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of pranav/vlm_capture_support branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-30T01:59:30Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 27cbae7 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of pranav/vlm_capture_support branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-30T06:13:59Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 27cbae7 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of pranav/vlm_capture_support branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-30T06:25:22Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 86a7b18 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2026-09-30T06:43:15Z

✅ Submodule Fast-Forward Check Results

Check based on commit: a9332d4 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

lauradang · 2026-09-30T06:43:28Z

/ok to test a9332d4

github-actions · 2026-09-30T06:51:53Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 62f30b1 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Token capture on the Megatron Inference backend was text-only: receipt rows carried no pixel_values, so a VLM capture run trained image-blind and Gym refused Megatron capture requests with image or video parts. This adds media capture on the Megatron backend on top of the shared vLLM media contract (TQ media columns, validate_media_tensors, batched fetch_media, and the finalizer). - Expanded-prefix splicing: TQMegatronPromptPreparer splices the exact expanded staged chain into the compact render, checks the endpoint's media count against what the chain staged, and answers with the expanded prefix length so the engine expands only the media placeholders after it. - EOS sets: splice_prefix_tokens accepts a set of EOS ids, counts any member as a turn boundary, and keeps the EOS the model emitted. - Media as per-call deltas: the preparer records how many media items the parent chain already staged; the stager slices the engine's packed tensors at that boundary and stages the rest on the staging key. - Finalizer: RolloutReassembler validates each call's media columns against its staged geometry, concatenates deltas in chain order, and publishes packed patches as pixel_values with imgs_sizes/num_frames. - Setup: registers media columns for multimodal runs on either backend, requires megatron-core media payload fields for multimodal Megatron capture, and rejects grpo.deduplicate_multimodal_data. - Sampling defaults: the Megatron worker passes the policy's temperature/top_p/top_k as HTTP server defaults so omitted request fields are not filled from the model's generation_config.json. Documents multimodal rollouts on Megatron Inference in the token-capture ledger design doc. Signed-off-by: Laura Dang <laurad@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions · 2026-09-30T07:05:38Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2ed7f1b (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Remove the multi-line explanatory comment blocks the media-capture change added outside docstrings, keep single-line comments and the parametrize legend, and shrink deferred-import justifications to one line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

github-actions · 2026-09-30T07:18:56Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 0d5c0f8 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Moving it above the new PrefixChains helpers made the PR diff show the whole function as rewritten. Put it back where it was so the diff shows only the additions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

github-actions · 2026-09-30T21:50:07Z

✅ Submodule Fast-Forward Check Results

Check based on commit: c270320 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

terrykong

Follow-up to my earlier review. All 10 resolved threads were checked against the code at 27d5432: 9 are fixed. The empty-media-rows change behind the last one works, but it turned the Lint job red, so there is one inline comment with a one-click fix.

The PR also has merge conflicts with main. Please rebase so the full CI can run.

Generated by Claude Code

terrykong

lgtm @lauradang . all that's left is to resolve the linting issues and rebase to resolve the conflicts

Resolve conflicts with the zero-segment capture client change: Megatron capture builds its client with segment_size=0 and still pins the vision pixel dtype; setup starts storage units before token-capture setup. Also type the empty media segment lists for pyrefly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

github-actions · 2026-10-12T17:53:21Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2231b67 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Upstream now passes the training sampling profile to the Megatron HTTP server through _http_sampling_defaults, which this branch also did inline; keep upstream's helper and drop the inline copy so start_text_gen_server no longer receives the default_* keywords twice. The parse tests keep the shared stub-server helper and upstream's sampling-defaults cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

lauradang · 2026-10-12T20:10:31Z

/ok to test 2526ab4

github-actions · 2026-10-12T20:12:14Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2526ab4 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

…r pyrefly pyrefly types `[t] * n` as list[Tensor] even under a list[Tensor | None] annotation, so PackedTensor rejected it; a comprehension passed straight to PackedTensor is checked against the parameter type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

lauradang · 2026-10-12T20:29:18Z

/ok to test de162d9

github-actions · 2026-10-12T20:30:40Z

✅ Submodule Fast-Forward Check Results

Check based on commit: de162d9 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

…agent The pinned Gym rejects agent instances that no environment server references, so the sanity fixture failed at spinup. Front the agent with the legacy_agent environment server, matching Gym's own example_multi_step config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>

lauradang · 2026-10-13T16:35:01Z

/ok to test 621842a

github-actions · 2026-10-13T16:36:39Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 621842a (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

…e-bump

lauradang · 2026-10-13T18:38:30Z

/ok to test 502d2e5

github-actions · 2026-10-13T18:40:10Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 502d2e5 (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

lauradang · 2026-10-14T07:44:39Z

/ok to test 8f7e92a

github-actions · 2026-10-14T07:44:43Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 8f7e92a (PR #4332 from laurad/pr4129-multimodal-capture)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions Bot added the Documentation Improvements or additions to documentation label Sep 29, 2026

lauradang mentioned this pull request Sep 29, 2026

feat(token-capture): carry media through Megatron capture rollouts lauradang/RL#4

Closed

lauradang mentioned this pull request Sep 29, 2026

feat(token-capture): admit multimodal Megatron capture requests NVIDIA-NeMo/Gym#3787

Merged

lauradang force-pushed the laurad/pr4129-multimodal-capture branch from 7bf8f11 to f5d3810 Compare September 29, 2026 23:25

terrykong deleted the branch NVIDIA-NeMo:main September 30, 2026 05:07

terrykong closed this Sep 30, 2026

lauradang reopened this Sep 30, 2026

lauradang changed the base branch from pranav/vlm_capture_support to main September 30, 2026 06:13

lauradang added the CI:L1 Run doctests, unit tests, and functional tests label Sep 30, 2026

lauradang force-pushed the laurad/pr4129-multimodal-capture branch from 62f30b1 to 2ed7f1b Compare September 30, 2026 07:04

lauradang requested review from pthombre and terrykong September 30, 2026 21:54

lauradang assigned macandro96 and lauradang Sep 30, 2026

terrykong reviewed Oct 12, 2026

View reviewed changes

Comment thread nemo_rl/experience/rollout_reassembler.py Resolved

terrykong enabled auto-merge (squash) October 12, 2026 05:36

terrykong previously approved these changes Oct 12, 2026

View reviewed changes

lauradang dismissed terrykong’s stale review via 2231b67 October 12, 2026 17:51

terrykong previously approved these changes Oct 12, 2026

View reviewed changes

lauradang mentioned this pull request Oct 12, 2026

Stage request payloads off the dynamic engine loop NVIDIA/Megatron-LM#8169

Merged

3 of 7 tasks

lauradang dismissed terrykong’s stale review via 621842a October 13, 2026 16:35

Merge remote-tracking branch 'upstream/main' into laurad/pr4332-bridg…

502d2e5

…e-bump

Merge branch 'main' into laurad/pr4129-multimodal-capture

8f7e92a

terrykong approved these changes Oct 14, 2026

View reviewed changes

terrykong merged commit 3911e10 into NVIDIA-NeMo:main Oct 14, 2026
257 of 265 checks passed

terrykong mentioned this pull request Oct 22, 2026

feat(rollout): replay MInf routing indices #4307

Merged

yfw mentioned this pull request Oct 23, 2026

fix(opd): score captured rollouts before replay readiness #4551

Merged

4 tasks done

ZhiyuLi-Nvidia mentioned this pull request Oct 26, 2026

fix(data-plane): preserve sparse media in mixed TransferQueue batches #4684

Merged

3 of 4 tasks

yuki-97 mentioned this pull request Oct 29, 2026

fix: Add chat completion parity tests for TRTLLM v.s. vLLM and fix disparities #3422

Merged

4 tasks done

Conversation

lauradang commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Worked example

Dependencies

Testing

End-to-end Nemotron Omni video run (earlier compact-chain revision)

Uh oh!

copy-pr-bot Bot commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 29, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 29, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

lauradang commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 30, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

terrykong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

terrykong left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 12, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

lauradang commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

lauradang commented Oct 12, 2026

Uh oh!

github-actions Bot commented Oct 12, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

lauradang commented Sep 29, 2026 •

edited

Loading