gb200 Omni SingleController functional test fails: pixel_values loses its channel dim over the data plane #4208

gb200_L1_Functional_Tests_GB200_Megatron_Omni_Single_Controller has been failing on main since 2026-09-18. nemotron_omni_clevr_megatron_single_controller_1n2g.sh is disabled for now; the second test in that wrapper has not run since and is unverified.

Symptom

RuntimeError: Megatron inference prompt lengths do not match the training processor input lengths: inference=[0], training=[361]

That is two layers removed from the cause. inference=[0] comes from Megatron's placeholder request for a rejection before admission (dynamic_engine.py, prompt_tokens=torch.empty(0) + Status.FAILED). The real error is a rank-0 warning in the same log:

Request N rejected before admission: RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x256 and 768x1280)

Byte-identical across two independent runs, so not flaky.

Cause

768 = 3 * 16**2, 256 = 1 * 16**2 — pixel_values reaches the vision tower with 1 channel instead of 3. patch_dim=16 is correct.

_patchify_segments in nemo_rl/data/multimodal_utils.py computes the patch width from whatever channel count it receives:

.reshape(count * rows * columns, channels * patch_dim * patch_dim)

The pre-patchified branch a few lines above validates against patch_features = 3 * patch_dim**2 and raises; the 4-D pixel branch does not. A 1-channel segment passes through silently and fails much later inside Megatron.

Scope

The non-SingleController sibling (gb200_L1_Functional_Tests_GB200_Megatron_Omni) passes with the same model and the same patchify path. The difference is that SingleController sends multimodal payloads across the data plane, which points at the wire round-trip rather than the processor.

Bisect

#4105 is the last commit whose CI actually exercised this job. After it, only #3898 and #4106 touched nemo_rl/data_plane/ or multimodal_utils.py, and both merged at CI:Lfast, which skips gb200 functional tests entirely (cicd-functional-tests-gb200 is gated on contains('L1 L2', test_level)).

#4106 is the stronger suspect: it rewrote the multimodal wire tuple in nemo_rl/data_plane/worker_mixin.py, replacing pad_to_max_shape with preprocess_mode + preprocess_kwargs, and that is exactly the path SingleController uses and the non-SC path does not. Not proven — confirming it needs the job run on 54b6aae96 vs a6845608e, which needs gb200.

Suggested follow-ups

1. Fix the channel loss in the data-plane round-trip.
2. Make the 4-D branch of _patchify_segments validate channels == 3 like the 3-D branch does, so this fails at the source instead of as an opaque matmul error two layers down.
3. Consider whether a PR touching the multimodal data plane should be mergeable at CI:Lfast.