[WS1][CUDA][MiniMax-H3] ws1_one_h3_block + full-model qualification: block 0 node-by-node and 50-block/generation gates (RFC #420 stack 10/10)

[fusheng-ji]

RFC #420 stack 10/10 (base test-h3, merge bottom-up): #483 → #484 → #485 → #486 → #487 → #488 → #489 → #493 → #494 → #539. This PR merges after #494; until the lower PRs merge, GitHub also shows their commits. This PR's own diff.

What

RFC #420 row ws1_one_h3_block: one real H3 block, forward and backward, with every intermediate and a first-drift report.

• The block. Block 0 of MiniMax-H3@42ed227 runs as a 17-node graph, from the AdaLN projection to the second gated residual. It runs on packed FL2VA layouts whose position_ids and tags are bitwise equal to the pinned diffusers pipeline's build_packed_sequence; the hashes are pinned in a CPU test.
• Three executions per node. Each node runs as:
  • the diffusers replay (op for op, no diffusers import);
  • the RL-Kernel binding, both chained and isolated (fed diffusers' inputs);
  • an FP32 golden.
• Ownership. Each node names the RFC row that owns its arithmetic, and whether it is bound to that row's operator, an interim deterministic operator, or the provider replay.
• Forward report, per node:
  • repeat and batch-row bitwise checks;
  • chained and isolated agreement with diffusers (including the first differing packed row);
  • candidate and diffusers error against the golden;
  • the node's sha256;
  • first_{,isolated_,repeat_,batch_}drift.
• Backward report:
  • gradients of the block input, temb and all 12 block parameters (repeat, error vs golden);
  • the gradient reaching every node (repeat, batch row), with first_grad_{repeat,batch}_drift.
• Weights. Block-0 attention/FFN weights are pinned in a new manifest section (block_tensors). They are extracted by prepare_h3_weights.py --block from the same shard, so the existing conditioning file is unchanged.

Bindings

Node	Owning row	Bound to
adaln_projection	adaln_projection_3mod	row (#485), via the registry
norm1, norm2	h3_rmsnorm	row (#487), via the registry
residual_attn, residual_mlp	adaln_gate_residual	row (#488), via the registry
q/k/v_proj, o_proj, ffn_gate_up, ffn_down	h3_qkv_gemm, h3_attention_o_gemm, h3_ffn_gate_up_gemm, h3_ffn_down_gemm	interim: Triton FP32-accumulate GEMM, one BF16 rounding
q_norm, k_norm	h3_qk_rmsnorm_d128	interim: #487's plain H3RMSNormCudaOp on the 128-wide heads (bitwise equal to nn.RMSNorm)
rope_q, rope_k	h3_mm_rope_3axis_partial	reference: the provider replay, since no MM-RoPE kernel exists yet
attention	h3_full_attention	interim: DeterministicAttentionOp(causal=False)
swiglu	h3_swiglu	interim: SwiGLUCudaOp

When an owning row lands, swapping its node's candidate callable is the whole integration, and the report entries become that operator's block-level evidence.

Full-model and generation qualification (added 2026-10-12, in progress). This part covers the WS1 closeout rows ws1_full_h3_transformer and ws1_dual_trajectory, which are not claimed yet; it reuses the block graph above for every layer.

• tools.validation.models.h3_full_model runs the real 50 transformer blocks, both text-refiner layers and the input/conditioning/output components. Each candidate block binds its own parameters to the block-0 graph, and the row/interim/reference labels are kept. Every parameter and input gradient is checked.
• A reference pass (independent FP32 and BF16 provider runs) freezes per-tensor error bounds with a committed formula: 764 tensor bounds in examples/models/minimax_h3/full-model-thresholds.json, and a 1,635-file SHA-256 reference inventory.
• The candidate gate builds its own H3 extension, verifies every reference and checkpoint hash, then runs the block-0 golden tests (no skips), full-model forward/backward with every candidate node required to run (CUDA fallback rejected) and first drift ordered by real forward/backward execution. Only a passing numerical comparison allows the preflight and the three-seed candidate generation.
• Generation follows generation-contract.json: the pinned public reference image and prompt, seeds 42/43/44, 124 frames, 1344×768, 24 fps with joint 32 kHz audio. The comparator checks identical initial noise and shapes and reports PSNR/SSIM, temporal RMSE and audio differences.

Prior art & reuse decision

This is a validation closeout, not a new kernel; no operator is written here.

• diffusers. MiniMaxH3TransformerBlock is the executable reference. It is replayed op for op as the provider (h3_provider), which pins its dtypes and op order without importing diffusers.
• [ws1]: WS1 Full Qwen3-8B Dense Train-Inference Closeout #315 Qwen3-8B chain gate (rl_engine/validation/models/chain_gate.py). Its method is reused: per-node digests and first drift in graph order. Its graph is Qwen3's (causal, no AdaLN, Qwen shapes), so the H3 block is built on the h3_chain stage pattern of [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483.
• Interim operators are reused from RL-Kernel. The one decision taken here is the GEMM entry point. TritonDetGemmOp.__call__ reduces K as a tree with BF16 nodes; its FP32-accumulate kernel is equally deterministic and row-invariant, matches cuBLAS's accuracy, and is 9–17× faster (B200, error vs FP64):

Shape	Tree path	FP32 K loop → BF16	cuBLAS	Tree time	FP32 K loop time
264 × 5376 → 7168	7.0e-3	2.4e-3	2.4e-3	10.4 ms	1.2 ms
925 × 14336 → 5376	7.5e-3	2.6e-3	2.6e-3	47 ms	2.7 ms

Direct bitwise comparison

The H3 block is composed of the WS1 rows below it, and their reuse candidates are compared byte for byte with tools/validation/models/h3_bitwise_reuse.py (committed in #483, merged down the stack). It runs every candidate that h3_prior_art.candidates() defines on identical inputs and compares every output and leaf gradient with diffusers' path and with the rl-kernel op. Each row's report, written from a clean tree on its own branch: #483 timestep_sinusoid; #484 timestep_mlp; #485 adaln_projection; #486 adaln_row_gather; #487 norm_modulate; #488 gate_residual; #489 final_adaln_out.

Results

Block-0 replay, generated from a clean tree at 41e4293 on an otherwise idle B200 (torch 2.13.0+cu130), with tools/validation/models/h3_block_replay.py --layouts tiny,small,medium: report.json.

S	First drift (chained / isolated)	Repeat or batch drift, any node or gradient	Output err: RL-Kernel / diffusers	Fwd ms	Fwd+bwd ms
264	adaln_projection / adaln_projection	none	5.21e-3 / 5.21e-3	7.4 vs 0.61	22.7 vs 2.8
925	adaln_projection / adaln_projection	none	7.87e-3 / 7.87e-3	30.1 vs 1.12	81.7 vs 5.0
3160	adaln_projection / adaln_projection	none	8.97e-3 / 8.97e-3	225 vs 3.8	577 vs 11.5

• Isolated bitwise promises. Every node that promises it is bitwise equal to diffusers on diffusers' inputs at every size: norm1/norm2, q_norm/k_norm, RoPE, and both gated residuals. The first drift is the first reduction whose tree differs from cuBLAS.
• Forward accuracy. Per-node error vs the FP32 golden is 0.93–1.20× diffusers'. The largest is the interim attention at S = 3160.
• Gradient accuracy:
  • The candidate is up to 7× more accurate than diffusers on adaln_proj.weight, adaln_proj.bias and temb. diffusers' index_select backward accumulates in BF16 atomics; the row operators use FP32 segment sums.
  • The candidate is within 1.16× on the other leaves, except attn.norm_q.weight at S = 3160 (9.5e-3 vs 5.5e-3). At S = 925 the same leaf is 2× more accurate than diffusers, so the 128-element gradient's error follows BF16 rounding, not one side.
• Control H-C3, reported, not promised. Permuting the packed rows with their positions and indices changes 79 / 450 / 2199 output rows (max 8). Attention sums keys in packed order; settling this belongs to h3_full_attention.
• Speed. The 12–60× forward cost is the interim GEMMs and attention (FP32 scores materialised), not the row operators.

Status

• Full-model candidate acceptance is incomplete.
  • Reference run (clean source 2681cadb): FP32/BF16 provider passes over all 50 blocks, both refiners and the boundaries, with 638 parameter-gradient artifacts per precision. Reference generation and repeats passed for seeds 42/43/44 at 124 frames, 1344×768, 24 fps and 32 kHz audio, and all 49 predicted-velocity steps matched their repeated provider controls. These are provider results, not candidate acceptance.
  • Candidate run (clean source 98b07a3f, one B200): native build and every reference/checkpoint hash check passed, then the block-0 golden regression gave 8 passed, 1 failed: the attn.norm_q.weight gradient's relative error 0.03398705939217804 exceeds the unchanged bound 0.03044327425907444 (provider 0.016962182839382958). The full-model and generation stages therefore did not run. Logs, JUnit and the binary are kept.
  • Diagnostic run: reproduced the q-norm gradient failure. With identical inputs and upstream gradients, the isolated Q/K norm candidate and provider errors are equal (Q 0.0016656962, K 0.0029305728), so the cause is upstream of the norm. A diagnostic from 9317a5e1 with 34 single-node substitution controls (one provider node in the candidate graph, and the reverse) is queued. These hybrids localise the drift and cannot establish acceptance. Thresholds are unchanged.
• Block-0 replay. The evidence at 41e4293 / caaf754 (Results) has no known failures. The full-model gate's stricter block-0 bound is what fails above.
• CPU checks. Lint, unit tests and docs run on this PR and pass at 9317a5e1. The DCO check reports a failure because it cannot read more than 250 commits; all 262 commits on this branch carry Signed-off-by (checked with git log).
• Fixed: the flaky test_h3_adaln_modulation.py::TestFusedModulation::test_backward_keeps_table_gradient_fp32 in [WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486 (46347118), merged here. Its golden rounded an FP64 SiLU where the kernel rounds the FP32 SiLU, and temb was unseeded; the kernel was deterministic.

History

• 2026-10-13: fast-math guard (from the #483 review). KERNEL_ALIGN_USE_FAST_MATH=1 adds --use_fast_math to the whole _C build, which approximates expf/sinf/cosf/rsqrtf/division and flushes denormals. Before the fix a fast-math build of the top of the stack failed 33 H3 tests (the timestep_sinusoid_h3 and h3_rmsnorm bitwise tests and the chain/block tests built on them; det_linear, adaln_row_gather and gate_residual still passed). Every csrc/cuda/h3 source is now in H3_BITWISE_SOURCES and left out of fast-math builds, the bindings are gated on KERNEL_ALIGN_WITH_H3_KERNELS, and dispatch falls back to the PyTorch references. On this branch: merge 4ba3a8fb and 9cb10d03 (block dispatch-check skip). Results for both builds are under Tests.
• 2026-10-13: site identifiers removed from the full-model kit (0348e37b). The .sbatch scripts keep only site-neutral resource requests (partition, account and job name are given at submission), the reference manifest is reference-manifest.json without a job ID (its 1,635 file hashes are unchanged, so the reference evidence still verifies), and generation-inputs.json records the source suite relative to its pinned revision.
• 2026-10-13: documentation repair (9317a5e1). 14 report links outside the MkDocs tree now point to immutable GitHub URLs at 68e472b7; the strict docs build passes. No evidence or numbers changed.
• 2026-10-12: full-model and generation qualification added (ae620486 onward; see What). An earlier reference attempt completed its backward stages but failed generation because torchvision/the video processor was missing; the processor repair and a matching torchvision overlay fixed the rerun. Earlier local expanded CPU results (149 passed, 358 hardware skips, three subprocess-startup timeouts) are historical, not an all-pass claim.
• 2026-10-12: evidence directory migration (03cd824b). The one-block report.json and figure.png moved to reports/experiments/h3-one-block-b200/ with identical SHA-256; the report keeps its 41e4293 measurement revision. Docs, the plotting example and the layout audit map were updated; no GPU numbers were rerun.
• Earlier test runs, superseded by Tests below: 2026-10-11 at caaf754, tests/models/minimax_h3 on 2× B200: 501 passed, 5 skipped; with the fast-math guard (2× B200, passed / skipped / failed): default 545 / 7 / 0, fast-math 196 / 356 / 0; after the flaky-test fix at 27bb0295: default 550 passed / 7 skipped, fast-math 201 / 356, 0 failed.

Tests

At this PR's head 3bd2300a, 8× B200, pinned MiniMax-H3@42ed227 weights; each build compiled from a clean checkout of that commit:

python tools/weights/prepare_h3_weights.py --out <dir> --block && export RL_KERNEL_H3_WEIGHTS=<dir>
python setup.py build_ext --inplace                                  # default build
python -m pytest tests/models/minimax_h3 tests/build tests/runtime/test_dispatch.py -q
KERNEL_ALIGN_USE_FAST_MATH=1 python setup.py build_ext --inplace     # fast-math build, same pytest

Build	H3 kernels in _C	passed / skipped / failed
default	yes	554 / 3 / 0
KERNEL_ALIGN_USE_FAST_MATH=1	no (PyTorch reference fallback)	201 / 356 / 0

Default-build skips (environment only): 1 × could not import 'matplotlib'; 1 × could not import 'sglang.kernels.ops.diffusion.modulate.timestep_embedding_jit'; 1 × extension enforcement is CI-only (orchestrator sets RL_KERNEL_REQUIRE_EXT=1). The branch head has since moved to 0348e37b, which only changes the full-model kit's job scripts, manifest name and README (see History).

Fast-math regression tests: tests/build/test_build_platform_collectives.py::test_cuda_build_compiles_h3_kernels_without_fast_math, ::test_fast_math_build_excludes_bitwise_h3_kernels, tests/runtime/test_dispatch.py::test_h3_timestep_falls_back_when_fast_math_build_omits_kernel, ::test_h3_ops_prefer_cuda_and_fall_back_to_pytorch_reference.

tests/models/minimax_h3/test_h3_one_block.py checks the block-0 replay: on CPU, that the layouts are pinned to diffusers, the graph is well formed and MM-RoPE rotates 96 of 128 channels; on GPU forward (S = 264, 925), that every node repeats and is batch-row bitwise, promised nodes are isolated-bitwise and node and output errors are at most 1.25 × diffusers' + 1e-3; on GPU backward (S = 264), that every leaf repeats, dX is batch-row bitwise, no gradient drifts and leaf errors are at most 1.5 × diffusers' + 5e-3.

On this PR the GitHub checks are DCO, CodeRabbit and the CPU lint/unit/docs workflow; GPU results come from the run above. The full-model candidate gate is separate (Status).

Scope / Known limitations

• ROCm. CUDA only. The interim attention and Triton GEMM build for ROCm, but the block has not been run on AMD hardware.
• Interim bindings. 11 of 17 nodes are interim or reference. Their speed and rounding are not their owning rows' final ones, and RoPE carries no RL-Kernel evidence until h3_mm_rope_3axis_partial lands.
• Stand-in inputs. The block input is a seeded stand-in (the input projections are other rows), and temb is the provider's (the conditioning chain is [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483–[WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486).
• Block-0 replay inputs. The 17-node replay runs block 0 only; the 50-block, refiner and generation gates are the full-model qualification below, which is in progress.
• Full-model qualification. Boundary components (input projections, conditioning, output heads) stay provider/reference. Candidate backward qualification, three-seed candidate generation and quality acceptance are outstanding (see Status). The generation environment needs torchvision matching PyTorch, and Qwen3VLProcessor needs its video processor even for image conditioning.