Skip to content

cuda : fuse repeated residual MUL and ADD - #139

Draft
GenerelSchwerz wants to merge 13 commits into
reference/upstream-kernels-20261001from
kernel/repeat-mul-add-20261006
Draft

GenerelSchwerz wants to merge 13 commits into
reference/upstream-kernels-20261001from
kernel/repeat-mul-add-20261006

Conversation

@GenerelSchwerz

Copy link
Copy Markdown
Owner

Explicit REPEAT -> MUL -> ADD currently launches REPEAT followed by the residual arithmetic kernel from PR138. This folds the three stages into one generic launch while still storing the repeated tensor, rounded product and final sum. Multiplication and addition remain separate round-to-nearest operations.

REPEAT accepts its existing advertised F32/F16 types. Either MUL operand can be repeated; arithmetic keeps the existing F32/F16/BF16 combinations. Dynamic four-axis repeats, source strides and either ADD order are handled without model names or fixed dimensions. Canonical prior-root VIEW/RESHAPE/PERMUTE/TRANSPOSE nodes may intervene. Strict physical checks include NONE inputs; allocation dependencies keep original inputs and intermediates alive through the final ADD. Longer HC/affine fusions keep priority. BF16 REPEAT, arbitrary intervening compute, unsupported layouts/types and concurrent paths keep their existing execution.

This is a 4-file leaf on PR138 commit ce6ddc95589080cf0a0cf0c0fa42652fa962f940: 118 CUDA additions/8 deletions and 32 additions/5 deletions in the existing backend test. The reference-base diff includes prerequisite kernel-series work; review this leaf against that commit.

Component evidence

Own-PR138 control versus candidate on RTX 5070 Ti (SM120a), driver 615.71.09. Six alternating pairs per timed case, 2,000 graph replays per arm. Inputs and every retained output are checked before/after timing. Shared build/GPU locks cover each whole suite; timing is serial and reviewer CPU work is stopped.

Case Control us Candidate us Less time Faster pairs
F32 repeated second operand 4.09561 2.04997 49.95% 6/6
F16 repeated second operand, reversed ADD 4.09606 2.05068 49.94% 6/6
BF16 arithmetic with repeated F32 coefficient 4.09610 2.05070 49.94% 6/6
Strided four-axis F32 / mixed F16 repeat 4.09567 / 4.09619 2.05047 / 2.05083 49.94% / 49.93% 6/6 each
Tall F32, T=4096 6.14448 4.10461 33.20% 6/6
Singleton columns, H=128 4.09518 2.05088 49.92% 6/6
Larger F16 / wide F32 / H=64 6.15100 / 6.14998 / 4.10025 6.14380 / 6.14415 / 4.09682 0.12% / 0.09% / 0.08% (flat) 6/6 each

Validation and limits

  • 18 matched cases / 36 profiles / 744 CPU stage comparisons over four refreshed datasets. Full GPU/reference/upload bytes match control; maximum stage NMSE is 3.39e-15 under the unchanged fixture limit. Complete remaining kernel and launch/resource maps match. All 18 GPU arenas match control, including normal dead/all-live scheduler allocation.
  • Seven completed cases (88 commands) are explicitly revalidated from the first boundary packet; eleven cases (104 commands) are fresh after fixture corrections. The earlier failures occurred in the control: CPU REPEAT requires contiguous columns, and retained scheduler inputs need OUTPUT pins. Failed artifacts remain preserved; numerical/stability checks were not relaxed.
  • 123/123 existing backend tests per arm at their original limits. The 28 added whole-graph cases compare REPEAT, PRODUCT and SUM independently at 1e-7. Profiles confirm all 28 use the new kernel, with the full remaining kernel/launch maps matching control.
  • Ten clean serial memcheck/racecheck/synccheck jobs, 280 additional CPU stage comparisons and exact previously qualified reference/upload/output bytes. Includes strided repeated second operand, all-live scheduler, mixed BF16 arithmetic, 40 views, finite edge values and strict NONE alias rejection.
  • All 1,573 prior functions/resources and 1,178,982 encoded instruction rows in six audited translation units remain unchanged. Host/unary are freshly audited; four unchanged units are inherited through exact objects and prior default-proof bindings. One added kernel uses 28 registers, zero stack/local/shared memory; CUDA library grows 13,160 bytes. Eight host/unary compile-only checks pass for SM75/80/89/90; older hardware was not benchmarked.

These are isolated component results. Model C++ expressions do not prove graph adjacency: Qwen4exp's optional unfused post branch, Gemma3n and delta-net can have coefficient arithmetic between REPEAT and MUL. No default Flash Next, model-serving, MTP or universal speed claim. Full stream-reduction trees, concatenation and private dispatcher/allocation integration remain separate.

AI disclosure: Codex assisted with implementation, testing, review and this description under repository-owner authorization for this maintained fork. No upstream submission or merge.

Carry the reviewed kernel sources from PR125, PR129 and PR132 as the
matching control for the separate tensor-affine HC follow-up.

Assisted-by: Codex
Preserve gate intermediates, original rounding and live HC outputs.
Retain strict input alias checks and standard allocation dependencies.

Assisted-by: Codex
Use the exact tested control sources from published PR130, PR129 and PR132.

Assisted-by: Codex
Reuse the allocated affine match to find the planned HC_POST and retain prepared emission priority. Reject expanded normalization cuts whose terminal is not written by this kernel.

Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant