Skip to content

cuda: share banked activation images - #148

Draft
GenerelSchwerz wants to merge 22 commits into
reference/upstream-kernels-20261001from
kernel/banked-activation-reuse-20261007
Draft

GenerelSchwerz wants to merge 22 commits into
reference/upstream-kernels-20261001from
kernel/banked-activation-reuse-20261007

Conversation

@GenerelSchwerz

Copy link
Copy Markdown
Owner

Share the original activation image when multiple ordinary MUL_MAT nodes read the same contiguous F32 tensor with bank or sample dimensions. Use the existing reuse planner, image-family keys, overwrite guards and stream handling. Keep the original quantization, matrix kernels and NVFP4 row-scale layout.

This follow-up is tested against #147 at 7ef1d05483aad1e3bf974c2cfe6b1f0ba7f41496. The branch includes earlier series dependencies; the leaf changes one host file (24 additions, 11 deletions) and adds eight calls to the existing matrix test (6 lines).

Existing prepared norm writers keep their singleton image contract. Banked norm output can share the ordinary converter image. Strided banked inputs, input writes, incompatible image families and disabled fusion retain their original behavior. This leaf does not extend routed MUL_MAT_ID, remove a multi-consumer REPEAT, or integrate a private executor.

On RTX5070Ti/SM120a, six alternating pairs of 2,000 captured replays per build gave these median complete-component wall times in microseconds. Each of the four material reductions was faster in all six pairs; the two flat cases are not claimed as gains.

Component #147 control us Candidate us Result
Q4_K MMVQ 6.143596 4.328582 29.54% less time
Q4_0 MMQ 12.284708 10.881724 11.42% less time
Native NVFP4 MMQ 12.284506 10.238008 16.66% less time
Native MXFP4 MMQ 12.285233 10.247407 16.59% less time
RMS/SCALE + gap + Q4_0 MMQ 14.345047 14.331282 Flat
Already-shared flat Q4_0 14.330447 14.331118 Flat

Validation used the original CPU limits (5e-4; native FP4 2e-2), four updated datasets and byte-identical candidate/control GPU outputs, CPU references and uploaded inputs. The six utility pairs had 104 stage comparisons; 20 boundary pairs had 488. Coverage includes eight readers (144 converter calls reduced to 18), separate bank/sample axes, MMQ layouts, tails, native/forced activation precision, normal GPU/CPU scheduler allocation and rebuilds, live inputs, weight broadcasting and negative stride/write/family/disable cases. The original prepared flat path preserved 16 fused emissions and two separate conversions: two updated datasets intentionally use nonzero SCALE bias, which the existing emitter rejects.

Each build passed 268 existing regression cases and eight matrix-pair whole-graph tests, with the same six original unsupported TQ skips. Ten serial sanitizer runs were clean and checked 136 CPU stages. Fresh host compiles passed for SM75/80/89/90 (compile-only). All 26 measured component/boundary CUDA arenas had zero delta. The fresh host object's existing shader remained identical in resources and encoded instructions; 143 CUDA objects were reused literally, no device kernel was added and CUDA library size growth was zero.

All builds and GPU checks held the ordered shared build/GPU locks for each complete batch. An initial qualification stopped on the incorrect flat-emitter count; its 77 successful commands were independently revalidated, followed by 28 fresh continuation commands. No source, arithmetic, ownership check or tolerance changed during that correction.

These are isolated component results, not model-serving, MTP, peak-VRAM, additive series gains or older-GPU runtime results.

AI disclosure: Codex implemented the scoped change and prepared the validation and description under the fork owner's explicit authorization; a bounded GPT-6.1 Sol review independently checked source and raw evidence.

Carry the reviewed kernel sources from PR125, PR129 and PR132 as the
matching control for the separate tensor-affine HC follow-up.

Assisted-by: Codex
Preserve gate intermediates, original rounding and live HC outputs.
Retain strict input alias checks and standard allocation dependencies.

Assisted-by: Codex
Use the exact tested control sources from published PR130, PR129 and PR132.

Assisted-by: Codex
Reuse the allocated affine match to find the planned HC_POST and retain prepared emission priority. Reject expanded normalization cuts whose terminal is not written by this kernel.

Assisted-by: Codex
Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant