Skip to content

HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes - #28907

Merged
ggerganov merged 2 commits into
ggml-org:masterfrom
IMbackK:cdnammafattn256
Sep 27, 2026
Merged

HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes#28907
ggerganov merged 2 commits into
ggml-org:masterfrom
IMbackK:cdnammafattn256

Conversation

@IMbackK

@IMbackK IMbackK commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Overview

For large head sizes the mma kernel still eventually outscales the tile kernel even at occupancy 1 limited by smem

Additional information

Details
Backend GGML op Op parameters TFLOPS master TFLOPS cdnammafattn256 Speedup
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.46 0.45 0.97
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 24.42 24.63 1.01
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.58 0.56 0.97
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 25.24 24.98 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.37 0.36 0.97
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 23.75 23.42 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.63 0.62 0.98
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 25.37 25.26 1.00
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.46 0.43 0.93
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[1,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 24.61 24.03 0.98
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.31 2.29 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 32.53 32.33 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.63 2.60 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 32.91 32.70 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.48 1.39 0.94
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 29.92 31.06 1.04
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.82 2.79 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.12 32.96 1.00
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.89 1.84 0.97
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[4,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 32.13 31.86 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.96 3.84 0.97
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.49 33.33 1.00
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 4.47 4.20 0.94
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 34.02 33.81 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.44 2.41 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 32.35 31.95 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 4.82 4.78 0.99
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 34.51 34.49 1.00
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.34 3.17 0.95
ROCm0 FLASH_ATTN_EXT hsk=128,hsv=128,nh=8,nr23=[8,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.11 32.87 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=10000,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.63 1.59 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=10000,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.08 1.06 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=10000,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 29.81 29.22 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=10000,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 28.26 28.75 1.02
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=1024,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.80 0.79 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=128,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.14 0.15 1.01
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=20000,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.75 1.71 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=20000,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.15 1.08 0.95
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=20000,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.56 29.92 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=20000,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.06 29.41 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=2048,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.11 1.08 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.16 1.17 1.00
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=2,nr23=[16,1],kv=512,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.52 0.52 1.00
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=131072,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.00 2.74 0.91
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=131072,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.12 25.74 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.55 2.24 0.88
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=16384,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 25.58 24.96 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.59 1.56 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=4096,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 23.78 23.22 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.93 2.66 0.91
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=4,nr23=[6,1],kv=65536,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.08 25.49 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.48 0.46 0.96
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 27.63 27.43 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.55 0.53 0.96
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 28.22 28.05 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.38 0.34 0.89
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.16 25.88 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.62 0.61 0.97
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.24 26.86 1.02
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.50 0.48 0.96
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[1,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 27.12 26.83 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.46 2.44 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 31.36 31.17 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.68 2.63 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 31.67 31.52 1.00
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.80 1.55 0.86
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.02 29.71 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.87 2.83 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 32.34 32.10 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.05 2.09 1.02
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[4,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.89 30.59 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 4.80 4.74 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 36.06 35.67 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 5.08 5.00 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 36.64 36.28 0.99
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.64 3.34 0.92
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.19 33.58 1.01
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 5.35 5.26 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 37.00 36.84 1.00
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 4.21 4.13 0.98
ROCm0 FLASH_ATTN_EXT hsk=256,hsv=256,nh=8,nr23=[8,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 35.25 34.79 0.99
ROCm0 FLASH_ATTN_EXT hsk=512,hsv=512,nh=1,nr23=[8,1],kv=49152,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.21 3.17 0.99
ROCm0 FLASH_ATTN_EXT hsk=512,hsv=512,nh=1,nr23=[8,1],kv=49152,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=2048 3.21 3.18 0.99
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=1,nr23=[16,1],kv=49152,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 6.17 6.04 0.98
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=1,nr23=[16,1],kv=49152,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=2048 6.17 6.08 0.98
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 2.46 2.45 1.00
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.01 20.90 1.49
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 2.71 2.67 0.99
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 13.46 21.58 1.60
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 1.94 1.72 0.89
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.08 19.21 1.36
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 2.72 2.81 1.03
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.16 21.85 1.54
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 2.14 2.11 0.98
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[4,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 13.98 20.63 1.48
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 4.58 4.58 1.00
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.10 21.19 1.50
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 4.82 4.73 0.98
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 11.59 21.54 1.86
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 3.93 3.67 0.93
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.12 19.70 1.39
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 5.06 4.97 0.98
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.14 21.85 1.55
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 4.36 4.31 0.99
ROCm0 FLASH_ATTN_EXT hsk=576,hsv=512,nh=8,nr23=[8,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=1,n_kv_max=0 14.10 20.49 1.45
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.36 0.33 0.92
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 13.25 13.01 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.45 0.41 0.92
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 13.66 13.01 0.95
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.20 0.20 1.00
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 14.10 12.59 0.89
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.54 0.52 0.97
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 14.26 13.58 0.95
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.21 0.22 1.03
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[1,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 12.76 12.79 1.00
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.52 1.37 0.90
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 31.65 31.36 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.94 1.90 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 31.89 31.58 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.88 0.88 1.00
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.76 30.32 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.36 2.32 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 30.16 31.83 1.06
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.03 1.01 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[4,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 31.36 31.05 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=16384,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 2.62 2.50 0.95
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=16384,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.83 33.52 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=32768,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 3.47 3.15 0.91
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=32768,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 34.12 33.78 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=4096,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.01 1.01 1.00
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=4096,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.14 32.56 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=65536,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 4.08 3.97 0.97
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=65536,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 34.34 33.91 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.82 1.81 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.16 1.14 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 0.84 0.83 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=4,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 8.91 8.83 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q4_0,type_V=q4_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.17 25.72 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=7680,nb=512,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=q8_0,type_V=q8_0,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 26.07 25.50 0.98
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=8192,nb=1,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 1.70 1.68 0.99
ROCm0 FLASH_ATTN_EXT hsk=64,hsv=64,nh=8,nr23=[8,1],kv=8192,nb=4096,mask=1,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 33.61 33.17 0.99
ROCm0 FLASH_ATTN_EXT hsk=72,hsv=72,nh=16,nr23=[1,1],kv=5776,nb=5776,mask=0,sinks=0,max_bias=0.000000,logit_softcap=0.000000,prec=f32,type_K=f16,type_V=f16,permute=[0,1,2,3],kv_view=1,v_is_view_of_k=0,n_kv_max=0 8.74 8.14 0.93

<\details>

Requirements

@IMbackK
IMbackK requested a review from a team as a code owner September 14, 2026 17:57
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Sep 14, 2026
JohannesGaessler
JohannesGaessler approved these changes Sep 15, 2026
View reviewed changes

@JohannesGaessler JohannesGaessler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GPU Model Microbatch size Test t/s b10952 t/s a7899f0 Speedup
MI100 deepseek2 16B Q4_0 1 pp1024@d32768 88.53 89.16 1.01
MI100 deepseek2 16B Q4_0 2 pp1024@d32768 95.42 95.35 1.00
MI100 deepseek2 16B Q4_0 4 pp1024@d32768 126.17 126.23 1.00
MI100 deepseek2 16B Q4_0 8 pp1024@d32768 195.22 194.84 1.00
MI100 deepseek2 16B Q4_0 16 pp1024@d32768 257.18 312.10 1.21
MI100 deepseek2 16B Q4_0 32 pp1024@d32768 314.76 407.08 1.29
MI100 deepseek2 16B Q4_0 64 pp1024@d32768 332.68 453.24 1.36
MI100 deepseek2 16B Q4_0 128 pp1024@d32768 378.64 551.63 1.46
MI100 deepseek2 16B Q4_0 256 pp1024@d32768 412.25 580.00 1.41
MI100 deepseek2 16B Q4_0 512 pp1024@d32768 435.17 731.22 1.68

Comment thread ggml/src/ggml-cuda/fattn-mma-f16.cuh Outdated
@IMbackK

IMbackK commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

force push was for: braces where unbalanced after applying suggestion above.

@IMbackK IMbackK added merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. and removed merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. labels Sep 21, 2026
@JohannesGaessler

JohannesGaessler commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Is it intentional that you removed the merge ready label again?

@IMbackK

IMbackK commented Sep 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

it was, since the quality check was failing on vgpr spills, but since i added them to the ignore list this can be merged now.

@IMbackK IMbackK added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Sep 26, 2026
@ggerganov
ggerganov merged commit 2ebd9ae into ggml-org:master Sep 27, 2026
15 of 16 checks passed
@j7z j7z mentioned this pull request Sep 28, 2026
Closed
pierreguillot pushed a commit to Ircam-Partiels/llama.cpp that referenced this pull request Oct 1, 2026
@IMbackK @pierreguillot
HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch si…
ff02cc4 
…zes (ggml-org#28907)

* HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes

* CI: hip-quality-check: ignore spills for very large mfma mma kernels
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

Assignees

No one assigned

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@IMbackK @JohannesGaessler @ggerganov