Skip to content

[webgpu] Use workgroup memory to reduce register pressure - #24286

Merged
Sushanth Rajasankar (sushraja-msft) merged 9 commits into
mainfrom
opt_flash_attention
Apr 11, 2025
Merged

[webgpu] Use workgroup memory to reduce register pressure#24286
Sushanth Rajasankar (sushraja-msft) merged 9 commits into
mainfrom
opt_flash_attention

Conversation

@qjia7

Copy link
Copy Markdown
Contributor

On Qualcomm Adreno X1 GPUs, the previous implementation of the FlashAttentionProgram shader in the WebGPU backend was causing high register pressure, leading to performance degradation. This PR uses workgroup memory to reduce the register pressure and improve performance.

TTFT for phi4 with 1K inputs becomes 10s from 40s on Qualcomm Adreno X1 GPU.

@qjia7
Jiajia Qin (qjia7) marked this pull request as ready for review April 3, 2025 06:30
@guschmue Guenther Schmuelling (guschmue) added the ep:WebGPU ort-web webgpu provider label Apr 3, 2025
@sushraja-msft

Copy link
Copy Markdown
Contributor

LGTM thanks

Comment thread onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc Outdated
Comment thread onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc Outdated
Comment thread onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc
Comment thread onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc Outdated
@qjia7

Copy link
Copy Markdown
Contributor Author

Guenther Schmuelling (@guschmue) Need your help on full perf test to ensure it won't bring regressions on other GPUs. Thanks.

@guschmue

Copy link
Copy Markdown
Contributor

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline,web_Debug / build_onnxruntime_web,web_Release / build_onnxruntime_web

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 5 pipeline(s).

@guschmue

Copy link
Copy Markdown
Contributor

azp /run web_Debug / build_onnxruntime_web,web_Release / build_onnxruntime_web,webgpu_build_x64_RelWithDebInfo, webgpu_external_dawn_build_x64_RelWithDebInfo,webgpu_minimal_build_edge_build_x64_RelWithDebInfo,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64, QNN CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline,
Windows OpenVINO CI Pipeline

@guschmue

Copy link
Copy Markdown
Contributor

/azp run Windows OpenVINO CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@guschmue

Copy link
Copy Markdown
Contributor

/azp run web_Debug / build_onnxruntime_web,web_Release / build_onnxruntime_web,webgpu_build_x64_RelWithDebInfo, webgpu_external_dawn_build_x64_RelWithDebInfo,webgpu_minimal_build_edge_build_x64_RelWithDebInfo,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64, QNN CI Pipeline,Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows GPU TensorRT CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@guschmue

Copy link
Copy Markdown
Contributor

/azp run build_x64_release

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@guschmue

Copy link
Copy Markdown
Contributor

sorry, needs to be merged with main

@guschmue

Copy link
Copy Markdown
Contributor

emscripten build complains about:

onnxruntime/onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc:934:72: error: '&&' within '||' [-Werror,-Wlogical-op-parentheses]
  934 |          (context.AdapterInfo().vendor == std::string_view{"qualcomm"} && parameters.head_size_ % 8 == 0 || parameters.head_size_ % 4 == 0);
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
/mnt/vss/_work/onnxruntime/onnxruntime/onnxruntime/contrib_ops/webgpu/bert/flash_attention.cc:934:72: note: place parentheses around the '&&' expression to silence this warning
  934 |          (context.AdapterInfo().vendor == std::string_view{"qualcomm"} && parameters.head_size_ % 8 == 0 || parameters.head_size_ % 4 == 0);
      |                                                                        ^
      |           (                 

@sushraja-msft
Sushanth Rajasankar (sushraja-msft) deleted the opt_flash_attention branch April 11, 2025 16:54
Ashrit Shetty (ashrit-ms) pushed a commit that referenced this pull request Apr 24, 2025
On Qualcomm Adreno X1 GPUs, the previous implementation of the
FlashAttentionProgram shader in the WebGPU backend was causing high
register pressure, leading to performance degradation. This PR uses
workgroup memory to reduce the register pressure and improve
performance.

TTFT for phi4 with 1K inputs becomes 10s from 40s on Qualcomm Adreno X1
GPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ep:WebGPU ort-web webgpu provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants