Skip to content

ggml webgpu: minor set rows optimization - #16810

Merged
CISC merged 9 commits into
ggml-org:masterfrom
reeselevine:master
Nov 5, 2025
Merged

ggml webgpu: minor set rows optimization#16810
CISC merged 9 commits into
ggml-org:masterfrom
reeselevine:master

Conversation

@reeselevine

@reeselevine reeselevine commented Oct 27, 2025

Copy link
Copy Markdown
Contributor
  • Better parallelization of SET_ROWS by having multiple threads work on each row, as well as vectorization
  • Adds more useful labels to buffers for debugging
  • Adds Dawn-specific toggles which disable some safety protections when running natively, for better performance

Better matrix multiplication coming soon!

reeselevine and others added 5 commits October 15, 2025 19:04
@reeselevine
Add buffer label and enable dawn-specific toggles to turn off some ch…
b566811 
…ecks
@neha-ha Neha Abbas
Neha Abbas @reeselevine
Minor set_rows optimization (#4)
f2e187c 
* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>
@reeselevine
Merge remote-tracking branch 'upstream/master'
2aa05c6
@reeselevine
Comment on dawn toggles
51aae63
@reeselevine
Remove some comments
f0cfae4
@github-actions github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Oct 28, 2025
@reeselevine reeselevine mentioned this pull request Oct 31, 2025
Merged
@reeselevine
reeselevine requested a review from slaren as a code owner November 1, 2025 20:55
@github-actions github-actions Bot added the testing Everything test related label Nov 1, 2025
slaren
slaren reviewed Nov 1, 2025
View reviewed changes
Comment thread tests/test-backend-ops.cpp Outdated
std::vector<ggml_tensor *> expert_views(n_expert_used);
for (int64_t i = 0; i < n_expert_used; ++i) {
expert_views[i] = ggml_view_2d(ctx, weighted, n_embd, n_tokens, weighted->nb[2], i * weighted->nb[1]);
expert_views[i] = ggml_view_2d(ctx, weighted, n_embd, n_tokens, weighted->nb[1], i * weighted->nb[1]);

@slaren slaren Nov 1, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this change is correct.

@reeselevine reeselevine Nov 2, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This update was testing some changes to the addition kernels in response to the discussion in #16857. But, it looks like the CUDA CI is failing with this change too, so if it's confirmed that the nb[2] is correct here I'll need to do a little more debugging to understand why the WebGPU add op is failing as currently written. I'll mark this PR as a draft for now to avoid it accidentally being merged.

@reeselevine reeselevine Nov 2, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick update: Realized this due to the non-contiguity in the view tensors, which isn't supported yet in the kernels. I disabled support for non-contiguous tensors here and added a note so it can be added in the future.

@reeselevine
reeselevine marked this pull request as draft November 2, 2025 04:24
@reeselevine
reeselevine marked this pull request as ready for review November 2, 2025 04:40
@reeselevine

reeselevine commented Nov 4, 2025

Copy link
Copy Markdown
Contributor Author

Just a quick ping here @slaren and/or @CISC, hopefully getting this merged will fix the WebGPU CI errors for now and allow some more PRs to be opened.

@CISC
CISC merged commit 03ea041 into ggml-org:master Nov 5, 2025
71 of 72 checks passed
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas shaobo.xie
ggml webgpu: minor set rows optimization (#16810)
84dc560 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b36f51ab3f53fa13db15c1685dc8678a32a.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas
ggml webgpu: minor set rows optimization (ggml-org#16810)
7f79539 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (ggml-org#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b3.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas
ggml webgpu: minor set rows optimization (ggml-org#16810)
7ba8f4c 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (ruixiang63#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b3.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas
ggml webgpu: minor set rows optimization (ggml-org#16810)
cfcb0ce 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (AtomicBot-ai#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b3.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas
ggml webgpu: minor set rows optimization (#16810)
95a1f5b 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b36f51ab3f53fa13db15c1685dc8678a32a.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas @zommiommy
ggml webgpu: minor set rows optimization (ggml-org#16810)
320d453 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (ggml-org#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b3.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
frostyautumnleaf pushed a commit to frostyautumnleaf/llama.cpp that referenced this pull request Oct 5, 2026
@reeselevine @neha-ha
Neha Abbas Neha Abbas
ggml webgpu: minor set rows optimization (ggml-org#16810)
02f9583 
* Add buffer label and enable dawn-specific toggles to turn off some checks

* Minor set_rows optimization (ggml-org#4)

* updated optimization, fixed errors

* non vectorized version now dispatches one thread per element

* Simplify

* Change logic for set_rows pipelines

---------

Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>

* Comment on dawn toggles

* Remove some comments

* Implement overlap binary operators

* Revert "Implement overlap binary operators"

This reverts commit ed710b3.

* Disable support for non-contiguous binary_op tensors and leave note for future support

---------

Co-authored-by: neha-ha <137219201+neha-ha@users.noreply.github.com>
Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan>
Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren slaren left review comments
@CISC CISC CISC approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@reeselevine @CISC @slaren @neha-ha