[ServeurpersoCom]

Overview

Fix CI:
Models Backend webgpu-nvidia segfaults on master since #29622, on the very first model of test-llama-archs.

The new mixed batch path writes the token rows with set_rows into a ggml_dup of the embeddings. WebGPU had no GGML_OP_DUP, so the dup stayed on the CPU, while the set_rows writing in place into it was scheduled on WebGPU. WebGPU then got a destination living in a CPU buffer, read its context as a WebGPU buffer, and Dawn crashed. CUDA and Vulkan support DUP, so they never hit it.

DUP is the same copy as CPY and CONT, so it now goes through the same path and the whole mixed chain stays on WebGPU.

Additional information

Tested on an RTX PRO 6000 through WebGPU with the CI Dawn build:

• test-llama-archs -s 1: segfault -> 160 rows, 0 FAIL
• test-backend-ops DUP 7/7, CPY 144/144, CONT 72/72

Follow-up #29622

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[ggerganov]

@ServeurpersoCom Thanks. You can try to tackle the failing server test next:

https://github.com/ggml-org/llama.cpp/actions/runs/37903861970/job/113732486606#step:6:4633

I've root-caused it to be due to graph reallocations when we use backend sampling. Basically, we reserve a graph for the worst case of all sequences having a sampling enabled. But then the test runs a decoding batch using only one sequence, which changes the number of nodes.

There are 2 possible ways to fix it:

• When backend sampling is enabled, the graph always attaches all sampling chains for all sequences. And via ggml_build_forward_select we "activate" only those that are actually used in the batch
• Update the reserve logic in llama_context to not reserve for all possible sequences, but only for those that have a sampler

Both options are a bit tricky and I am not sure if it would increase the complexity a lot. So worst case is to adapt the failing tests to not trigger this and add TODOs to rework this in the future.

[yomaytk]

LGTM, thanks!

[ServeurpersoCom]

I've root-caused it to be due to graph reallocations when we use backend sampling. Basically, we reserve a graph for the worst case of all sequences having a sampling enabled. But then the test runs a decoding batch using only one sequence, which changes the number of nodes.

There are 2 possible ways to fix it:

* When backend sampling is enabled, the graph always attaches all sampling chains for all sequences. And via `ggml_build_forward_select` we "activate" only those that are actually used in the batch

* Update the reserve logic in `llama_context` to not reserve for all possible sequences, but only for those that have a sampler

Both options are a bit tricky and I am not sure if it would increase the complexity a lot. So worst case is to adapt the failing tests to not trigger this and add TODOs to rework this in the future.

Thanks for the root cause! Option 1 turned out to be small: build_sampling already did it for inactive samplers, so every sampler now always builds n_outputs_max_per_seq chains and decodes keep the topology of the reserve.
The draft test passes and the server suite is green with and without backend sampling on a NO_REALLOC build, opening a PR shortly.

[ServeurpersoCom]

The two WebGPU self-hosted reds are on master too, not related to DUP (DUP, CPY and CONT all pass in that log) I'll see if I can follow up