ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC by ggerganov · Pull Request #27960 · ggml-org/llama.cpp

ggerganov · 2026-08-29T11:50:37Z

Overview

Some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. The RPC backend had a hardcoded list of such ops (FLASH_ATTN_EXT, MUL_MAT_ID): if a backend started expanding the alloc size of another op and the list was not updated, the RPC client would under-allocate.

add ggml_backend_op_alloc_size_may_expand() to the backend utils, returning true for the ops that expand the alloc size in any backend (FLASH_ATTN_EXT, MUL_MAT_ID, CUMSUM, ARGSORT, TOP_K)
add a GGML_ASSERT in ggml_backend_buft_get_alloc_size() catching a backend that expands the alloc size of a compute op that is not listed in the helper
use the helper in the RPC backend instead of the hardcoded list, which also fixes the previously missing CUMSUM/ARGSORT/TOP_K for e.g. a Metal server

Additional information

The op list was derived by auditing all get_alloc_size implementations in the tree. All relevant sites are marked with a [TAG_ALLOC_SIZE_EXPAND] comment so that they can be found easily when adding a new op.

The assert exempts non-compute tensors (ggml_op_is_empty), since type-dependent expansions (e.g. quantized weight padding) apply to weights regardless of op.

ref: #15966

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES. pi:llama.cpp/Qwen3.8-27B

some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. add ggml_backend_op_alloc_size_may_expand() to the backend utils, listing these ops, and assert in ggml_backend_buft_get_alloc_size that a backend expanding the alloc size of a compute op only does so for ops listed in the helper. use the helper in the RPC backend to decide whether to query the remote server for the actual alloc size, instead of a hardcoded list. Assisted-by: pi:llama.cpp/Qwen3.8-27B

ggerganov · 2026-08-29T11:58:49Z

This fixes Qwen4 garbage generation when running over RPC with Metal.

The pin was 50 commits behind the PR head and sat before "Add MTP support", so the nightly shipped GLM-5-Next without the NextN draft head, without the master merge, and without the pooled-key shift fix. Verified against b10705 by replaying the resolve step: the new commit fetches from ggml-org, is a commit of ggml-org#27754, and merges onto the base plus the pins listed before it with no conflict. Unrelated, and not fixed here: ggml-org#25731 stops merging at b10705. Upstream ggml-org#27960 touched ggml/src/ggml-rpc/ggml-rpc.cpp, which the Inkling branch also edits, and additive_merge.py correctly refuses it. It merges on b10698, the base of the last shipped nightly, so the next run on a newer base will fail there until that branch is merged forward.

cont ggml-org/llama.cpp#27960

* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont #27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml

cont ggml-org/llama.cpp#27960

* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org/llama.cpp#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml

…-org#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. add ggml_backend_op_alloc_size_may_expand() to the backend utils, listing these ops, and assert in ggml_backend_buft_get_alloc_size that a backend expanding the alloc size of a compute op only does so for ops listed in the helper. use the helper in the RPC backend to decide whether to query the remote server for the actual alloc size, instead of a hardcoded list. Assisted-by: pi:llama.cpp/Qwen3.8-27B