Repository navigation
meta : handle host views - #30217
Merged
Merged
meta : handle host views#30217
Conversation
github-actions
Bot
added
the
ggml
ggerganov
marked this pull request as ready for review
October 9, 2026 10:25
Member
Author
|
@JohannesGaessler PTAL when you have some time and let me know if this changes looks OK to you. |
Contributor
|
Related PR by @yomaytk : #28075 . I'll take a look at this PR as well but the fundamental issue as I see it is that the backend scheduler is assigning backend IDs in contradiction of what the ggml backend API says should be supported in terms of buffer types. So my understanding is that this is a bug in the ggml backend scheduler that is simply surfacing through the meta backend. |
JohannesGaessler
left a comment
JohannesGaessler
left a comment
Contributor
There was a problem hiding this comment.
Okay, I think I misunderstood what this PR attempts to do. Still, in my view the fundamental issue is that there are seemingly unrelated tensors in the graph's nodes that don't actually do anything.
Contributor
|
My opinion is that we can merge something like this as an ad-hoc fix but please add a TODO that going forward we should look into a better solution. |
A view shares the memory of its view_src, so ggml-alloc never allocates a view in the buffer of the split it lands in - the scheduler copies the source into the split and the ops that use the view read that copy. The view node itself is a noop and does not need a split of its own, but the meta backend asserted when one was left inside a meta split: - ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context - the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor() Accept such nodes when they are views of host tensors, which also generalizes the previous s_copy_main workaround. This fixes the assert hit by KV cache views when using --split-mode tensor with partial offload. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
ggerganov
force-pushed
the
gg/meta-handle-host-views
branch
from
October 9, 2026 10:49
27a975a to
ba2454e
Compare
Member
Author
|
I added a TODO in the latest commit. |
JohannesGaessler
approved these changes
Oct 9, 2026
ckti
pushed a commit
to ckti/llama.cpp-consolidated
that referenced
this pull request
Oct 12, 2026
* meta : handle views of tensors allocated on the host A view shares the memory of its view_src, so ggml-alloc never allocates a view in the buffer of the split it lands in - the scheduler copies the source into the split and the ops that use the view read that copy. The view node itself is a noop and does not need a split of its own, but the meta backend asserted when one was left inside a meta split: - ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context - the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor() Accept such nodes when they are views of host tensors, which also generalizes the previous s_copy_main workaround. This fixes the assert hit by KV cache views when using --split-mode tensor with partial offload. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * archs : re-enable sm tensor for K2 Horizon * cont : add TODO and reference
6 of 7 tasks
thelittlefireman
pushed a commit
to thelittlefireman/llama.cpp
that referenced
this pull request
Oct 12, 2026
* meta : handle views of tensors allocated on the host A view shares the memory of its view_src, so ggml-alloc never allocates a view in the buffer of the split it lands in - the scheduler copies the source into the split and the ops that use the view read that copy. The view node itself is a noop and does not need a split of its own, but the meta backend asserted when one was left inside a meta split: - ggml_backend_meta_get_split_state() dereferenced tensor->buffer->context - the graph rebuild mapped every node with ggml_backend_meta_buffer_simple_tensor() Accept such nodes when they are views of host tensors, which also generalizes the previous s_copy_main workaround. This fixes the assert hit by KV cache views when using --split-mode tensor with partial offload. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * archs : re-enable sm tensor for K2 Horizon * cont : add TODO and reference
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
ref: #29535 (comment)
My analysis is that views of host-buffer tensors can end up assigned into the meta backend splits. This usually happens when we perform some reshape of the extracted token embeddings before using them:
I tried to modify the scheduler to prevent that (i.e. always keep the reshapes/views in the host split), but it gets a bit tricky since we don't have a guarantee where the reshapes/views nodes will end up in the graph - they are not always arranged next to the host ops (GET_ROWS in this case) depending on what other ops are performed before the
rescomputation.I don't see a reason why the meta backend cannot just ignore these nodes. After all, they don't cause any problem with other backends, so it seems something that should be fixed in the Meta backend. There is already a related FIXME comment. This PR extends the logic to handle this.
Additional information
The K2 Horizon model can now be used with
-sm tensor. This is also helps with a follow-up refactoring in #30160 .Requirements