llama: improve TENSOR_READ_LAZY handling - #27837
Merged
Merged
Conversation
ngxson
changed the title
Contributor
Collaborator
Author
|
hmm ok I thought |
Contributor
|
As |
Contributor
|
Maybe add LLAMA_LAZY_MODE_ALL or something to disable prefetching completely. |
ggerganov
approved these changes
Aug 29, 2026
CISC
approved these changes
Aug 29, 2026
sdougbrown
added a commit
to sdougbrown/r9700-llama-cpp
that referenced
this pull request
Aug 30, 2026
|
Previewing this PR, definitely seeing a --load-mode none prefill speed boost of 15% over --load-mode mmap. Not quite the 2x speed of loading the whole table to RAM, but a boost nonetheless, and the PLE tables stay on disk as intended, so that's a win in my book. Disabling --tensor-read-lazy (which will be renamed --lazy-mode -lzm per #27969) along with --load-mode none shows the expected behavior of loading PLE table to RAM, the old no-mmap behavior incl. the full 2x speed gain and RAM use. |
ServeurpersoCom
approved these changes
Aug 30, 2026
ServeurpersoCom
left a comment
ServeurpersoCom
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The precedence rule is the right call. Tested on Qwen3.8-Flash-Next: with and without -ot per_layer_token_embd=CPU gives the same numbers, so it lands in auto and the -ot is correctly ignored.
Collaborator
Author
|
@Kononnable there should already been a |
|
One more update on performance gains. The mmap bottleneck isn't architectural overhead from memory mapping, it's probably just latency sensitive. I run llama.cpp in a Docker VM and moving the model files from the vdisk to direct storage on PCIe passthrough restores the no-mmap PLE lazy load prefill performance to speeds with the full model+PLE in resident memory. |
4 tasks
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Aug 31, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Aug 31, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Aug 31, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
TrevorS
added a commit
to TrevorS/llama.cpp
that referenced
this pull request
Aug 31, 2026
This reverts upstream 2578138. It costs 73% of prefill on this box. The commit routes every lazily-read tensor to the generic CPU buffer type (lazy_read::buft() returns ggml_backend_dev_buffer_type of the CPU device). Before it, our 28.8 GB per_layer_token_embd lived in CPU_Mapped -- mmap'd host memory that GB10's GPU reads directly, since the part reports pageableMemoryAccessUsesHostPageTables=1. Forced into a plain CPU buffer, the per-layer PLE gathers get scheduled on the CPU backend instead, and the GPU sits idle waiting on them: an nsys capture of the regressed build shows 31.5% GPU-busy and 22-38 W board power during prefill, with the SM clock pinned at 2190 MHz and no throttle reason active. The work is not slower, it is on the wrong device. Found by bisecting the 21 commits between c589f0e and a7cc83b on pure upstream, with none of our code in the tree: 0b5be7e hip: tune rdna 3 mmq config 741.98 t/s f1793c1 CUDA: fast mm_ids_helper for any n_eu 812.70 t/s +9.5% 2578138 llama: improve TENSOR_READ_LAZY 222.70 t/s -72.6% a7cc83b (tip) 211.67 t/s a7cc83b + this revert 813.86 t/s llama-bench pp4096, ub4096, UD-IQ4_XS. The revert restores the number exactly, which is the evidence that this commit and nothing else in that range is responsible. Worth being precise about what this is not: the mm_ids_helper commit immediately before it is a real +9.5% win here (n_expert_used = 10 was excluded from the fast path by the old warp_size % n_expert_used == 0 gate), and it survives the revert. The two are independent. This is a local deviation, not a claim the upstream change is wrong in general -- it plausibly fixes lazy reads on discrete-GPU systems where a device-buffer lazy tensor cannot work. It is wrong specifically for a UMA part that can read mmap'd host pages from the GPU, which is the only configuration we serve.
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 2, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 19, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 19, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 19, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 20, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 21, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 21, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 21, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 22, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 23, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 23, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 25, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 25, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 25, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 25, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 26, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 27, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 27, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 27, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 27, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
kurochan001
added a commit
to kurochan001/llama.cpp
that referenced
this pull request
Sep 27, 2026
… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Follow-up #27794 #27742
--tensor-read-lazytake full precedence over--load-modeor-ot: If tensor is decided to be "lazy-read", it will be mmap'ed no matter what. This is because some use cases use--load-mode nonefor faster weight offloading to GPU, but that doesn't mean user want to offload the PLE tensorllama_model_loader::lazy_readRequirements