[ggerganov]

Overview

Add an optional ggml_backend_sched_realloc_callback that user code registers with ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a graph are not sufficient and have to be reserved again. The callback reports whether the backend assignments and the graph size changed since the previous scheduling, letting the user decide what to do with the event (trace, count, metrics, abort) instead of ggml maintaining a counter internally.

llama_context registers the callback in the new sched_new helper, which wraps the scheduler creation that was duplicated in sched_reserve. It treats a reallocation that follows a change of the backend assignments as expected and ignores it, and counts the rest by graph size - unchanged or changed from the previously scheduled graph. Both counters are reported on one line in the perf stats:

llama_perf_context_print:    graphs reused =         30
llama_perf_context_print:   graph reallocs =          0    /     0        (same size) / (changed size)

A non-zero same size count means the compute buffers were re-reserved for a graph that was identical in size to a previously scheduled one - i.e. the initial reservation was not sufficient. The same size check reuses the same condition as the existing GGML_SCHED_DEBUG_REALLOC debug check in ggml_backend_sched_alloc_splits, so the previously debug-only graph_size/prev_graph_size tracking (renamed from debug_graph_size/debug_prev_graph_size) is now the single source of truth for both.

Additional information

The per-event trace is emitted with LLAMA_LOG_INFO from src/, which maps to verbosity 4 (-lv 4, "trace") in common_log_get_verbosity - i.e. it is hidden at the default -lv 3.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. pi:llama.cpp/Qwen3.8-Flash-Next

[jacekpoplawski]

I'm seeing n_same increasing continuously. I assume these logs were added to help detect cases like this. Is this expected behavior? If more information is needed, I can open an issue.

22.07.558.481 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6688, n_diff = 0
22.07.562.642 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6689, n_diff = 0
22.07.599.424 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6690, n_diff = 0
22.07.603.083 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6691, n_diff = 0
22.07.639.960 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6692, n_diff = 0
22.07.642.931 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6693, n_diff = 0
22.07.679.715 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6694, n_diff = 0
22.07.683.441 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6695, n_diff = 0
22.07.720.196 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6696, n_diff = 0
22.07.724.302 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6697, n_diff = 0
22.07.761.091 I sched_graph_realloc: graph reallocation: nodes = 66 (changed = 0), splits = 2, n_same = 6698, n_diff = 0

[ggerganov]

It's not a huge problem to see these, though it means there is some inefficiency that can be improved. What params are you running with?

(depending on how many reports we get and how well we handle these, I'll consider hiding these logs behind some extra dev flag to not annoy developers/users running with -lv 4)

[jacekpoplawski]

I'm running 4 parallel slots, and I'm currently debugging some issues with slot allocation (or maybe prompt processing again?), so I need verbose logs:

llama-server \
  -sm tensor \
  -lv 4 \
  -m /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-UD-Q6_K_M.gguf \
  -mm /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-mmproj-BF16.gguf \
  --host 0.0.0.0 \
  --jinja \
  -fa on \
  --keep 4096 \
  -b 8192 \
  -ub 512 \
  --parallel 4 \
  --ctx-size 940000 \
  --no-kv-unified \
  --fit-target 1024 \
  --ctx-checkpoints 4 \
  --cache-ram 0 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --presence-penalty 0 \
  --repeat-penalty 1.0 \
  --spec-type ngram-mod \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --chat-template-kwargs '{"preserve_thinking":true}'

I can disable these logs myself, so that's not really an issue for me. My main question is whether this particular use case is useful for your debugging. :)

[ggerganov]

Hm, not sure - I don't reproduce on my end using 2 virtual Metal devices and -hf ggml-org/Qwen3.8-27B-GGUF:Q8_0. Do you do any thing specific to trigger it, or just submit a single prompt?

[sasa7812]

I am bombarded (see the times) on mtp experiment setup:

4.17.677.476 I sched_graph_realloc: graph reallocation: nodes = 7842, splits = 65, n_same = 124, n_diff = 0
4.19.238.603 I sched_graph_realloc: graph reallocation: nodes = 202, splits = 1, n_same = 20, n_diff = 0
4.19.257.752 I sched_graph_realloc: graph reallocation: nodes = 7842, splits = 65, n_same = 125, n_diff = 0
4.19.353.351 I sched_graph_realloc: graph reallocation: nodes = 7842, splits = 65, n_same = 126, n_diff = 0

config:

--model /models/Qwen3.8-Flash-Next/Qwen3.8-Flash-Next-UD-IQ4_XS-merged.gguf
--load-mode none
--lazy-mode auto
--port 8080
--parallel 1
--ctx-size 262144
--flash-attn on
--fit off
--temp 1.0
--min-p 0.0
--top-p 0.95
--top-k 20
--presence-penalty 0.0
--repeat-penalty 1.0
-ngl 99
--split-mode layer
-ts 1,0
--override-tensor "blk\.(2|4|30|3[6-9]|4[0-7])\.ffn_(up|down|gate|gate_up)_(ch|)exps=Vulkan1,per_layer_token_embd=CPU"
-ncmoe 48
--moe-cache-mib 8700
--no-mmproj
-lv 4
--batch-size 4096
--ubatch-size 512
--threads 8
--jinja
--no-kv-unified

--spec-type draft-mtp,ngram-map-k4v
--spec-draft-model /models/Qwen3.8-Flash-Next/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf
--spec-draft-n-max 2
--spec-draft-p-min 0.66
-ngld 0

If run without mtp it is roughly 1 message in 3s of ~25 tps decode.

It is not about logs - can deal with them, but these reallocs do add some cost on decode and maybe the reason mtp is slower (just a guess)

[ggerganov]

@sasa7812 Yes, that's the reason to have them - I usually fix each one when I see it.

I'm aware of the graph reallocs when using --moe-cache-mib - it's most likely because of the different graphs depending on the ubatch size.

[sasa7812]

@ggerganov lmk, if you need more data, I am trying to squeeze maximum of my setup

[ggerganov]

What is your setup?

[sasa7812]

Vulkan, RTX 3090 + RX 7900 XT, 80Gb DDR5 with slow PCIe4x4 RX 7900 connection

[ggerganov]

Some things you can try:

• Use mtp-Qwen3.8-Flash-Next-Q4_0.gguf for the MTP draft model
• Keep it fully on the GPU (i.e. remove the -ngld 0)
• You can try to move it to only one of the cards: f.ex. --spec-draft-devices Vulkan1
• For better prompt processing keep the -ub larger: f.ex -ub 2048 or -ub 4096
• I would drop the manual overrides and just go with -cmoe
• The -ts 1,0 seems wrong - but I don't have much experience with multi-GPU setups
• Try to work the other params around a large moe cache: f.ex --moe-cache-mib 16384. That's the most important param for the TG in this setup.

[sasa7812]

Wow! @ggerganov thanks a lot for that! I really appreciate it.
--spec-draft-devices Vulkan1 - thanks, trying that.
-ts 1,0 - this is to deal with a slow PCIe link, if I put more on Vulkan2 - cache goes there and saturates the link fully, makes things slower. With -ts 1,0 it makes Vulkan1 to act only as MUL_MAT executor of allocated exp layers which doesn't require much traffic.
The -ot cherrypicking of experts to Vulkan1 because exp layers are unevenly quantized in that particular model (I tried your model with the same expert sizes but it is slightly bigger just enough to cause nMaj on RAM portion of things), hence all different layers go to Vulkan1 and Vulkan0 cache gets only layers with the same sizes; that prevents cache fragmentation (bucketing).
Mainly because of that I also have pressure on VRAM for Vulkan0, and cannot give it more ubatch without sacrificing ctx or moe-cache, ctx I need, and moe-cache reduction harms decode more than I get wins from ubatch pp increase.
That's also the reason my moe-cache is on maximum (for the VRAM available on Vulkan0).
This setup is on it's local maximum, that's why I am excited to see that there's a room for improvement in reallocs.
The reason I am saying this, mainly to tell that I am doing a lot of experiments and have a lot of data, and ready to share if that is helpful.

[sasa7812]

You made my day! Your mtp head and moving it to Vulkan1 had a great impact ~25 -> 33 tps on decode, thank you Georgi

[rhjdvsgsgks]

You made my day! Your mtp head and moving it to Vulkan1 had a great impact ~25 -> 33 tps on decode, thank you Georgi

did that solved your reallocation message in log?

[sasa7812]

did that solved your reallocation message in log?

Oh, no. It made things faster. But reallocations do happen quite with a fairly steady rate. My guess would be since it is almost always different experts in place, due to the cache. I am looking into it, but so far I'm not that deep into the kernel internals to come up with a root cause or a solution.