server: save and clear idle slots on new task (--clear-idle)

[yychyo]

In unified KV cache mode, idle slots' KV cells stay in the [0, n_kv) range
and inflate attention cost for all active sequences (even though they're masked).

--clear-idle saves idle slots to --cache-ram and clears them from VRAM, reducing n_kv to only active tokens.

Requires --cache-ram and unified KV cache. Disable with --no-clear-idle.

Benchmarks (agentic workload with multiple agents)

• Qwen 3.5 4B, +25% server: save and clear idle slots on new task (--clear-idle) #20993 (comment)
• Qwen 3.5 27B, +20% server: save and clear idle slots on new task (--clear-idle) #20993 (comment)
• feedback from @strawberrymelonpanda - server: save and clear idle slots on new task (--clear-idle) #20993 (comment)

Open questions

• Misc. bug: llama-server with -kvu and --parallel 4 slows down tg with more inactive slots #19523 (comment) - should we enable this flag by default for CUDA backend only? - All GPU backends iterate [0, n_kv) in attention (Metal/Vulkan skip masked blocks but still iterate).
• decide on server: save and clear idle slots on new task (--clear-idle) #20993 (comment) - do we need a cooldown period? - out of scope of this PR, will create an issue or a draft PR.

Related

• Parallelization / Batching Explanation #4130 (reply in thread)
• KV cache swapping behaviour? #17283
• server : host-memory prompt caching #16391

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES ✨, assisted with research and test scaffolding. All code reviewed and understood.

[ggml-gh-bot]

Hi @yychyo, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[yychyo]

Related

• Parallelization / Batching Explanation #4130 (reply in thread)

[yychyo]

That's makes it possible to read an entry from cache to empty slot.

[yychyo]

Main logic - under feature flag.

[ggerganov]

Isn't it better to clear the idle slot as soon as it becomes idle? I.e. when we call slot.release()?

[yychyo]

Good catch! That is my first contribution, so my vision quite limited - thank you so much for guidance! Done in 2563b4a

[yychyo]

Added f6c4a3d . Also clear idle slots on launch, not just release. Release alone doesn't cover sequential requests (last slot has no reason to clear if nobody else needs VRAM).

[yychyo]

Enables the cache for empty slot.

[ggerganov]

Thanks, this should be a good improvement for unified KV cache setups. I have yet to take a detailed look at how you implemented it. In the meantime, hopefully we get some feedback from people with CUDA backend and -np 4 -kvu - the performance over multiple long requests should degrade much less if this works correctly.

[ggerganov]

I think we might want kv_keep_only_active to always be on by default when kv_unified == true and can be disabled with a CLI argument.

[yychyo]

Done in ced2476

[yychyo]

Thanks, this should be a good improvement for unified KV cache setups. I have yet to take a detailed look at how you implemented it. In the meantime, hopefully we get some feedback from people with CUDA backend and -np 4 -kvu - the performance over multiple long requests should degrade much less if this works correctly.

I've made a bench script for "agentic" - like workflow:

Code

#!/usr/bin/env bash

HOST=127.0.0.1
PORT=8990
URL="http://$HOST:$PORT"
N_PREDICT="${N_PREDICT:-512}"
DELAY="${DELAY:-0}"

SERVER_CMD="${SERVER_CMD:-./build-cuda/bin/llama-server \
    -hf bartowski/Qwen_Qwen3.5-4B-GGUF:IQ4_NL \
    --host $HOST --port $PORT -ngl 99 -c 131072 --cache-ram 40960 \
    --swa-checkpoints 32 --cache-type-k q8_0 --jinja \
    --slots --temp 0.0 -b 256 -ub 256}"

make_prompt() {
    local user=$1 turn=$2
    local filler="The ancient library stood at the edge of the forgotten city where scholars gathered for centuries to study mysteries of the universe and decode symbols carved into walls of the great hall. "
    local grow="Additional context was provided by the researchers who continued their investigation into the deeper layers of understanding that emerged from careful analysis of primary sources and artifacts. "
    local p="Agent ${user}: "
    p+=$(printf "${filler}%.0s" $(seq 1 500))
    for ((r=0; r<(turn+1)*140; r++)); do p+="$grow"; done
    echo "$p"
}

post_agent() {
    local turn=$1 id=$2 outdir="$3"
    local t0=$SECONDS delay=0
    if [[ "$turn" -gt 0 && "$DELAY" -gt 0 ]]; then delay=$((RANDOM % DELAY)); sleep "$delay"; fi
    printf "  → agent %d turn %d%s\n" "$id" "$turn" "$( (( delay )) && echo " (delay=${delay}s)" )" >&2
    local prompt=$(make_prompt "$id" "$turn")
    local resp=$(printf '%s' "$prompt" | jq -Rs --argjson np "$N_PREDICT" '{prompt:., n_predict:$np, cache_prompt:true}' |
        curl -sf "$URL/completion" -H "Content-Type: application/json" -d @-) || { echo "curl failed for agent $id turn $turn" >&2; return 1; }
    echo "$resp" | jq -r --arg t "$turn" --arg aid "$id" '"\($t) \($aid) \(.timings.predicted_per_second) \(.timings.prompt_n)"' > "$outdir/turn${turn}_agent${id}.txt"
    local tg=$(echo "$resp" | jq .timings.predicted_per_second)
    local pn=$(echo "$resp" | jq .timings.prompt_n)
    printf "  ← agent %d turn %d: TG=%.1f prompt_n=%d (%ds)\n" "$id" "$turn" "$tg" "$pn" "$((SECONDS-t0))" >&2
}

if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
  set -euo pipefail
  export SCRIPT_DIR="$(realpath "$(dirname "${BASH_SOURCE[0]}")")"
  export N_AGENTS="${1:-4}"
  export N_WORKERS="${2:-2}"
  export N_PREDICT DELAY
  LOGDIR="logs/bench-$(date +%Y%m%d-%H%M%S)-agentic"
  mkdir -p "$LOGDIR"
  export LOGDIR

  SERVER_PID=""
  stop_server() {
      if [[ -n "$SERVER_PID" ]]; then
          kill "$SERVER_PID" 2>/dev/null || true
          wait "$SERVER_PID" 2>/dev/null || true
          SERVER_PID=""
      fi
  }
  trap stop_server EXIT

  start_server() {
      local logfile="$1"; shift
      # shellcheck disable=SC2086
      $SERVER_CMD "$@" >"$logfile" 2>&1 &
      SERVER_PID=$!
      for _ in $(seq 1 60); do curl -sf "$URL/health" >/dev/null 2>&1 && break; sleep 1; done
      curl -sf "$URL/health" >/dev/null 2>&1 || { echo "server failed to start" >&2; exit 1; }
  }

  wait_port_free() { while curl -sf "$URL/health" >/dev/null 2>&1; do sleep 1; done; }

  run_phase() {
      local label="$1"; shift
      local outdir="$LOGDIR/$label"
      mkdir -p "$outdir"
      export OUTDIR="$outdir"

      echo "--- $label ---" >&2
      start_server "$LOGDIR/$label.log" "$@"
      make -j"$N_WORKERS" AGENTS="$(seq -s' ' 0 $((N_AGENTS-1)))" -f <(cat <<'__EOF'
SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
export BASH_ENV := $(SCRIPT_DIR)/bench-kv-agentic.sh

LAST_TURN := $(foreach a,$(AGENTS),$(OUTDIR)/turn3_agent$(a).txt)

.DELETE_ON_ERROR:
.SUFFIXES:
.PHONY: all
all: $(LAST_TURN)

# chain turns per agent: turn1 depends on turn0, turn2 on turn1, turn3 on turn2
$(foreach a,$(AGENTS),\
  $(foreach t,1 2 3,\
    $(eval $(OUTDIR)/turn$(t)_agent$(a).txt: $(OUTDIR)/turn$(shell echo $$(($(t)-1)))_agent$(a).txt)))

# pattern rule per agent: extract turn from filename stem (turn1_agent0 -> turn=1)
$(foreach a,$(AGENTS),\
  $(eval $(OUTDIR)/turn%_agent$(a).txt: ; @post_agent $$(firstword $$(subst _, ,$$*)) $(a) $(OUTDIR)))
__EOF
    )
      stop_server
      wait_port_free
  }

  echo "=== Agentic Parallel ($N_AGENTS agents, $N_WORKERS workers, n_predict=$N_PREDICT) ==="
  echo "SERVER_CMD: $SERVER_CMD"

  run_phase baseline --no-kv-clear-idle
  run_phase feature

  echo ""
  echo "| Turn | Agent | Baseline TG | Feature TG | Prompt tokens |"
  echo "|------|-------|-------------|------------|---------------|"
  for turn in 0 1 2 3; do
      for agent in $(seq 0 $((N_AGENTS-1))); do
          bl=($(cat "$LOGDIR/baseline/turn${turn}_agent${agent}.txt"))
          ft=($(cat "$LOGDIR/feature/turn${turn}_agent${agent}.txt"))
          printf "| %s | %s | %.1f t/s | %.1f t/s | %s |\n" "$turn" "$agent" "${bl[2]}" "${ft[2]}" "${bl[3]}"
      done
  done
  echo ""
  echo "Logs: $LOGDIR/"
fi

Runs N agents, M active at a time (make -j), with optional random delay between turns. Each agent has a unique prefix and growing context (~20k→32k tokens over 4 turns).

RTX 3090, Qwen3.5-4B IQ4_NL, 8 agents / 4 slots / 2 concurrent, --cache-ram 40960, 131K ctx:

Benchmark

❯ DELAY=5 ./bench-kv-agentic.sh 8 2 2>&1 | tee agentic.log
=== Agentic Parallel (8 agents, 2 workers, n_predict=512) ===
SERVER_CMD: ./build-cuda/bin/llama-server     -hf bartowski/Qwen_Qwen3.5-4B-GGUF:IQ4_NL     --host 127.0.0.1 --port 8990 -ngl 99 -c 131072 --cache-ram 40960     --swa-checkpoints 32 --cache-type-k q8_0 --jinja     --slots --temp 0.
0 -b 256 -ub 256
...

Log

Turn	Agent	Baseline TG	Feature TG	Prompt tokens
0	0	44.8 t/s	44.3 t/s	20425
0	1	42.1 t/s	41.1 t/s	20425
0	2	67.3 t/s	84.4 t/s	20425
0	3	64.7 t/s	88.4 t/s	20425
0	4	29.4 t/s	43.9 t/s	20425
0	5	30.1 t/s	40.8 t/s	20425
0	6	29.7 t/s	43.8 t/s	20425
0	7	71.8 t/s	88.1 t/s	20425
1	0	68.0 t/s	70.2 t/s	3924
1	1	54.2 t/s	83.7 t/s	3924
1	2	51.8 t/s	81.5 t/s	3924
1	3	65.3 t/s	70.0 t/s	3924
1	4	64.8 t/s	78.0 t/s	3924
1	5	50.9 t/s	62.3 t/s	3924
1	6	51.7 t/s	55.8 t/s	3924
1	7	65.2 t/s	58.2 t/s	3924
2	0	66.0 t/s	55.3 t/s	3924
2	1	42.6 t/s	44.3 t/s	3924
2	2	60.5 t/s	35.6 t/s	3924
2	3	47.8 t/s	50.7 t/s	3924
2	4	42.7 t/s	81.1 t/s	3924
2	5	47.8 t/s	54.0 t/s	3924
2	6	67.5 t/s	36.1 t/s	3924
2	7	47.2 t/s	51.1 t/s	3924
3	0	59.9 t/s	48.2 t/s	3924
3	1	28.0 t/s	80.1 t/s	3924
3	2	32.2 t/s	69.4 t/s	3924
3	3	39.2 t/s	58.3 t/s	3924
3	4	44.6 t/s	47.6 t/s	3924
3	5	23.9 t/s	80.3 t/s	3924
3	6	31.5 t/s	68.4 t/s	3924
3	7	61.9 t/s	79.6 t/s	3924

Metric	Baseline	Feature	Delta
TG avg	49.8 t/s	61.7 t/s	+24%
Wall time	321s	213s	-34%
Cache updates avg	1892ms	768ms	-59%

[yychyo]

💡 Idea: cooldown before eviction

In parallel agentic workloads with fast tool calls (seconds apart), slots get evicted and immediately restored. --no-kv-clear-idle N (default 0 = immediate) could add a cooldown before clearing on release.

👍 / 👎 ?

[strawberrymelonpanda]

feedback from people with CUDA backend and -np 4 -kvu
RTX 3090, Qwen3.5-4B IQ4_NL, 8 agents / 4 slots / 2 concurrent, --cache-ram 40960, 131K ctx:

Was just playing around with --parallel and -kvu today. -np 2 and -kvu together seem pretty good on my system with a 3090 and Qwen 27B, but @yychyo's 8 agents, 2 workers, 88+ t/s makes me feel like I'm doing something very wrong. 😆

Haven't had a chance to bench this branch yet, but watching the PR with interest.

[yychyo]

feedback from people with CUDA backend and -np 4 -kvu
RTX 3090, Qwen3.5-4B IQ4_NL, 8 agents / 4 slots / 2 concurrent, --cache-ram 40960, 131K ctx:

Was just playing around with --parallel and -kvu today. -np 2 and -kvu together seem pretty good on my system with a 3090 and Qwen 27B, but @yychyo's 8 agents, 2 workers, 88+ t/s makes me feel like I'm doing something very wrong. 😆

Haven't had a chance to bench this branch yet, but watching the PR with interest.

Thanks for the feedback. My use-case (not only mine, I think it is quite popular nowadays) is a variety of agents, each of them are either a) waiting either on user input or a tool call b) calling llama-server. For the llama-server point of view those are calls with "growing" prompts (with added user input/tool output) that are done sporadically.

The issue I faced is that current (CUDA mostly I think) implementation operates on the whole unified cache (even with non-active sequences in it). So, if you used several agents, and have some slots filled in, and running just 1 sequence, the inference speed will be much worse (25-30%) compared if you runned the same sequence alone.

This PR "offloads" the idle sequences to VRAM. I use llama-server with it locally, and really satisfied how it works with opencode or pi. The "proper" fix would be to make CUDA kernels to be able work efficiently in this use case - do not process what is not needed to be processed (I explored this path a bit, and it seems to be very complex) .

Overall, during work on this PR I got some understanding on further future improvement of the whole caching logic (the most impactful thing for me seems ability to support prefix sharing in ram cache + save/restore path, but that needs discussion, design and so on, and I'm limited to 1 PR as a first-time contributor, so I'll try to get it to completion first)

@strawberrymelonpanda For the -np 2 : this PR should improve the use cases , when you have an active sequence in one of two slots in you system, and not active in the other. Without the PR, llama-server will "waste" compute of operating on the whole unified cache (active sequence, and not active, and then discarding uneeded data with KQ mask). With this PR, unactive sequence will be offloaded to RAM, infrerence should be notable faster. It also depends on the length of you r sequences. If you have sequences in say 8k range, you will barely noticed the difference. If you have them like 32k+ range, it will be noticeable.

[strawberrymelonpanda]

Oh yeah, I think I get the idea and I like it. Right now my main consideration is just:

paraelle 1 = 100K unified fp16 cache
paraelle 2 = 95K
paraelle 4 = 86K
(etc)

-NP 1 runs noticeably slower than -NP 2 in some scenarios, so the 5K ctx loss is worth it. NP 4 however doesn't run noticeably faster in my tests than NP 2, and sometimes actually slower, so losing 9K cache isn't great.

But this branch could make higher parallel values more worthwhile.

[yychyo]

Did a test for qwen3.5-27b.

Code of the bench-kv-agentic.sh

#!/usr/bin/env bash
# Sourced by Make recipes via BASH_ENV for post_agent/make_prompt.
# Executed directly as the main entry point.

HOST=127.0.0.1
PORT=8990
URL="http://$HOST:$PORT"
N_PREDICT="${N_PREDICT:-512}"
DELAY="${DELAY:-0}"

SERVER_CMD="${SERVER_CMD:-./build-cuda/bin/llama-server \
    -hf bartowski/Qwen_Qwen3.5-27B-GGUF:IQ4_NL \
    --host $HOST --port $PORT -ngl 99 -c 131072 --cache-ram 40960 \
    --swa-checkpoints 32 --cache-type-k q8_0 --jinja \
    --slots --temp 0.0 -b 256 -ub 256}"

agents() { seq 0 $((N_AGENTS - 1)); }

make_prompt() {
    local user=$1 turn=$2
    local filler="The ancient library stood at the edge of the forgotten city where scholars gathered for centuries to study mysteries of the universe and decode symbols carved into walls of the great hall. "
    local grow="Additional context was provided by the researchers who continued their investigation into the deeper layers of understanding that emerged from careful analysis of primary sources and artifacts. "
    local p="Agent ${user}: "
    p+=$(printf "${filler}%.0s" $(seq 1 500))
    for ((r=0; r<(turn+1)*140; r++)); do p+="$grow"; done
    echo "$p"
}

post_agent() {
    local turn=$1 id=$2 outdir="$3"
    local t0=$SECONDS delay=0
    if [[ "$turn" -gt 0 && "$DELAY" -gt 0 ]]; then delay=$((RANDOM % DELAY)); sleep "$delay"; fi
    printf "  → agent %d turn %d%s\n" "$id" "$turn" "$( (( delay )) && echo " (delay=${delay}s)" )" >&2
    local prompt=$(make_prompt "$id" "$turn")
    local resp=$(printf '%s' "$prompt" | jq -Rs --argjson np "$N_PREDICT" '{prompt:., n_predict:$np, cache_prompt:true}' |
        curl -sf "$URL/completion" -H "Content-Type: application/json" -d @-) || { echo "curl failed for agent $id turn $turn" >&2; return 1; }
    echo "$resp" | jq -r --arg t "$turn" --arg aid "$id" '"\($t) \($aid) \(.timings.predicted_per_second) \(.timings.prompt_n)"' > "$outdir/turn${turn}_agent${id}.txt"
    local tg=$(echo "$resp" | jq .timings.predicted_per_second)
    local pn=$(echo "$resp" | jq .timings.prompt_n)
    printf "  ← agent %d turn %d: TG=%.1f prompt_n=%d (%ds)\n" "$id" "$turn" "$tg" "$pn" "$((SECONDS-t0))" >&2
}

main() {
    set -euo pipefail
    export SCRIPT_DIR="$(realpath "$(dirname "${BASH_SOURCE[0]}")")"
    export N_AGENTS="${1:-4}"
    export N_WORKERS="${2:-2}"
    export N_PREDICT DELAY
    LOGDIR="logs/bench-$(date +%Y%m%d-%H%M%S)-agentic"
    mkdir -p "$LOGDIR"
    export LOGDIR

    SERVER_PID=""
    stop_server() {
        if [[ -n "$SERVER_PID" ]]; then
            kill "$SERVER_PID" 2>/dev/null || true
            wait "$SERVER_PID" 2>/dev/null || true
            SERVER_PID=""
        fi
    }
    trap stop_server EXIT

    start_server() {
        local logfile="$1"; shift
        # shellcheck disable=SC2086
        $SERVER_CMD "$@" >"$logfile" 2>&1 &
        SERVER_PID=$!
        for _ in $(seq 1 60); do curl -sf "$URL/health" >/dev/null 2>&1 && break; sleep 1; done
        curl -sf "$URL/health" >/dev/null 2>&1 || { echo "server failed to start" >&2; exit 1; }
    }

    wait_port_free() { while curl -sf "$URL/health" >/dev/null 2>&1; do sleep 1; done; }

    run_phase() {
        local label="$1"; shift
        local outdir="$LOGDIR/$label"
        mkdir -p "$outdir"
        export OUTDIR="$outdir"

        echo "--- $label ---" >&2
        start_server "$LOGDIR/$label.log" "$@"
        make -j"$N_WORKERS" AGENTS="$(agents | tr '\n' ' ')" -f <(cat <<'__EOF'
SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
export BASH_ENV := $(SCRIPT_DIR)/bench-kv-agentic.sh

LAST_TURN := $(foreach a,$(AGENTS),$(OUTDIR)/turn3_agent$(a).txt)

.DELETE_ON_ERROR:
.SUFFIXES:
.PHONY: all
all: $(LAST_TURN)

# chain turns per agent: turn1 depends on turn0, turn2 on turn1, turn3 on turn2
$(foreach a,$(AGENTS),\
  $(foreach t,1 2 3,\
    $(eval $(OUTDIR)/turn$(t)_agent$(a).txt: $(OUTDIR)/turn$(shell echo $$(($(t)-1)))_agent$(a).txt)))

# pattern rule per agent: extract turn from filename stem (turn1_agent0 -> turn=1)
$(foreach a,$(AGENTS),\
  $(eval $(OUTDIR)/turn%_agent$(a).txt: ; @post_agent $$(firstword $$(subst _, ,$$*)) $(a) $(OUTDIR)))
__EOF
        )
        stop_server
        wait_port_free
    }

    echo "=== Agentic Parallel ($N_AGENTS agents, $N_WORKERS workers, n_predict=$N_PREDICT) ==="
    echo "SERVER_CMD: $SERVER_CMD"

    run_phase baseline --no-kv-clear-idle
    run_phase feature

    echo ""
    echo "| Turn | Agent | Baseline TG | Feature TG | Prompt tokens |"
    echo "|------|-------|-------------|------------|---------------|"
    for turn in 0 1 2 3; do
        for agent in $(agents); do
            bl=($(cat "$LOGDIR/baseline/turn${turn}_agent${agent}.txt"))
            ft=($(cat "$LOGDIR/feature/turn${turn}_agent${agent}.txt"))
            printf "| %s | %s | %.1f t/s | %.1f t/s | %s |\n" "$turn" "$agent" "${bl[2]}" "${ft[2]}" "${bl[3]}"
        done
    done
    echo ""
    echo "Logs: $LOGDIR/"
}

if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
    main "$@"
fi

So I called it like this : DELAY=5 ./bench-kv-agentic.sh 8 2 2>&1 - it means 8 agents, only 2 active at a time, put 5 sec as a delay to emulate tool call/user input

Results

Turn	Agent	Baseline TG	Feature TG	Prompt tokens
0	0	12.0 t/s	11.3 t/s	20425
0	1	11.0 t/s	12.0 t/s	20425
0	2	22.9 t/s	25.1 t/s	20425
0	3	22.3 t/s	26.1 t/s	20425
0	4	8.9 t/s	11.9 t/s	20425
0	5	8.9 t/s	11.1 t/s	20425
0	6	9.1 t/s	11.9 t/s	20425
0	7	16.2 t/s	19.3 t/s	20425
1	0	16.1 t/s	19.1 t/s	3924
1	1	15.0 t/s	16.5 t/s	3924
1	2	14.5 t/s	16.5 t/s	3924
1	3	15.5 t/s	18.8 t/s	3924
1	4	15.3 t/s	18.6 t/s	3924
1	5	12.4 t/s	16.8 t/s	3924
1	6	12.5 t/s	16.8 t/s	3924
1	7	15.4 t/s	18.5 t/s	3924
2	0	15.0 t/s	18.1 t/s	3924
2	1	13.1 t/s	15.9 t/s	3924
2	2	13.2 t/s	15.8 t/s	3924
2	3	14.5 t/s	17.7 t/s	3924
2	4	14.3 t/s	17.6 t/s	3924
2	5	13.2 t/s	16.6 t/s	3924
2	6	13.2 t/s	16.7 t/s	3924
2	7	14.4 t/s	17.6 t/s	3924
3	0	14.2 t/s	17.2 t/s	3924
3	1	12.7 t/s	13.5 t/s	3924
3	2	12.5 t/s	13.5 t/s	3924
3	3	13.6 t/s	16.8 t/s	3924
3	4	13.3 t/s	16.7 t/s	3924
3	5	11.0 t/s	14.5 t/s	3924
3	6	10.9 t/s	14.6 t/s	3924
3	7	18.2 t/s	22.1 t/s	3924

Overall it's +~20% of TG in this scenario (~13.9 t/s vs ~16.7 t/s avg). It's not a "speed up" per se, it's workaround to avoid slowdown though.

[strawberrymelonpanda]

Ran the branch through my own agent evals, which performance-wise is very naive - just opencode run on CLI against various coding tasks sequentially. Note: The variance between runs is high here because I made this script more for evaluating outcome rather than speed.

I can say I see no regression, and early results good. Looks like I indeed no longer see the slowdown in the NP 4 scenario now that I had mentioned earlier. NP 4 may actually be faster than NP 2 now.

(server restarts between runs)

PR:
NP=2: 7m 11s
NP=4 1st run: 7m 14s <-- on par with np2 on average on PR
NP=4 2nd run: 6m 39s <-- or faster

Master:
NP=2: 7m 35s <-- faster than np4 on average on master
NP=4: 1st run: 7m 56s
NP=4 2nd run: 8m 37s

hopefully we get some feedback from people with CUDA backend and -np 4 -kvu

@ggerganov It's not super scientific, but as a "CUDA -np 4 -kvu" user, it's looking good so far for my use.

[ggerganov]

In parallel agentic workloads with fast tool calls (seconds apart), slots get evicted and immediately restored.

Hm that's a good point. Maybe my suggestion earlier to clear the idle slot immediately on release wasn't good. If you remove that logic, and move the "clear idle slots" logic to run after the new task has been assigned to a slot, would it work better?

[Farmadupe]

probably a bit too late now as the boat has probably sailed, but --clear-idle is a very awkward name.

• it's missing a noun (it clears idle what?)
• 'clear' strongly suggests suggests deletion, but it's actually a caching operation

would have suggested --cache-idle-slots (which would have also paired up with --cache-ram)

[yychyo]

would have suggested --cache-idle-slots (which would have also paired up with --cache-ram)

Fair point on the naming. Since it's enabled by default, most users won't need to type it - --no-clear-idle is the more likely flag in practice.

P.S.: If a rename makes sense, happy to do a follow-up.

[ggerganov]

Yes good point @Farmadupe - let's rename.

[yychyo]

Yes good point @Farmadupe - let's rename.

@ggerganov done:

• server: rename --clear-idle to --cache-idle-slots #21741

[kiwixz]

I'm afraid this breaks the automatic slot selection algorithm. With this PR all slots but the last are considered empty because slot.prompt.tokens gets cleared, so it only ever uses the same slot.

llama.cpp/tools/server/server-context.cpp

Lines 975 to 994 in 45cac7c

	if (slot.is_processing()) {
	continue;
	}
	const auto & tokens = slot.prompt.tokens;
	// skip the slot if it does not contains cached tokens
	if (tokens.empty()) {
	continue;
	}
	// fraction of the Longest Common Prefix length with respect to the input prompt length
	const float sim_cur = float(tokens.get_common_prefix(task.tokens)) / task.tokens.size();
	// select the current slot if the criteria match
	if (sim_cur > sim_best && sim_cur > slot_prompt_similarity) {
	sim_best = sim_cur;
	ret = &slot;
	}

[yychyo]

I'm afraid this breaks the automatic slot selection algorithm. With this PR all slots but the last are considered empty because slot.prompt.tokens gets cleared, so it only ever uses the same slot.

llama.cpp/tools/server/server-context.cpp

Lines 975 to 994 in 45cac7c

	if (slot.is_processing()) {
	continue;
	}
	const auto & tokens = slot.prompt.tokens;
	// skip the slot if it does not contains cached tokens
	if (tokens.empty()) {
	continue;
	}
	// fraction of the Longest Common Prefix length with respect to the input prompt length
	const float sim_cur = float(tokens.get_common_prefix(task.tokens)) / task.tokens.size();
	// select the current slot if the criteria match
	if (sim_cur > sim_best && sim_cur > slot_prompt_similarity) {
	sim_best = sim_cur;
	ret = &slot;
	}

@kiwixz - thanks for the feedback! There is a -no-clear-idle option (which is a subject to rename - #20993 (comment) ), you can use it to opt-out of clearing idle slots.

For the behavior change - I assume it's the expected trade-off. Idle slots are freed aggressively so active sequences don't pay attention cost. Content matching still works via the RAM cache.

Also, there was an idea of cooldown period - #20993 (comment) - it could also help (if implemented) in your case.

WDYT?

[ggerganov]

I'm afraid this breaks the automatic slot selection algorithm.

Why do you think it is a problem? Do you experience any performance regressions? The change should be at very least performance neutral and as @yychyo explained, it will improve backends such as CUDA + unified KV cache where we no longer compute the extra cross-sequence attention. So there shouldn't be a reason to use the -no-clear-idle flag.

[kiwixz]

Selecting the same slot causes cache trashing, this was made worse by this PR but actually was already a problem before.
I think I found a compatible solution in #22083.