server: add ctx-per-slot (--kv-unified-per-slot)

[bartowski1182]

Overview

Adds two new arguments for llama-server context provisioning

kv-unified-per-slot

First argument, --kv-unified-per-slot

This specifies how much context each slot is limited to in the kvu mode, so for example if a user wants to have N slots with each having 4096 context, they can specify --kv-unified-per-slot 4096, it will provision 16384 total context:

./build/bin/llama-server -m Qwen3.5-4B-Q4_K_M.gguf --kv-unified-per-slot 4096
...
0.02.635.267 W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
...
0.03.025.317 I srv    load_model: capping per-slot context (16384) to --kv-unified-per-slot (4096)
...
0.03.084.366 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 4096
0.03.084.370 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 4096
0.03.084.370 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 4096
0.03.084.370 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 4096
...

If specified with -c, you will then limit the total context:

./build-cpu/bin/llama-server -m Qwen3.5-4B-Q4_K_M.gguf -c 8192 --kv-unified-per-slot 4096
...
0.02.650.929 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
...
0.02.921.467 I srv    load_model: capping per-slot context (8192) to --kv-unified-per-slot (4096)
...
0.02.973.263 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 4096
...

If you set -c to a value lower than --kv-unified-per-slot, it will warn and cap to -c's value:

load_model: --kv-unified-per-slot (4096) exceeds the per-slot pool capacity (2048) - cap has no effect, slots are limited to 2048 (raise the KV pool with -c, or unset -c to size it to n_parallel * kv_unified_per_slot)

removed ctx-pool-slots

Details

ctx-pool-slots

Also adds another new flag, --ctx-pool-slots, which specifies how many slots worth of context should be allocated, this is similar to setting -c to an explicit value but doesn't require the user to manually do the math ahead of time:

./build-cpu/bin/llama-server -m Qwen3.5-4B-Q4_K_M.gguf --kv-unified-per-slot 4096 --ctx-pool-slots 2
...
0.02.650.929 W llama_context: n_ctx_seq (8192) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
...
0.02.921.467 I srv    load_model: capping per-slot context (8192) to --kv-unified-per-slot (4096)
...
0.02.973.263 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 4096
0.02.973.267 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 4096
...

If you specify more pool slots than np, it will warn and clamp to np:

srv llama_server: --ctx-pool-slots (6) exceeds n_parallel (4), clamping

Since this is purely a helper argument, I wouldn't mind dropping it from the PR, I think it would be nice to have but I've been told the server arguments are already getting a bit bloated...

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, used for brainstorming and code changes/testing

[bartowski1182]

@ngxson can you take a look?

[ngxson]

IMO this change is quite excessive:

• --ctx-per-slot make sense, but can be implemented as a simple n_ctx = n_ctx_per_slot * n_parallel, there is no need to implement any changes on server
• --ctx-pool-slots is not very intuitive to understand. I don't think it worth adding because the use case is quite too small. you can simply do -c $(( 4096*2 )) in bash, which looks even cleaner

[bartowski1182]

What do you mean by your first point?

Right now I don't believe there is any way to limit how much context an individual slot can use when in unified mode, the only limit is the training context

I don't think setting n_ctx will affect anything in unified mode specifically, that would work for non-unified

As for the second, yeah I just liked it as a minor convenience, happy to remove it if it's too excessive

[bartowski1182]

Removed the ctx-pool-slots and put ctx-per-slot to an int (was intending to later add a max argument but will put it back to stoi if and when that happens)

I double checked and I'm 95% sure there is no way to limit the context per slot without adding the server code, there's probably a different way to do it in the server code, but something needs to happen there regardless

[bartowski1182]

Any other thoughts @ngxson ?

[adderek]

I am using

llama-server -m bartowski-Qwen_Qwen3.6-27B-IQ4_NL.gguf \
--port 8081 --cache-type-k turbo4 --cache-type-v turbo4 \
-fa on -ngl 99 -b 4096 -ub 4096 -t 12 --offline --models-max 1 --cont-batching --tools all --jinja \
--cache-ram 8192 --temperature 0.10 --ctx-checkpoints 4 --reasoning-budget 256 \
--spec-type ngram-cache --spec-draft-n-max 16 \
--kv-offload \
-c 200000 -np 4

and the last part is the key: context 200k num-processes 4
each gets 50k
Isn't that the same effect as you would get with -ctx-per-slot 50000 ?

What I would instead like is to have 1 slot with 100k and 2 slots with 50k each. Currently I can have non-unified with each slot divided equally or unified with context limit for total... and this is probably what I should be using for my own agent :) like --kv-unified -c 200000 -np3 allowing any to take up to 200k but crash (?) when 200k total is reached.

[ngxson]

Right now I don't believe there is any way to limit how much context an individual slot can use when in unified mode, the only limit is the training context

hmm right, I was thinking about per-request max_tokens but that only limit the number of generated tokens, not whole context size

since this feature is KV-unified-only, I'd suggest renaming it to --kv-unified-per-slot

Will be ok to merge after the renaming