Misc. bug: Server often does "selected slot by LRU" despite high level of similarity with earlier prompts

[Nekotekina]

Name and Version

version: 7218 (ec18edf)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Other? (Please let us know in description)

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Problem description & steps to reproduce

When doing consecutive requests to the server, sometimes it selects existing slot by similarity. But sometimes, it does not, it switches to new slot by LRU (least recent) and recomputes the prompt. It doesn't make sense, I couldn't find any clue about how it "chooses" the desired behaviour and undesired one. I used --keep -1, otherwise it doesn't do any similarity check at all?

First Bad Commit

I'll try to debug it myself later, it seems easier than do bisecting.

Relevant log output

[adhusch]

Same problem here on b7244, but i cannot see any connection to the --keep parameter (also NOT using context shift in my case so to my understanding of --keep it should not matter?)

[ggerganov]

--keep does not matter. Provide logs and repro.

The similarity thold can be controlled the -sps CLI argument.

[adhusch]

Thanks! In the meantime i noticed that it does seem to work normal for a short context variant of the same model but fails for long context, after realizing this and reading a bit through server-context.cpp i set --slot-prompt-similarity 0.01, assuming that the similarty value is to some extend sequence length dependent.

This indeed had an effect and revealed i get unexpectedly short LCPs, matching only like 10% of the squence that should share >90% of common prefix, however for for 1:1 regeneration of the same prompt as immedatly before LCP similarity computes correctly to 1, so that looks good. Will rule out client issue/unexpected non derministic behaviour of the input sequence first, there is a potential that this is not a llama.cpp issue at all but just surfaces here in this specific way...

[ggerganov]

@adhusch Which client are you using? It is possible that the client decides to insert stuff at the start of the context, so maybe this is causing it.

[adhusch]

@ggerganov yes, thats exactly what i am suspecting now, its openwebui in this case, with some fixed extracted documents after the system prompt. The behaviour is rather new, KV caching worked very fine for this in the past, i will evalute on that end now and have a look if the requests are prefix-stable. Looks like i started debbung exactly on the wrong end, sorry for that...

[Nekotekina]

I've had a linear multi-turn chat in SillyTavern. Sometimes the server picks up slot by LCP, sometimes it doesn't. SillyTavern prints json of the requests in terminal, and I didn't notice anything suspicious in it (that would warrant extremely low similarity, for example).

[adhusch]

For my side i finally found the time to disect this in detail and can confirm a bug in openwebui, they since some time (accidently i guess) inject provided "full context" knowledge always at the end of the chat flow, which in turn breaks any kind of prefix caching for each follow up question up to the system/user prompt part. For long contexts, the common prefix then falls below the default 0.1 threshold of LCP matching and leads to the LRU selection message in llama server, but its very bad in any case and in no way a llama error, it just got visible here. Will open bug report over there. Thank you very much.

[adhusch]

@Nekotekina if you could provide your json logging i could have a look to your case. Feel free to reach out.

[Nekotekina]

@adhusch Thanks. I did some debugging and it's most likely an issue with SillyTavern. I'll close this issue for now.

slot get_availabl: id  1 | task -1 | common = 89, sim_cur = 0.020
slot get_availabl: id  2 | task -1 | common = 82, sim_cur = 0.019
slot get_availabl: id  3 | task -1 | common = 79, sim_cur = 0.018
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1