[Feature request] Hybrid model cache: add --checkpoint-every-nb

[aoleg]

Mainline llama.cpp added ggml-org#20087

I suggest adding a UI feature for hybrid models (such as Nemotron-H, Qwen 3.5, Jamba2-Mini) in the Context section to account for this feature (notably, context shifting is not supposed to work with RNN models, so these should likely be mutually exclusive).

Essentially, this creates cache checkpoints after processing every n batches during prompt processing. Why: it is really helpful when using apps such as SillyTavern that can insert an "anchoring" prompt N messages above the user's prompt (this is common practice to make the model adhere to system prompt guidelines). Without this feature, RNN models invalidate the whole cache, and the entire prompt must be reprocessed. With this feature, however, only the n batches of prompt caching are invalidated and must be reprocessed, which speeds up the usage significantly.

[LostRuins]

Yes we have that, its enabled by default in RNN models now if you select context shift, it will attempt to do this instead,

you can control the cache slots with --smartcache N. it will intelligently set and clear the checkpoints.

[aoleg]

No, this is something else. The PR I mentioned was only merged 5 days ago, please check it out: ggml-org#20087

What you are referring to is SmartCache, which (as far as I understand it) creates a checkpoint in RAM after processing the whole prompt. 20087 provides a better/more complete fix for the issue.

Basically, PR ggml-org#20087 adds intermediate prefix checkpoints calculated automatically during prefill, at every n batches, so hybrid/recurrent models are more likely to successfully resume from an earlier saved prefix. While it stores several snapshots at different prefix lengths, which is similar to the option you mentioned, the idea here is more advanced, as these checkpoints are created automatically during the prefill and not after it. So when SillyTavern invalidates e.g. the last 1000 tokens by inserting a system message at level 4, llama.cpp can reuse one of the automatically created checkpoints and recompute only the suffix after that point.

I don't know how to better explain it, but in practice I find that, with ggml-org#20087, llama.cpp will not reprocess the prompt more often than, currently, koboldcpp. Which is why I suggested koboldcpp to also include this feature.

[LostRuins]

KoboldCpp uses its own caching algorithm, it caches during prefill too. It uses a 3 part cache, one done slightly before the end of prefill, one done after prefill is complete, and one done after generation is complete.

https://github.com/LostRuins/koboldcpp/blob/concedo_experimental/gpttype_adapter.cpp#L4464-L4482

Have you tried it?
It is incorrect to assume that our implement always matches upstream. Rather than spamming snapshots every N tokens, koboldcpp intelligently processes the prompt until X (currently set to 32) tokens before the end of the input. This is usually enough to remove boundary tokens as well as short inputs or trimmed portions of a user's previous input

[aoleg]

This. 32 tokens is obviously too little for when SillyTavern inserts anchoring message at depth 4 (default); that's 4 messages higher than the current prompt, which obviously breaks caching with koboldcpp in this scenario. How can we change this value in koboldcpp? Is there a command line option? A reasonable value for me personally would be 2048 or even 4096 tokens.
(I'm talking about things like Character Notes and Author Notes; they are widely used to ground the model into desired character voice, for example. They reinforce system messages, which have diminishing effect as the context grows.)

[LostRuins]

Hmm... The 3 part caching is triggered every turn. However, by the time 4 turns have passed its likely the checkpoint has been rotated out.

For shorter contexts you can use a higher number of checkpoints e.g. --smartcache 15, however for longer stories this becomes a little too RAM intensive. I will have to figure out a better snapshotting strategy.

[LostRuins]

I improved the snapshotting behavior, running it with about 8 slots should now be enough for this case in the next release

[wypiki]

I'm currently using OpenClaw with KoboldCpp, Qwen 3.5 122B with 128 GB of VRAM on Strix Halo, and SmartCache somehow most of the time doesn't keep KoboldCPP from reprocessing around 50000 tokens around every turn. RAM is maxed out pretty quickly with those smart cache slots, too. Those snapshots are a few hundred megabytes each and the NVMe SSD is capable of around 3 Gbit/s. So having around 100 slots ready to load into and out of the VRAM which shouldn't take more than a second would be very nice solution to make up for the slow prompt processing of the Strix Halo. Could that be done?

[LostRuins]

Ahh openclaw. Openclaw changes the context contents very frequently. Caching won't really work for it because it's constantly swapping different stuff

[wypiki]

With LMStudio the context caching worked really well with openclaw using checkpoints and rarely needed full reprocessing, but then unfortunately it crashed after 100.000-300.000 processed tokens every time which KoboldCPP doesn't.

[LostRuins]

I'm adjusting the snapshot logic, hopefully the next version should require less reprocessing

[EmanuelOverride]

I guess I'm late to the party on this, as I'm only now discovering that my previous stable strategies for dynamic data insertion, author's notes etc, completely break down with RNN/Hybrid models, triggering reprocessing of the entire context. What solution did you go for in the end, @LostRuins ?

Thinking about how I would attempt to resolve it within my own means, I considered adding an async safety margin between stable data and volatile/dynamic fields. Something like:

"...[Turn_5][Turn_6][X]~[Author's_Note][Turn_7]"

If enabled, a user hits "Send" and the model processes the input up to the [X], generates 1 token that gets discarded and triggers the save-state, then resumes from that save-state as if this chunk of data was recently added, carrying on as normal, and at the end of their output, no save-state needs to be triggered. [Edit: In fact, now that I think about it, the model should theoretically prep the snapshot in the background while the user is typing, making the eventual response feel instantaneous, similar to how snapshots are already saved at the end of output. "Pre-compute" until [X] after every output. Well, it's an option.]

For setting the [X] I considered either,

Token slider: Set the number of tokens to count from the most recent ones, to truncate or "cut" the prompt in half, and we determine the exact window for dynamic data injections, and the minimum number of tokens we can expect to be reprocessed on each turn. Works fine if insertion at depth likewise counts tokens, just set it to +1 of whatever that value is.

Regex marker: Pick any obscure symbol that isn't being used for anything, like ⛆, or a specific string, make it inert and invisible to the model so that there's no chance of it being generated, and place it at the very front of whatever dynamic data, author's note etc. That's the stopping point for the first leg of the prompt, the intermediary save-state creation loop, and the resumption point for the second leg. A more flexible and dynamic approach, in case the dynamic injection field needs to grow -- maybe we're doing variable image analysis at the very bottom, who knows? Won't ever trigger full reprocessing, but less control over the expected number of reprocessed tokens per turn.

I think that might be feasible to implement as part of a front-end, though maybe you had a more elegant approach in mind.

[LostRuins]

I did a very simple approach of snapshotting the context near the end (~30 tokens) of each processing run for every turn. It allows rewinding up to that point which includes changing the latest instruction, and adjusting the boundary tokens near the end of the AI's last input.

[Elypha]

Hi @LostRuins ! Jumping in on this thread. SmartCache handles boundary tokenisation brilliantly, but I want to raise a workload that currently falls through the cracks and propose a complementary mode.

SillyTavern Dynamic Tails -
The background is that, when using ST character cards, the context typically has a massive stable prefix (20k~ tokens of lore/persona/past messages) and a highly dynamic tail. Every turn, the last 500 to 3,000 tokens change constantly due to rotating summaries, dynamic World Info injection, and injected author's notes (e.g., fixed at @depth 4+).

Because SmartCache snapshots roughly at end - 32 and relies on an exact match, a tail that changes further back than 32 tokens causes a total cache miss. We end up doing a full prefill every time. (Aoleg's earlier point about the @depth 4 anchor is exactly the same issue). For pure transformer models they can build a workaround with context shift, but for hybrid models there isn't currently such a good solution afaik.

Could we add an additional periodic-checkpoint mode alongside SmartCache?

The Idea is that, similar to what llama.cpp does (ggml-org#19264, ggml-org#16382, ggml-org#20087), we take periodic prefill checkpoints (e.g., every N tokens) and use longest-common-prefix matching on restore.
For hybrid/recurrent models, these partial snapshots are tiny and cheap, making dense checkpointing practical.
This can be added separately or built into SmartCache. For the latter case, we could maybe add a flag (like --smartcache-interval N) to feed the cache slot pool with these periodic hooks too. Restore just picks whichever matches the longest prefix.

Ollama has a similar method (https://github.com/ollama/ollama/blob/main/kvcache/recurrent.go#L13-L17) that it builds 24 segments * 1664 tokens from the bottom of the prompt specifically for recurrent models. I've tried that and the solution was verified for my case - instead of processing the whole prompt (150 seconds with KCPP+current SmartCache) only the changed tail needs to be processed (12 seconds with Ollama's checkpoints). I had a quick scan of llama.cpp's work on this and if I'm not mistaken it seems that most of the required infrastructure/API is already production ready, which might be good news.

[EmanuelOverride]

It solves the problem in the event that the dynamic tail falls within the set interval, but if out of alignment, worst case we go back by an extra segment. I suspect that in the majority of (maybe all?) cases, we have at most three logical segments, of a [Static Data] block (world info, instructions etc), [Stable History] (interaction) and the [Dynamic Tail], and breaking them up into further sub-segments seems like extra trouble for no additional benefit. I guess the benefit is "brute force" of setting any random interval and forgetting, and having it "work" even if sub-optimally.

Static & Stable blocks do work as a single segment, until and unless we reset the interaction, which means that there is some benefit to keeping the two apart. I still think a "smart" system would look for delineators in the context, to figure out with perfect precision where to introduce a "cut", though I'm making the assumption here that there's no reason why segments would need to be uniform, correct?

[LostRuins]

KoboldCpp actually does this, as you play the session and it gets longer you accumulate snapshots. Eventually once the limit is reached, the least use snapshot is dropped and replaced.

So since you say RNN is cheap and can afford more snapshots, simply increase the --smartcache amount, maybe --smartcache 30 or something. That way you approximately have the last 30 turns, and if say you need to rewind to turn 18 for some reason, it will be able to do so.

So long as the "mega memory prefix" is accessed somewhat often it shouldn't ever be evicted. And if you've left it alone for long enough that even 30 turns evicts the base memory, then you weren't referencing it that much anyway and a full reprocess shouldn't be an issue.