server : make 2 checkpoints near the end of the prompt by ggerganov · Pull Request #20288 · ggml-org/llama.cpp

ggerganov · 2026-03-09T12:21:33Z

fix #20239 (comment)

In some cases, reprocessing the last 512 tokens of the prompt could be too slow. In other cases it is necessary in order to allow mutating the last user message.

Make 2 checkpoints to satisfy all needs.

alex2robotic · 2026-03-09T15:16:29Z

fix #20239 (comment)

In some cases, reprocessing the last 512 tokens of the prompt could be too slow. In other cases it is necessary in order to allow mutating the last user message.

Make 2 checkpoints to satisfy all needs.

Thanks, this helps a lot.

From reading the patch, my understanding is that the new logic intentionally keeps two near-end checkpoints:

one around n_ubatch before the end of the prompt
one around 64 tokens before the end

My guess is that this is a tradeoff between:

reducing TTFT for replay/regenerate of the same request, and
still allowing the last user message to be edited without falling back too far.

Is that the right way to think about it?

If so, what is the main practical benefit of still keeping a small tail (for example ~64 tokens) uncheckpointed, instead of trying to checkpoint all the way to the end of the prompt? Is that mainly for robustness when the last user message changes slightly or prompt boundaries shift?

Also, in my testing, this behavior does not show up with Qwen3-30B-A3B: replay/regenerate there can have near-zero prompt re-eval, while Qwen3.5-35B-A3B re-evaluates the tail much more noticeably. Is that difference mainly because Qwen3-30B-A3B can reuse the live KV/cache path more directly, while Qwen3.5-35B-A3B relies more on this checkpoint heuristic due to its hybrid/recurrent-like architecture?

Finally, do you think near-zero or zero prompt re-eval is achievable in principle for Qwen3.5 regenerate/replay, or is some small amount of tail reprocessing still expected by design?

ggerganov · 2026-03-09T15:54:55Z

Is that the right way to think about it?

Yes, that's correct.

If so, what is the main practical benefit of still keeping a small tail (for example ~64 tokens) uncheckpointed, instead of trying to checkpoint all the way to the end of the prompt? Is that mainly for robustness when the last user message changes slightly or prompt boundaries shift?

The main restriction is to guarantee that no "reasoning" tokens will get included in the checkpoint because they will be removed for the next user message. I guess 64 tokens is quite big, considering that typically only one "reasoning" token gets added (f.ex <think>). We should probably reduce this to 4?

schynce · 2026-03-09T18:31:43Z

Thanks a lot for this! I just tested this branch and It is working well and reduces the time to first token quite a lot :)

ggerganov · 2026-03-09T18:40:08Z

Thanks, could you confirm that changing the secondary checkpoint from 64 -> 4 tokens works OK?

schynce · 2026-03-09T19:23:00Z

Thanks, could you confirm that changing the secondary checkpoint from 64 -> 4 tokens works OK?

It seems to be working okay with 4 as well for Qwen3.5 with the default template in both instruct and reasoning modes.

ggerganov · 2026-03-09T19:27:23Z

@pwilkin @aldehir Is my assumption correct that "begin thinking" tokens that get appended after a new user message and before the generation starts, cannot be more than 4 (typically a single token)? Or do we know about models with longer "begin thinking" incantations?

aldehir · 2026-03-09T19:58:25Z

The longest I know is gpt-oss at 3 tokens: <|channel|> analysis, <|message|>.

Seems like a safe assumption.

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

Upstream breaks the prompt batch twice near the end so checkpoints can be taken (PR ggml-org#20288, offsets {4 + n_ubatch, 4}). Each break is a separate llama_decode, and a decode boundary drains the whole pipeline -- a cost proportional to stage count on a multi-stage RPC split. Measured on DeepSeek-V4-Flash across 10 stages (8 BC-250s + shredder x2), 15,360-token real-text prompts: the two breaks showed up in the daemon graph traces as gaps of 15.0 s and 10.3 s in a 60 s board span, against a median inter-graph gap of 0.11 s. The deep break only buys a fallback checkpoint n_ubatch tokens further back. Per-turn restores come from the user-message-boundary rule above it, and end-of-prompt continuation from the shallow break, so drop it by default and keep upstream's pair behind LLAMA_CKPT_DEEP_BREAK=1. Prefill 180.3 -> 205.4 t/s warm (+13.9%), 158.1 -> 182.0 cold; the board trace now shows exactly one drain and duty rises 50% -> 66%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…a one Upstream cuts the prompt batch 4 + n_ubatch and 4 tokens before the end so a checkpoint can be snapshotted before a decode, when the memory holds everything except the batch about to run (PR ggml-org#20288). Each cut is a separate llama_decode, and a decode boundary drains the whole pipeline. That cost is proportional to backend count and, crucially, independent of how many tokens the extra decode carries: on a 10-stage RPC split the surviving 4-token break still cost 11.6 s of an 85 s request. Snapshot in post_decode() instead, right after the decode that completes the prompt. That decode has already drained the pipeline so the caller can sample, so the state read rides a synchronization that is paid anyway, and the checkpoint covers the whole prompt instead of stopping 4 tokens short -- a strictly later recovery point. LLAMA_CKPT_PROMPT_BREAKS = 0 (default, deferred) | 1 (shallow break) | 2 (upstream's pair), replacing LLAMA_CKPT_DEEP_BREAK from the previous commit. DeepSeek-V4-Flash, 8 BC-250s + shredder x2, 15,360-token real-text prompts: breaks cold warm 2 (upstream) 158.1 180.3 1 (shallow only) 182.0 205.4 0 (deferred) 180.7 239.8 0 + checkpoints disabled 184.0 243.0 <- ceiling +33% over upstream and within 1.3% of the ceiling; the residual is the 17 MiB state read itself. Restore verified end to end: a request diverging from the cached sequence re-processes 10 tokens of 1210 -- checking checkpoint with [1199, 1199] against 1200... restored context checkpoint (pos_min = 1199, n_past = 1200) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

…rs with measured data The doc previously said the fork's percentages do not transfer because the port leaves out the value-based eviction policy, without saying what the ported mechanism actually delivers. It now has the A/B measurement (10 turns per side, qwen35 hybrid model, feature flag the only difference): feature off 21 tokens / ~426 ms per turn (restores the prompt_n-4 checkpoint) feature on 18 tokens / ~382 ms per turn (restores the prompt_n-1 snapshot) No overlap across ten turns, and the arithmetic explains it exactly: the saving is an absolute 3 tokens of replay per turn. Upstream already places boundary checkpoints at 4+n_ubatch and 4 before the end of a prompt (PR ggml-org#20288), and the fork's own base carries that same array, so this port effectively replaces the -4 checkpoint with a -1 snapshot. Hence the gain is inversely proportional to suffix length: -14% of tokens at a ~17-token suffix, ~-2.3% at 128. The fork's +14.08% at +128 is not reproduced and should not be used for planning; the likely regime (nearest surviving checkpoint far from the prompt end) was not reachable because neither run filled the checkpoint list. Also recorded: the feature is now actually exercised rather than assumed (test_previous_recurrent_snapshot passes with max logit drift=0, top1=314/314, and the server log shows a replay_boundary=1 checkpoint created at prompt_n-1), plus the finding that test_rollback's dirty-context failure pre-exists this port - identical numbers after reverting all 12 files and rebuilding, and none of the three forks merged here ever touched the recurrent memory or delta-net files.

…ry prompt Two halves of one measured loss (qwen38-27b-q8-tp, 8011-token prompt, 2026-09-16): the server's end-of-prompt checkpoint logic (ggml-org#20288) breaks the prompt at 4 + n_ubatch and at 4 tokens before the end, so every request became three llama_decode calls [N-2052][2048][4]; the 2048-token tail had only two sub-batches and the meta overlap refused to run below four, and the batch before it got a ragged sub-batch. 885 -> 930 t/s at 8k. - llama-context: k_meta_overlap_min_subbatches 4 -> 2 (the rolling loop only needs a pair). - server: under WP_SPEC_PREFILL_NO_CHECKPOINTS=1 keep only the final (4-before-end) checkpoint, which is the one a continuation on a recurrent / SWA model needs; the mid-prompt one is dropped. The env had silently stopped applying to these models when the "structurally required" gate landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BEPveQj6uvGKsJBVqx6Jo2

…in through a different door He runs the notary desk at the county clerk's office. The rule on the wall is from 1987 and it is a good rule: one stamp per eight thousand pages, nobody needs a certified copy of a ledger that changed by one line. He follows it. Walk in with a ledger, he checks the last stamp, he sends you home. The side door is for new business. A new matter walks in, it gets a stamp on the spot, no questions, the rule on the wall is for the front door. Every afternoon a man comes in the front with his ledger, gets stamped, walks three steps down the hall, and comes back through the side door with the same ledger, because the hall is three steps long and the side door is right there. The notary sees new business. He stamps. One hundred and twenty-three milliseconds, a hundred and fifty megabytes of certified copy, every afternoon, of a page that differs from the last one by the word "the". The side door now has a window. He looks through it, sees the man leaving the front desk, and goes back to his sandwich. The man still walks the hall. He has nowhere else to be. --- tools/server/server-context.cpp, the do_checkpoint guard in the prompt batch loop ([TAG_CKPT_DEDUP], ~46 lines). Trace (2026-10-22): a short chat turn opens at the cached end P with checkpoint A (free: [TAG_REC_REUSE] publishes the .rec snapshot into it). The batch loop breaks at the last user-message start, 3 tokens later on a Qwen template ("<|im_end|>\n" closing the previous reply), and the next batch starts there. is_last_user_message (and near_prompt_end on any turn shorter than n_ubatch) bypass the checkpoint_min_step test, the state_gen no longer matches the snapshot, create_checkpoint() does a full PARTIAL_ONLY readback: ~123 ms and 149.6 MiB per turn for a state 3 tokens from one we already hold. --checkpoint-min-step 8192 never got a vote. Rule: skip when a checkpoint of this prompt timeline (token_prefix_hash still matches the current tokens) sits within min-step before this position. Kept regardless: checkpoint A itself (n_tokens_start == n_prompt_cached; the rec-reuse path only feeds A) and the two prompt-tail checkpoints at 4 and 4 + n_ubatch before the end (PR ggml-org#20288), which are the rollback target when the next request re-sends the prompt with a different tail (the Qwen template that strips prior-turn think). Rollback safety: the restore search takes the newest checkpoint with pos_min <= thold and falls back to older ones; nothing looks a checkpoint up by exact position (merge_checkpoint_spills dedups by pos_min); dropping the user-start one restores A, 3 tokens earlier. The production log pairs that looked like the 3-token case (220728 then 222566, 223078 then 223728) are A followed by the tail checkpoints at default verbosity; the 3-token pair only shows at SLT_TRC. The cost was in the timer, not the log. Tests (CPU, offline, skip-uncached-presets plugin): ctest test-generate-models, test-recurrent-state-rollback, test-prompt-cache-disk 3/3; pytest test_checkpoint_rollback, test_checkpoint_rollback_recurrent, test_prompt_cache_disk, test_slot_save, test_completion, test_spec_prebatch: 67 passed, 7 failed, all pre-existing (6 slot-save idle-slot-clear, 1 needs Phi-3.5 uncached). Not yet run on the 27B: expected effect is one readback per short turn instead of two; verify with LLAMA_SERVER_UBATCH_STATS n_checkpoints_created per turn.

server : make 2 checkpoints near the end of the prompt

8e56d7d

github-actions Bot added examples server labels Mar 9, 2026

ggerganov mentioned this pull request Mar 9, 2026

Eval bug: forcing full prompt re-processing due to lack of cache data in Qwen3.5 27B #20153

Closed

cont : adjust checkpoints

63a4450

ggerganov marked this pull request as ready for review March 10, 2026 06:02

ggerganov requested a review from ngxson as a code owner March 10, 2026 06:02

ggerganov merged commit a7b3dee into master Mar 10, 2026
16 of 75 checks passed

ggerganov deleted the gg/server-ckpt-near-end branch March 10, 2026 12:28

krystophny mentioned this pull request Mar 11, 2026

server: fix multi-turn cache reuse for hybrid/recurrent models #20428

Closed

srogmann mentioned this pull request Mar 14, 2026

server : speculative checkpointing #19493

Merged

crsawyer mentioned this pull request Mar 17, 2026

Misc. bug: Server 500 error (gpt-oss, AutoParser, cache) #20532

Closed

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

server : make 2 checkpoints near the end of the prompt (ggml-org#20288)

43de3b0

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026

server : make 2 checkpoints near the end of the prompt (ggml-org#20288)

a7b1513

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

ggerganov mentioned this pull request May 22, 2026

server: fix checkpoints creation #22929

Merged

fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026

server : make 2 checkpoints near the end of the prompt (ggml-org#20288)

d2cbe10

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026

server : make 2 checkpoints near the end of the prompt (#20288)

bcac744

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

Iwan-Teague mentioned this pull request Aug 18, 2026

Misc. bug: Context checkpoints always invalidated on hybrid/recurrent models #24055

Open

zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026

server : make 2 checkpoints near the end of the prompt (ggml-org#20288)

c70398a

* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints

masterFoad mentioned this pull request Sep 3, 2026

server : fix context checkpoint eviction on prompts shorter than checkpoint_min_step #28302

Merged

Tough-Respawn mentioned this pull request Oct 18, 2026

server : let the last-user-end checkpoint supersede earlier ones #30745

Draft

Navigation Menu

Repository navigation

server : make 2 checkpoints near the end of the prompt - #20288

server : make 2 checkpoints near the end of the prompt#20288
ggerganov merged 2 commits into
masterfrom
gg/server-ckpt-near-end

ggerganov commented Mar 9, 2026

Uh oh!

alex2robotic commented Mar 9, 2026 •

edited

Loading

Uh oh!

ggerganov commented Mar 9, 2026

Uh oh!

schynce commented Mar 9, 2026

Uh oh!

ggerganov commented Mar 9, 2026

Uh oh!

schynce commented Mar 9, 2026

Uh oh!

ggerganov commented Mar 9, 2026 •

edited

Loading

Uh oh!

aldehir commented Mar 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ggerganov commented Mar 9, 2026

Uh oh!

alex2robotic commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Mar 9, 2026

Uh oh!

schynce commented Mar 9, 2026

Uh oh!

ggerganov commented Mar 9, 2026

Uh oh!

schynce commented Mar 9, 2026

Uh oh!

ggerganov commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldehir commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alex2robotic commented Mar 9, 2026 •

edited

Loading

ggerganov commented Mar 9, 2026 •

edited

Loading

aldehir commented Mar 9, 2026 •

edited

Loading