Repository navigation
Eval bug: forcing full prompt re-processing due to lack of cache data in Qwen3.5 27B #20153
Description
Name and Version
version: 8210 (2cd20b7)
built with Clang 19.1.5 for Windows x86_64
Operating systems
Windows
GGML backends
CUDA
Hardware
2x RTX 3090
Models
bartowski/Qwen_Qwen3.5-27B-GGUF in Q8_0
Problem description & steps to reproduce
I run
llama-server -hf bartowski/Qwen_Qwen3.5-27B-GGUF `
-hff Qwen_Qwen3.5-27B-Q8_0.gguf `
-ngl all -fit on --swa-full
Then interact with model in llama-server's WebUI.
First Bad Commit
No response
Relevant log output
Logs
>llama-server -hf bartowski/Qwen_Qwen3.5-27B-GGUF -hff Qwen_Qwen3.5-27B-Q8_0.gguf -ngl all -fit on --swa-full ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes load_backend: loaded CUDA backend from C:\Users\lost\AppData\Local\Programs\llama.cpp\ggml-cuda.dll load_backend: loaded RPC backend from C:\Users\lost\AppData\Local\Programs\llama.cpp\ggml-rpc.dll load_backend: loaded CPU backend from C:\Users\lost\AppData\Local\Programs\llama.cpp\ggml-cpu-haswell.dll common_download_file_single_online: no previous model file found C:\Users\lost\AppData\Local\llama.cpp\bartowski_Qwen_Qwen3.5-27B-GGUF_preset.ini common_download_file_single_online: HEAD failed, status: 404 no remote preset found, skipping common_download_file_single_online: using cached file (same etag): C:\Users\lost\AppData\Local\llama.cpp\bartowski_Qwen_Qwen3.5-27B-GGUF_Qwen_Qwen3.5-27B-Q8_0.gguf main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true build: 8210 (2cd20b7) with Clang 19.1.5 for Windows x86_64 system info: n_threads = 8, n_threads_batch = 8, total_threads = 16system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 750,800,860,890,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Running without SSL
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model 'C:\Users\lost\AppData\Local\llama.cpp\bartowski_Qwen_Qwen3.5-27B-GGUF_Qwen_Qwen3.5-27B-Q8_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 3090): 24575 total, 23781 used, -793 free vs. target of 1024
llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 3090): 24575 total, 23351 used, -339 free vs. target of 1024
llama_params_fit_impl: projected to use 47132 MiB of device memory vs. 46000 MiB of free device memory
llama_params_fit_impl: cannot meet free memory targets on all devices, need to use 3180 MiB less in total
llama_params_fit_impl: context size reduced from 262144 to 211712 -> need 3794 MiB less memory in total
llama_params_fit_impl: entire model should be fit across devices by reducing context
llama_params_fit: failed to fit params to free device memory: n_gpu_layers already set by user to -2, abort
llama_params_fit: fitting params to free memory took 0.41 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:0a:00.0) - 23300 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090) (0000:0b:00.0) - 23300 MiB free
llama_model_loader: loaded meta data with 44 key-value pairs and 851 tensors from C:\Users\lost\AppData\Local\llama.cpp\bartowski_Qwen_Qwen3.5-27B-GGUF_Qwen_Qwen3.5-27B-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000
llama_model_loader: - kv 5: general.name str = Qwen3.5 27B
llama_model_loader: - kv 6: general.basename str = Qwen3.5
llama_model_loader: - kv 7: general.size_label str = 27B
llama_model_loader: - kv 8: general.license str = apache-2.0
llama_model_loader: - kv 9: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-2...
llama_model_loader: - kv 10: general.tags arr[str,1] = ["image-text-to-text"]
llama_model_loader: - kv 11: qwen35.block_count u32 = 64
llama_model_loader: - kv 12: qwen35.context_length u32 = 262144
llama_model_loader: - kv 13: qwen35.embedding_length u32 = 5120
llama_model_loader: - kv 14: qwen35.feed_forward_length u32 = 17408
llama_model_loader: - kv 15: qwen35.attention.head_count u32 = 24
llama_model_loader: - kv 16: qwen35.attention.head_count_kv u32 = 4
llama_model_loader: - kv 17: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 18: qwen35.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 19: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 20: qwen35.attention.key_length u32 = 256
llama_model_loader: - kv 21: qwen35.attention.value_length u32 = 256
llama_model_loader: - kv 22: qwen35.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 23: qwen35.ssm.state_size u32 = 128
llama_model_loader: - kv 24: qwen35.ssm.group_count u32 = 16
llama_model_loader: - kv 25: qwen35.ssm.time_step_rank u32 = 48
llama_model_loader: - kv 26: qwen35.ssm.inner_size u32 = 6144
llama_model_loader: - kv 27: qwen35.full_attention_interval u32 = 4
llama_model_loader: - kv 28: qwen35.rope.dimension_count u32 = 64
llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 30: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,247587] = ["─á ─á", "─á─á ─á─á", "i n", "─á t",...
llama_model_loader: - kv 34: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 35: tokenizer.ggml.padding_token_id u32 = 248044
llama_model_loader: - kv 36: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 37: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 38: general.quantization_version u32 = 2
llama_model_loader: - kv 39: general.file_type u32 = 7
llama_model_loader: - kv 40: quantize.imatrix.file str = /models_out/Qwen3.5-27B-GGUF/Qwen_Qwe...
llama_model_loader: - kv 41: quantize.imatrix.dataset str = /training_dir/calibration_datav5.txt
llama_model_loader: - kv 42: quantize.imatrix.entries_count u32 = 496
llama_model_loader: - kv 43: quantize.imatrix.chunks_count u32 = 802
llama_model_loader: - type f32: 449 tensors
llama_model_loader: - type q8_0: 402 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 26.69 GiB (8.52 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 5120
print_info: n_embd_inp = 5120
print_info: n_layer = 64
print_info: n_head = 24
print_info: n_head_kv = 4
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 6
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 17408
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 6144
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 48
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 27B
print_info: model params = 26.90 B
print_info: general.name = Qwen3.5 27B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 '─è'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 63 repeating layers to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors: CPU_Mapped model buffer size = 1288.28 MiB
load_tensors: CUDA0 model buffer size = 12764.92 MiB
load_tensors: CUDA1 model buffer size = 13273.46 MiB
.............................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 211712
llama_context: n_ctx_seq = 211712
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = true
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (211712) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 3.79 MiB
llama_kv_cache: CUDA0 KV buffer size = 6616.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 6616.00 MiB
llama_kv_cache: size = 13232.00 MiB (211712 cells, 16 layers, 4/1 seqs), K (f16): 6616.00 MiB, V (f16): 6616.00 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 311.72 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 286.78 MiB
llama_memory_recurrent: size = 598.50 MiB ( 4 cells, 64 layers, 4 seqs), R (f32): 22.50 MiB, S (f32): 576.00 MiB
llama_context: pipeline parallelism enabled
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: CUDA0 compute buffer size = 2069.28 MiB
sched_reserve: CUDA1 compute buffer size = 1362.07 MiB
sched_reserve: CUDA_Host compute buffer size = 1674.08 MiB
sched_reserve: graph nodes = 8409 (with bs=512), 4713 (with bs=1)
sched_reserve: graph splits = 3
sched_reserve: reserve took 500.80 ms, sched copies = 4
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv load_model: initializing slots, n_slots = 4
common_speculative_is_compat: the target context does not support partial sequence removal
srv load_model: speculative decoding not supported by this context
slot load_model: id 0 | task -1 | new slot, n_ctx = 211712
slot load_model: id 1 | task -1 | new slot, n_ctx = 211712
slot load_model: id 2 | task -1 | new slot, n_ctx = 211712
slot load_model: id 3 | task -1 | new slot, n_ctx = 211712
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use --cache-ram 0 to disable the prompt cache
srv load_model: for more info see https://github.com//pull/16391
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv log_server_r: done request: GET / 127.0.0.1 200
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 211712, n_keep = 0, task.n_tokens = 17
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot init_sampler: id 3 | task 0 | init sampler, took 0.18 ms, tokens: text = 17, total = 17
slot update_slots: id 3 | task 0 | prompt processing done, n_tokens = 17, batch.n_tokens = 17
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot print_timing: id 3 | task 0 |
prompt eval time = 559.29 ms / 17 tokens ( 32.90 ms per token, 30.40 tokens per second)
eval time = 18981.35 ms / 383 tokens ( 49.56 ms per token, 20.18 tokens per second)
total time = 19540.65 ms / 400 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 399, truncated = 0
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.208 (> 0.100 thold), f_keep = 0.038
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 399, total state size = 174.569 MiB
srv load: - looking for better prompt, base f_keep = 0.038, sim = 0.208
srv update: - cache state: 1 prompts, 174.569 MiB (limits: 8192.000 MiB, 211712 tokens, 211712 est)
srv update: - prompt 00000167B543A000: 399 tokens, checkpoints: 0, 174.569 MiB
srv get_availabl: prompt cache update took 90.53 ms
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 384 | processing task, is_child = 0
slot update_slots: id 3 | task 384 | new prompt, n_ctx_slot = 211712, n_keep = 0, task.n_tokens = 72
slot update_slots: id 3 | task 384 | n_past = 15, slot.prompt.tokens.size() = 399, seq_id = 3, pos_min = 398, n_swa = 1
slot update_slots: id 3 | task 384 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com//pull/13194#issuecomment-2868343055)
slot update_slots: id 3 | task 384 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot init_sampler: id 3 | task 384 | init sampler, took 0.01 ms, tokens: text = 72, total = 72
slot update_slots: id 3 | task 384 | prompt processing done, n_tokens = 72, batch.n_tokens = 72
slot print_timing: id 3 | task 384 |
prompt eval time = 402.91 ms / 72 tokens ( 5.60 ms per token, 178.70 tokens per second)
eval time = 128850.60 ms / 2704 tokens ( 47.65 ms per token, 20.99 tokens per second)
total time = 129253.51 ms / 2776 tokens
slot release: id 3 | task 384 | stop processing: n_tokens = 2775, truncated = 0
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.269 (> 0.100 thold), f_keep = 0.025
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 2775, total state size = 323.096 MiB
srv load: - looking for better prompt, base f_keep = 0.025, sim = 0.269
srv update: - cache state: 2 prompts, 497.664 MiB (limits: 8192.000 MiB, 211712 tokens, 211712 est)
srv update: - prompt 00000167B543A000: 399 tokens, checkpoints: 0, 174.569 MiB
srv update: - prompt 000001566DA8F2E0: 2775 tokens, checkpoints: 0, 323.096 MiB
srv get_availabl: prompt cache update took 257.52 ms
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 3089 | processing task, is_child = 0
slot update_slots: id 3 | task 3089 | new prompt, n_ctx_slot = 211712, n_keep = 0, task.n_tokens = 260
slot update_slots: id 3 | task 3089 | n_past = 70, slot.prompt.tokens.size() = 2775, seq_id = 3, pos_min = 2774, n_swa = 1
slot update_slots: id 3 | task 3089 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com//pull/13194#issuecomment-2868343055)
slot update_slots: id 3 | task 3089 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot init_sampler: id 3 | task 3089 | init sampler, took 0.02 ms, tokens: text = 260, total = 260
slot update_slots: id 3 | task 3089 | prompt processing done, n_tokens = 260, batch.n_tokens = 260
slot print_timing: id 3 | task 3089 |
prompt eval time = 384.33 ms / 260 tokens ( 1.48 ms per token, 676.49 tokens per second)
eval time = 179466.39 ms / 3764 tokens ( 47.68 ms per token, 20.97 tokens per second)
total time = 179850.73 ms / 4024 tokens
slot release: id 3 | task 3089 | stop processing: n_tokens = 4023, truncated = 0
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 0.528 (> 0.100 thold), f_keep = 0.064
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 4023, total state size = 401.110 MiB
srv load: - looking for better prompt, base f_keep = 0.064, sim = 0.528
srv update: - cache state: 3 prompts, 898.774 MiB (limits: 8192.000 MiB, 211712 tokens, 211712 est)
srv update: - prompt 00000167B543A000: 399 tokens, checkpoints: 0, 174.569 MiB
srv update: - prompt 000001566DA8F2E0: 2775 tokens, checkpoints: 0, 323.096 MiB
srv update: - prompt 00000167B570A320: 4023 tokens, checkpoints: 0, 401.110 MiB
srv get_availabl: prompt cache update took 228.99 ms
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 6854 | processing task, is_child = 0
slot update_slots: id 3 | task 6854 | new prompt, n_ctx_slot = 211712, n_keep = 0, task.n_tokens = 489
slot update_slots: id 3 | task 6854 | n_past = 258, slot.prompt.tokens.size() = 4023, seq_id = 3, pos_min = 4022, n_swa = 1
slot update_slots: id 3 | task 6854 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com//pull/13194#issuecomment-2868343055)
slot update_slots: id 3 | task 6854 | n_tokens = 0, memory_seq_rm [0, end)
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot init_sampler: id 3 | task 6854 | init sampler, took 0.05 ms, tokens: text = 489, total = 489
slot update_slots: id 3 | task 6854 | prompt processing done, n_tokens = 489, batch.n_tokens = 489
srv operator(): operator(): cleaning up before exit...
srv stop: cancel task, id_task = 6854
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - CUDA0 (RTX 3090) | 24575 = 1396 + (21761 = 12764 + 6927 + 2069) + 1416 |
llama_memory_breakdown_print: | - CUDA1 (RTX 3090) | 24575 = 1512 + (21538 = 13273 + 6902 + 1362) + 1525 |
llama_memory_breakdown_print: | - Host | 2962 = 1288 + 0 + 1674 |
Activity
Dampfinchen commentedon Mar 6, 2026
This is not a llama.cpp problem, it is a limitation in the model architecture. RNN's like Qwen 3.5 can't reuse the cache once the max context is exceeded. I know it sucks, it makes this model unusable for me.
It is not a lcpp bug however.
codebest-og commentedon Mar 8, 2026
Need some clarification on this, I'm seeing the issues on often the second prompt when the context window is very small, nowhere near max,
NovNovikov commentedon Mar 8, 2026
Yes, it happens when the context is not even near the limit.
Skylux70 commentedon Mar 8, 2026
I can confirm, It happens from context 0 to context 262144. Can repeat easily in llama web UI.
My Model: Unsloth/Qwen3.5-27B-Q5
GPU: 5090
ggerganov commentedon Mar 9, 2026
These logs show that you started a new request with 489 tokens prompt, and only the first 258 tokens matched the existing prefix of the current 4023 tokens prompt. It's normal to reprocess in this case, so this is not a bug.
lostmsu commentedon Mar 9, 2026
@ggerganov as you can see in the repro steps I used llama-server's built-in Web UI, which should have sent the prefix correctly. Skylux70 above confirms this happens for them also on different hardware. Please reconsider.
ggerganov commentedon Mar 9, 2026
@lostmsu Your issue is that you use very short prompts. The server currently does not cache prompts that are less than 512 tokens long. This will be addressed in #20288
server: recover from checkpoint on truncation failure for hybrid/recu…