[User] The server crashes when a wrong n_keep is set

Prerequisites

I am running the latest code. Development is very rapid so there are no tagged versions as of now.
I carefully followed the README.md.
I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Running the server with a specific context window and then passing a specific wrong n_keep with a specific number of tokens. Since I'm passing a wrong parameter, I expect the server to either give an error or just generate something anyway.

Current Behavior

The server crashes.

Environment and Context

Commit: db3abcc

OS: Kubuntu 23.04

❯ lscpu | grep -P 'Model name|Flags'
Model name: AMD Ryzen 9 7900 12-Core Processor
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d

❯ uname -a
Linux comp 6.2.0-34-generic #34-Ubuntu SMP PREEMPT_DYNAMIC Mon Sep  4 13:06:55 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

❯ make --version | head -1
GNU Make 4.3

❯ g++ --version | head -1
g++ (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0

Steps to Reproduce

I used this model:
https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/blob/main/llama-2-13b-chat.Q4_K_M.gguf
(but it also crashes with other models too)
The server is built with just make, no other params. But it also crashes with LLAMA_CUBLAS=1.
Start the server with the context window of 32:

❯ ./server -m /opt/models/text/llama-2-13b-chat.Q4_K_M.gguf -c 32

startup log

{"timestamp":1696785761,"level":"INFO","function":"main","line":1323,"message":"build info","build":1354,"commit":"db3abcc"}
{"timestamp":1696785761,"level":"INFO","function":"main","line":1325,"message":"system info","n_threads":12,"n_threads_batch":-1,"total_threads":24,"system_info":"AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "}
llama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from /opt/models/text/llama-2-13b-chat.Q4_K_M.gguf (version GGUF V2 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q4_K     [  5120, 32000,     1,     1 ]
llama_model_loader: - tensor    1:           blk.0.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor    2:            blk.0.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor    3:            blk.0.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor    4:              blk.0.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor    5:            blk.0.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor    6:              blk.0.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor    7:         blk.0.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor    8:              blk.0.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor    9:              blk.0.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   10:           blk.1.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   11:            blk.1.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   12:            blk.1.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   13:              blk.1.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   14:            blk.1.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   15:              blk.1.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   16:         blk.1.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   17:              blk.1.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   18:              blk.1.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   19:          blk.10.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   20:           blk.10.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   21:           blk.10.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   22:             blk.10.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   23:           blk.10.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   24:             blk.10.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   25:        blk.10.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   26:             blk.10.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   27:             blk.10.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   28:          blk.11.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   29:           blk.11.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   30:           blk.11.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   31:             blk.11.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   32:           blk.11.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   33:             blk.11.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   34:        blk.11.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   35:             blk.11.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   36:             blk.11.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   37:          blk.12.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   38:           blk.12.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   39:           blk.12.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   40:             blk.12.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   41:           blk.12.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   42:             blk.12.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   43:        blk.12.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   44:             blk.12.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   45:             blk.12.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   46:          blk.13.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   47:           blk.13.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   48:           blk.13.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   49:             blk.13.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   50:           blk.13.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   51:             blk.13.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   52:        blk.13.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   53:             blk.13.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   54:             blk.13.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   55:          blk.14.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   56:           blk.14.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   57:           blk.14.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   58:             blk.14.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   59:           blk.14.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   60:             blk.14.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   61:        blk.14.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   62:             blk.14.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   63:             blk.14.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   64:             blk.15.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   65:             blk.15.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   66:           blk.2.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   67:            blk.2.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   68:            blk.2.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   69:              blk.2.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   70:            blk.2.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   71:              blk.2.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   72:         blk.2.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   73:              blk.2.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   74:              blk.2.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   75:           blk.3.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   76:            blk.3.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   77:            blk.3.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   78:              blk.3.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   79:            blk.3.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   80:              blk.3.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   81:         blk.3.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   82:              blk.3.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   83:              blk.3.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   84:           blk.4.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   85:            blk.4.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   86:            blk.4.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   87:              blk.4.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   88:            blk.4.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   89:              blk.4.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   90:         blk.4.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   91:              blk.4.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   92:              blk.4.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   93:           blk.5.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   94:            blk.5.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor   95:            blk.5.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   96:              blk.5.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor   97:            blk.5.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor   98:              blk.5.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor   99:         blk.5.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  100:              blk.5.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  101:              blk.5.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  102:           blk.6.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  103:            blk.6.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  104:            blk.6.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  105:              blk.6.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  106:            blk.6.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  107:              blk.6.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  108:         blk.6.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  109:              blk.6.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  110:              blk.6.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  111:           blk.7.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  112:            blk.7.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  113:            blk.7.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  114:              blk.7.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  115:            blk.7.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  116:              blk.7.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  117:         blk.7.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  118:              blk.7.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  119:              blk.7.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  120:           blk.8.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  121:            blk.8.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  122:            blk.8.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  123:              blk.8.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  124:            blk.8.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  125:              blk.8.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  126:         blk.8.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  127:              blk.8.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  128:              blk.8.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  129:           blk.9.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  130:            blk.9.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  131:            blk.9.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  132:              blk.9.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  133:            blk.9.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  134:              blk.9.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  135:         blk.9.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  136:              blk.9.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  137:              blk.9.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  138:          blk.15.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  139:           blk.15.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  140:           blk.15.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  141:             blk.15.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  142:           blk.15.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  143:        blk.15.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  144:             blk.15.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  145:          blk.16.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  146:           blk.16.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  147:           blk.16.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  148:             blk.16.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  149:           blk.16.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  150:             blk.16.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  151:        blk.16.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  152:             blk.16.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  153:             blk.16.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  154:          blk.17.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  155:           blk.17.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  156:           blk.17.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  157:             blk.17.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  158:           blk.17.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  159:             blk.17.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  160:        blk.17.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  161:             blk.17.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  162:             blk.17.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  163:          blk.18.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  164:           blk.18.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  165:           blk.18.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  166:             blk.18.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  167:           blk.18.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  168:             blk.18.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  169:        blk.18.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  170:             blk.18.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  171:             blk.18.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  172:          blk.19.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  173:           blk.19.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  174:           blk.19.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  175:             blk.19.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  176:           blk.19.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  177:             blk.19.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  178:        blk.19.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  179:             blk.19.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  180:             blk.19.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  181:          blk.20.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  182:           blk.20.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  183:           blk.20.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  184:             blk.20.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  185:           blk.20.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  186:             blk.20.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  187:        blk.20.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  188:             blk.20.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  189:             blk.20.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  190:          blk.21.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  191:           blk.21.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  192:           blk.21.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  193:             blk.21.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  194:           blk.21.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  195:             blk.21.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  196:        blk.21.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  197:             blk.21.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  198:             blk.21.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  199:          blk.22.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  200:           blk.22.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  201:           blk.22.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  202:             blk.22.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  203:           blk.22.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  204:             blk.22.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  205:        blk.22.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  206:             blk.22.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  207:             blk.22.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  208:          blk.23.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  209:           blk.23.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  210:           blk.23.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  211:             blk.23.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  212:           blk.23.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  213:             blk.23.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  214:        blk.23.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  215:             blk.23.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  216:             blk.23.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  217:          blk.24.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  218:           blk.24.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  219:           blk.24.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  220:             blk.24.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  221:           blk.24.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  222:             blk.24.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  223:        blk.24.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  224:             blk.24.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  225:             blk.24.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  226:          blk.25.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  227:           blk.25.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  228:           blk.25.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  229:             blk.25.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  230:           blk.25.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  231:             blk.25.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  232:        blk.25.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  233:             blk.25.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  234:             blk.25.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  235:          blk.26.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  236:           blk.26.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  237:           blk.26.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  238:             blk.26.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  239:           blk.26.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  240:             blk.26.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  241:        blk.26.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  242:             blk.26.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  243:             blk.26.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  244:          blk.27.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  245:           blk.27.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  246:           blk.27.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  247:             blk.27.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  248:           blk.27.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  249:             blk.27.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  250:        blk.27.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  251:             blk.27.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  252:             blk.27.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  253:          blk.28.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  254:           blk.28.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  255:           blk.28.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  256:             blk.28.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  257:           blk.28.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  258:             blk.28.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  259:        blk.28.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  260:             blk.28.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  261:             blk.28.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  262:          blk.29.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  263:           blk.29.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  264:           blk.29.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  265:             blk.29.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  266:           blk.29.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  267:             blk.29.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  268:        blk.29.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  269:             blk.29.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  270:             blk.29.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  271:           blk.30.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  272:             blk.30.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  273:             blk.30.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  274:        blk.30.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  275:             blk.30.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  276:             blk.30.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  277:                    output.weight q6_K     [  5120, 32000,     1,     1 ]
llama_model_loader: - tensor  278:          blk.30.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  279:           blk.30.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  280:           blk.30.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  281:          blk.31.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  282:           blk.31.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  283:           blk.31.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  284:             blk.31.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  285:           blk.31.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  286:             blk.31.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  287:        blk.31.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  288:             blk.31.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  289:             blk.31.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  290:          blk.32.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  291:           blk.32.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  292:           blk.32.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  293:             blk.32.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  294:           blk.32.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  295:             blk.32.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  296:        blk.32.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  297:             blk.32.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  298:             blk.32.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  299:          blk.33.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  300:           blk.33.ffn_down.weight q4_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  301:           blk.33.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  302:             blk.33.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  303:           blk.33.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  304:             blk.33.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  305:        blk.33.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  306:             blk.33.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  307:             blk.33.attn_v.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  308:          blk.34.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  309:           blk.34.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  310:           blk.34.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  311:             blk.34.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  312:           blk.34.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  313:             blk.34.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  314:        blk.34.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  315:             blk.34.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  316:             blk.34.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  317:          blk.35.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  318:           blk.35.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  319:           blk.35.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  320:             blk.35.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  321:           blk.35.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  322:             blk.35.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  323:        blk.35.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  324:             blk.35.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  325:             blk.35.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  326:          blk.36.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  327:           blk.36.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  328:           blk.36.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  329:             blk.36.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  330:           blk.36.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  331:             blk.36.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  332:        blk.36.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  333:             blk.36.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  334:             blk.36.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  335:          blk.37.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  336:           blk.37.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  337:           blk.37.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  338:             blk.37.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  339:           blk.37.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  340:             blk.37.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  341:        blk.37.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  342:             blk.37.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  343:             blk.37.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  344:          blk.38.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  345:           blk.38.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  346:           blk.38.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  347:             blk.38.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  348:           blk.38.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  349:             blk.38.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  350:        blk.38.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  351:             blk.38.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  352:             blk.38.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  353:          blk.39.attn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  354:           blk.39.ffn_down.weight q6_K     [ 13824,  5120,     1,     1 ]
llama_model_loader: - tensor  355:           blk.39.ffn_gate.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  356:             blk.39.ffn_up.weight q4_K     [  5120, 13824,     1,     1 ]
llama_model_loader: - tensor  357:           blk.39.ffn_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - tensor  358:             blk.39.attn_k.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  359:        blk.39.attn_output.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  360:             blk.39.attn_q.weight q4_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  361:             blk.39.attn_v.weight q6_K     [  5120,  5120,     1,     1 ]
llama_model_loader: - tensor  362:               output_norm.weight f32      [  5120,     1,     1,     1 ]
llama_model_loader: - kv   0:                       general.architecture str
llama_model_loader: - kv   1:                               general.name str
llama_model_loader: - kv   2:                       llama.context_length u32
llama_model_loader: - kv   3:                     llama.embedding_length u32
llama_model_loader: - kv   4:                          llama.block_count u32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32
llama_model_loader: - kv   7:                 llama.attention.head_count u32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32
llama_model_loader: - kv  10:                          general.file_type u32
llama_model_loader: - kv  11:                       tokenizer.ggml.model str
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32
llama_model_loader: - kv  18:               general.quantization_version u32
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type q4_K:  241 tensors
llama_model_loader: - type q6_K:   41 tensors
llm_load_print_meta: format           = GGUF V2 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_head           = 40
llm_load_print_meta: n_head_kv        = 40
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: n_ff             = 13824
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = mostly Q4_K - Medium
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 7.33 GiB (4.83 BPW)
llm_load_print_meta: general.name   = LLaMA v2
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.12 MB
llm_load_tensors: mem required  = 7500.97 MB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size  =   25.00 MB
llama_new_context_with_model: compute buffer total size = 10.75 MB

llama server listening at http://127.0.0.1:8080

{"timestamp":1696785762,"level":"INFO","function":"main","line":1753,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}

Call the API with n_keep = 32 and a specific number of tokens in prompt (not quite sure how many):

❯ curl -sS --data '{"n_predict":32,"n_keep":32,"prompt":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"}' http://127.0.0.1:8080/completion

Failure Logs

The server just crashes with this:

malloc(): unaligned tcache chunk detected
fish: Job 1, './server -m /opt/models/text/ll…' terminated by signal SIGABRT (Abort)

Failure Information (for bugs)

The crash happens only with the parameters described above. For example, I tried other values other than 32, but with them the server works as usual. Maybe it crashes with any other combination of parameters, but I can only reproduce it like this.

I assume that n_keep is wrong because it's the same size as the context, which I assume makes no sense. However, if the prompt is one token longer, the server actually generates something.

1 more token in the prompt

❯ curl -sS --data '{"n_predict":32,"n_keep":32,"prompt":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"}' http://127.0.0.1:8080/completion | jq
{
  "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nCard\n\n\n\n\n",
  "generation_settings": {
    "frequency_penalty": 0,
    "grammar": "",
    "ignore_eos": false,
    "logit_bias": [],
    "mirostat": 0,
    "mirostat_eta": 0.10000000149011612,
    "mirostat_tau": 5,
    "model": "/opt/models/text/llama-2-13b-chat.Q4_K_M.gguf",
    "n_ctx": 32,
    "n_keep": 28,
    "n_predict": 32,
    "n_probs": 0,
    "penalize_nl": true,
    "presence_penalty": 0,
    "repeat_last_n": 64,
    "repeat_penalty": 1.100000023841858,
    "seed": 4294967295,
    "stop": [],
    "stream": false,
    "temp": 0.800000011920929,
    "tfs_z": 1,
    "top_k": 40,
    "top_p": 0.949999988079071,
    "typical_p": 1
  },
  "model": "/opt/models/text/llama-2-13b-chat.Q4_K_M.gguf",
  "prompt": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
  "stop": true,
  "stopped_eos": false,
  "stopped_limit": true,
  "stopped_word": false,
  "stopping_word": "",
  "timings": {
    "predicted_ms": 3919.781,
    "predicted_n": 31,
    "predicted_per_second": 7.908605098090939,
    "predicted_per_token_ms": 126.44454838709677,
    "prompt_ms": 1535.19,
    "prompt_n": 31,
    "prompt_per_second": 20.192940287521413,
    "prompt_per_token_ms": 49.52225806451613
  },
  "tokens_cached": 31,
  "tokens_evaluated": 35,
  "tokens_predicted": 31,
  "truncated": true
}

Also, the server crashes only on the very first request. If the abovementioned request is done after some other request, the server does not crash.

I can reproduce the problem.

Near the top of nextToken() (I added the printf):

for (size_t i = params.n_keep + 1 + n_discard; i < embd.size(); i++)
{
    printf("--> embd[%zu] = embd[%zu] (%d, %d, %zu)\n", i - n_discard, i, params.n_keep, n_discard, embd.size());
    embd[i - n_discard] = embd[i];
}

With -c 32 and your example curl command:

--> embd[29] = embd[15] (28, -14, 32)
--> embd[30] = embd[16] (28, -14, 32)
--> embd[31] = embd[17] (28, -14, 32)
--> embd[32] = embd[18] (28, -14, 32)

Since embd.size() == 32, of course embd[32] is not a valid index.

edit: Pretty sure something really serious is going wrong, as far as I can see n_left, n_discard should never be negative.

This actually weirder than just the size. Using the example with everything 32, there has to be at least 28 whitespace characters at the start of the prompt. For example aaaa triggers it (28 spaces + 4 a) but 27 spaces + 5 a doesn't. Even weirder, 28 spaces + 1 a + 3 more spaces doesn't trigger it.

I couldn't find anywhere in the code where it seemed to be doing whitespace trimming so I really have no idea of what's going on, but I'm pretty sure it's not just a problem with the calculating but something else affected by the actual prompt content that's causing this.

Yes, the bug happens only with a specific set of arguments and prompt length. I stumbled upon it by pure chance. The other weird thing is that the bug does not happen after any other successful request is performed. Maybe it has something to do with the caching.

Also, the prompt doesn't have to be whitespace-only or even have any whitespace to begin with. Initially I had this crash on a long random text (1000+ tokens, but the context window was still 32), where if you remove one word, the crash disappears. It also crashes on a sequence of ":"

❯ curl --data '{"n_predict":32,"n_keep":32,"prompt":":::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::"}' http://127.0.0.1:8080/completion

Again, remove or add any ":" and the crash does not happen.

Initially I thought that the crash solely depends on the number of tokens, however it doesn't seem to be the case. My first example with newlines has 33 tokens (according to the /tokenize endpoint), but the example with colons has 39 tokens. If I reduce the second prompt to 33 tokens, the server responds as usual.

Really weird. I may look into this more if I get a chance but I'm not really too familiar with the server (I've actually barely even used it). Maybe someone who knows more about that will respond, the bug label may make it more visible. This definitely seems like a bug, it's possible it may affect the other examples as well since they use similar logic.

This actually weirder than just the size. Using the example with everything 32, there has to be at least 28 whitespace characters at the start of the prompt. For example aaaa triggers it (28 spaces + 4 a) but 27 spaces + 5 a doesn't. Even weirder, 28 spaces + 1 a + 3 more spaces doesn't trigger it.

Going out of memory bounds in C/CPP doesn't immediately guarantee "crash", as memory blocks aren't fenced exactly on the boundary

It's rather "normal" (typical) to have to overflow in a specific and nondeterministic way in order to cause hard crash.

Many many years ago, there was libefence, I have no idea if it still exists or is compatible with modern compilers, but that was a library, which you could simply ld_preload, which replaced malloc with one that guarantees tight fencing, forcing crash on even most minute out of bounds operations. It was very very useful for debugging problems like this.

I think you can get the same bounds checking with address sanitizer.

I actually did my tests with address sanitizing turned on so I'm pretty confident this isn't the symptom of earlier memory corruption or something like that.

The initial calculation that results in negative values is definitely wrong, but the first actual invalid memory access was when it accessed embd 1 past its valid range. quick edit: So when I first managed to trigger it, I meant when the sanitizer saw the invalid access. After that, I actually copied out the embd[blah] = embd[blah] part and just printed out the values: address sanitizing didn't report any incorrect accesses at that point, but from the debug output I could see the values were definitely wrong. There weren't any other actual invalid memory accesses though.

I looked into server.cpp and there are a couple of bugs in the loadPrompt() for these borderline cases.

In particular I believe the logic for trimming the prompt in case it is too big for the context is flawed and leads to situations where there is no space for the next token which causes the segfault you mentioned, because n_past isn't initialized properly.

Also, the last_n_token array isn't initialized correctly in this case.

I created a pull request to fix in #3639

Give it a try.

Can confirm that with the patch in #3639, the server does not crash anymore on the provided examples.

@z80maniac

I did another attempt at fixing this in #3661 after @z80maniac found a performance regression in my patch #3639.

This was fixed. Not sure in which commit though.

Navigation Menu

Repository navigation

[User] The server crashes when a wrong `n_keep` is set #3550

Prerequisites

Expected Behavior

Current Behavior

Environment and Context

Steps to Reproduce

Failure Logs

Failure Information (for bugs)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[User] The server crashes when a wrong n_keep is set #3550

Description

Prerequisites

Expected Behavior

Current Behavior

Environment and Context

Steps to Reproduce

Failure Logs

Failure Information (for bugs)

Activity

KerfuffleV2 commented on Oct 8, 2023

KerfuffleV2 commented on Oct 9, 2023

z80maniac commented on Oct 9, 2023

KerfuffleV2 commented on Oct 9, 2023

staviq commented on Oct 9, 2023

slaren commented on Oct 9, 2023

KerfuffleV2 commented on Oct 9, 2023

coezbek commented on Oct 15, 2023

z80maniac commented on Oct 16, 2023

coezbek commented on Oct 18, 2023

z80maniac commented on Oct 21, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[User] The server crashes when a wrong `n_keep` is set #3550