Repository navigation
[Feature Request] Support InternLM Deploy #3133
Description
Dear llama.cpp developer,
Greetings! I am vansinhu, a community developer and volunteer at InternLM. InternLM is a large language model similar to llama2, and we look forward to InternLM being supported in llama.cpp. If there are any challenges or inquiries regarding support for InternLM, please feel free to join our Discord discussion at https://discord.gg/gF9ezcmtM3.
Best regards,
vansinhu
Reacted by Sam Purkis, lin72h and Zheng Qu
Reacted by Cran-May and lin72h
Activity
l29ah commentedon Nov 18, 2023
I've tried to run internlm-20b, converted and quantized apparently fine, but when running it i get
GGML_ASSERT: llama.cpp:2349: codepoints_from_utf8(word).size() > 0.Support attention_bias on LLaMA architecture
llama : support attention bias on LLaMA architecture (#4283)
[auto] Sync version 2312020046.0.0+llamacpp-release.b1601
Mixtral-ROCm Fanservice Edition
gaord commentedon Jan 15, 2024
try running internLM-chat-20b, convert and quantize fine, but server emit the following error while hosting the model. Any ideas for it working. Thanks.
{"timestamp":1705310958,"level":"INFO","function":"main","line":2971,"message":"HTTP server listening","port":"8086","hostname":"127.0.0.1"}
llama_model_loader: loaded meta data with 19 key-value pairs and 543 tensors from ../internlm-chat-20b-q5_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = LLaMA v2
llama_model_loader: - kv 2: llama.context_length u32 = 4096
llama_model_loader: - kv 3: llama.embedding_length u32 = 5120
llama_model_loader: - kv 4: llama.block_count u32 = 60
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 13824
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 40
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 40
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 10: general.file_type u32 = 8
llama_model_loader: - kv 11: tokenizer.ggml.model str = llama
llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,103168] = ["", "
", "", "<0x00>", "<...llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,103168] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,103168] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 = 2
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - type f32: 121 tensors
llama_model_loader: - type q5_0: 421 tensors
llama_model_loader: - type q6_K: 1 tensors
GGML_ASSERT: llama.cpp:2977: codepoints_from_utf8(word).size() > 0
Squash commits until Mixtral support for merging
github-actions commentedon Apr 3, 2024
This issue was closed because it has been inactive for 14 days since being marked as stale.
chigkim commentedon Jul 19, 2024
Still not supported, should be reopen
shaunck96 commentedon Aug 11, 2024
how to ensure llamacpp hosted internlm gguf version leverages GPU using installation ? any specific version of llamacpp I should be using ?
shaunck96 commentedon Aug 11, 2024
@chigkim you need to upgrade llamacpp version to load the internlm quantized version. But I cant seem to get GPU working for inference
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (ggml-org#4283)
llama : support attention bias on LLaMA architecture (#4283)