MiniCPM 2b model support?

[KnutJaegersberg]

Feature Description

Like Phi is supported, it would great to have this Mistral level 2b model ggufable.

Motivation

SOTA 2b model, a piece of art, read how they made it:

https://shengdinghu.notion.site/MiniCPM-Unveiling-the-Potential-of-End-side-Large-Language-Models-d4d3a8c426424654a4e80e42a711cb20

[ggerganov]

Seems like the only unusual thing about this architecture is some modification related to mixing the input/output embeddings of the layers:

From this description, I'm not 100% sure what it means, but I suppose it would not be too difficult to implement

[raymond-infinitecode]

Most impressive 2B model ever see.

[ShengdingHu]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.

Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:

Modification Name	Specific Operation
Embedding Output Scaling	We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling	The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling	The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.

These modifications can also be seen from our huggingface transformers code.

Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

[sweetcard]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.

Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:

Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.

Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository.
Could u push it back here ?

@ShengdingHu

[TonloOO]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?

@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight

, resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

[sweetcard]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:

zkh2016/llmfarm_core.swift@c7de12d

[TonloOO]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:

zkh2016/llmfarm_core.swift@c7de12d

Thanks, problem solved! But the output makes no sense

[sweetcard]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:
zkh2016/llmfarm_core.swift@c7de12d

Thanks, problem solved! But the output makes no sense

Is the prompt template is correct ?
Check this file :

https://github.com/OpenBMB/LLMFarm-MiniCPM/blob/main/LLMFarm/model_setting_templates/llama%20chat%202%207B%20iphone%2012.json

[lin-calvin]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:
zkh2016/llmfarm_core.swift@c7de12d

Thanks, problem solved! But the output makes no sense

Maybe Issue of tokenizer?

[runfuture]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.

Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:

Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.

Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

The information you summarized from the paper is very helpful. Thank you for your work.

[ShengdingHu]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?

@ShengdingHu

Sure, we will look into it today!

[ShengdingHu]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:
zkh2016/llmfarm_core.swift@c7de12d

Thanks, problem solved! But the output makes no sense

Is the prompt template is correct ? Check this file :

https://github.com/OpenBMB/LLMFarm-MiniCPM/blob/main/LLMFarm/model_setting_templates/llama%20chat%202%207B%20iphone%2012.json

Thanks, the output might not be due to the template. Could you point me to the code that's generating the nonsensical output? I'm a bit lost in all the information. Is it directly produced by our zkh2016/llmfarm_core.swift@c7de12d?

[TonloOO]

Thank you for interest in MiniCPM. I am one of the authors. In MiniCPM, we implement tie_word_embedding, which involves utilizing the same matrix for both input embedding and the output projection (lm_head). To adapt this from a standard architecture like Llama, you would need to make adjustments such as replacing lm_head.projection with something like input_embedding.projection.
Additionally, our model incorporates $\mu$P (https://arxiv.org/abs/2203.03466), a technique that applies numeric scaling to various model parameters and forward hiddens. Here are some specific modifications related to inference (not training) we've made:
Modification Name Specific Operation
Embedding Output Scaling We multiply the output of the embedding by scale_emb=12.
Residual Connection Scaling The increment at each residual connection in every layer is scaled by scale_depth/√(num_layers), which equals 1.4/√(40).
lm_head Scaling The output logits are adjusted to 1/(dim_model/256) = 1/9 of their original value.
These modifications can also be seen from our huggingface transformers code.
Llama.cpp is an exceptional framework for running edge-side LLMs. Please feel free to contact me at shengdinghu@gmail.com if you have any questions or need further clarification.

There is a branch of llama.cpp to support Minicpm in your official repository. Could u push it back here ?
@ShengdingHu

The branch here appears to have an issue with it. I am using the openbmb/MiniCPM-2B-dpo-fp16 model, when using the python3 convert.py model, the converted model lacks an output.weight , resulting in an error. Additionally, the convert-hf-to-gguf.py script has not yet implemented support for MiniCPM, leading to the error: "Architecture 'MiniCPMForCausalLM' not supported!"

Some changes are applied to other llama.cpp in this project:
zkh2016/llmfarm_core.swift@c7de12d

Thanks, problem solved! But the output makes no sense

Is the prompt template is correct ? Check this file :
https://github.com/OpenBMB/LLMFarm-MiniCPM/blob/main/LLMFarm/model_setting_templates/llama%20chat%202%207B%20iphone%2012.json

Thanks, the output might not be due to the template. Could you point me to the code that's generating the nonsensical output? I'm a bit lost in all the information. Is it directly produced by our zkh2016/llmfarm_core.swift@c7de12d?

Not really, I actually referenced from this repo, and integrated it with zkh2016/llmfarm_core.swift@c7de12d