Breaking change of models since PR #252

[PriNova]

After the PR #252, all base models need to be converted new.

For me, this is a big breaking change. The LoRa and/or Alpaca fine-tuned models are not compatible anymore.
Reconverting is not possible.

I see from the PR, that the tokenizer scores are written into the model.
Would it make sense to write the tokenizer scores into a seperate file to stay compatible with the (old) models?
The question then arrises, if

1. by loading the model the scoring file will be checked of existense and the sentencepiece tokenizer will be used, or
2. the user can decide which tokenizer to use.

What you think?

[loretoparisi]

I have the same issue, I cannot convert alpaca-lora models. I had to checkout previous commit then:

git checkout 5cb63e2493c49bc2c3b9b355696e8dc26cdd0380

[eiz]

Can you try this convert script? https://gist.github.com/eiz/828bddec6162a023114ce19146cb2b82 (it outputs .tmp files, you can uncomment the os.rename to do it in place if you want but I didn't want to overwrite your files)

[loretoparisi]

@eiz okay thanks, where I find the tokenizer file?

[eiz]

If you don't have access to the original LLaMA files I think someone uploaded it here https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer.model

[Puncia]

Can you try this convert script? https://gist.github.com/eiz/828bddec6162a023114ce19146cb2b82 (it outputs .tmp files, you can uncomment the os.rename to do it in place if you want but I didn't want to overwrite your files)

I'm calling python3 .\convert.py .\models\7B\ .\models\tokenizer.model from the llama directory but it doesn't seem to do anything. Doesn't even produce errors. Am I using it wrong?

EDIT: NEVERMIND I was just executing a blank .py file, I need to sleep. Thank you for your script

[loretoparisi]

confirmed it worked for both llama and alpaca 7B. 🥇

[loretoparisi]

Can you try this convert script? https://gist.github.com/eiz/828bddec6162a023114ce19146cb2b82 (it outputs .tmp files, you can uncomment the os.rename to do it in place if you want but I didn't want to overwrite your files)

I'm calling python3 .\convert.py .\models\7B\ .\models\tokenizer.model from the llama directory but it doesn't seem to do anything. Doesn't even produce errors. Am I using it wrong?

I did the same with no issue, tmp files were generated. For both 7B and 13B models it worked fine.

[tjohnman]

I can confirm this worked for 13B. Thank you!

I'm calling python3 .\convert.py .\models\7B\ .\models\tokenizer.model from the llama directory but it doesn't seem to do anything. Doesn't even produce errors. Am I using it wrong?

@loretoparisi Try python3 .\convert.py models\7B .\models\tokenizer.model

[PriNova]

If you don't have access to the original LLaMA files I think someone uploaded it here https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer.model

This works like charm. It would be amazing creating a PR for it.
It is a very import piece to have, if future changes will be made.

[Green-Sky]

@eiz can you make a pr with the script and name it something like convert_old-ggml_to_ggmlv1.py ?
maybe put it into a subfolder. eg whisper.cpp has a https://github.com/ggerganov/whisper.cpp/tree/master/extra extra folder with some scripts.

[Green-Sky]

if future changes will be made.

yes, there are discussions to change the fileformat again to make loading faster.

[loretoparisi]

@eiz It seems there is a problem with the alpaca 13B, after conversion, when loading it complains about the embedding size:

main: seed = 1679320340
llama_model_load: loading model from './alpaca-models/13B/ggml-alpaca-13b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: ggml ctx size = 8559.49 MB
llama_model_load: memory_size =   800.00 MB, n_mem = 20480
llama_model_load: loading model part 1/2 from './alpaca-models/13B/ggml-alpaca-13b-q4.bin'
llama_model_load: llama_model_load: tensor 'tok_embeddings.weight' has wrong size in model file
main: failed to load model from './alpaca-models/13B/ggml-alpaca-13b-q4.bin'

while the 7B works fine:

llama_model_load: loading model from './alpaca-models/7B/ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './alpaca-models/7B/ggml-alpaca-7b-q4.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

[Green-Sky]

@loretoparisi right now llama.cpp expects the 13b model to be 2 files. this is a known issue

[rabidcopy]

@eiz It seems there is a problem with the alpaca 13B, after conversion, when loading it complains about the embedding size:

main: seed = 1679320340
llama_model_load: loading model from './alpaca-models/13B/ggml-alpaca-13b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: ggml ctx size = 8559.49 MB
llama_model_load: memory_size =   800.00 MB, n_mem = 20480
llama_model_load: loading model part 1/2 from './alpaca-models/13B/ggml-alpaca-13b-q4.bin'
llama_model_load: llama_model_load: tensor 'tok_embeddings.weight' has wrong size in model file
main: failed to load model from './alpaca-models/13B/ggml-alpaca-13b-q4.bin'

That is due to llama.cpp expecting it to be in 2 parts. Alpaca.cpp hardcodes the n_parts to 1 to load the 13B weights. Change the 2 at https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#L37 to a 1 and re-make. This will break loading normal 2-part 13B llama weights however. Edit: Confirmed myself using eiz's very useful conversion script. Loads the same as before with that change in main.cpp.