GPT-NeoX has only minimal inference support

Steps to reproduce:

Download https://huggingface.co/EleutherAI/gpt-neox-20b
Convert the model and attempt to use it:

$ TMPDIR=/var/tmp ./convert-gptneox-hf-to-gguf.py gpt-neox-20b 1 --outfile gpt-neox-20b.f16.gguf
$ ./main -m gpt-neox-20b.f16.gguf
<snip>
llama_model_loader: - type  f32:  354 tensors
llama_model_loader: - type  f16:  178 tensors
error loading model: cannot find tokenizer scores in model file

llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'gpt-neox-20b.f16.gguf'
main: error: unable to load model

Even if you add dummy scores and token types in the conversion script, it fails here:
https://github.com/ggerganov/llama.cpp/blob/bc9d3e3971e5607a10ff4c24e39568ce1ac87271/llama.cpp#L2288
Was GPT-NeoX ever even implemented in GGUF?

Was GPT-NeoX ever even implemented in GGUF?

Yes, example inference code exists here: https://github.com/ggerganov/llama.cpp/blob/master/examples/gptneox-wip/gptneox-main.cpp

Oh, it has a separate implementation. So I can't currently use it with any third-party software that uses the llama.cpp API.

edit: This file is not listed in either of the build scripts. It doesn't seem to have GPU acceleration. It seems like that could be improved.

Yeah, there's just a poc implementation. We should add it in llama.cpp eventually

The situation is now that we do have code in this repository to "successfully" convert and quantize a gpt-neo-x model, but no way to run these models.
https://github.com/ggerganov/ggml/tree/master/examples/gpt-neox does have its own convert-script. The conversion here in the convert-hf-to-gguf.py does not seem to have any purpose at all.

I still would like to bring this forward again:
There is code inside the convert-hf-to-gguf.py since its first release in #3838 and before in the seperate convert-gptneox-script to somehow "sucessfully" convert gpt-neo-x models into gguf models. But there is no code whatsoever to run inference on a model that the convert script labels as a gptneox model.

I believe you can run this using https://github.com/ggerganov/ggml/blob/master/examples/gpt-neox/main.cpp
it's just not in llama.cpp. It'll be supported eventually.

Oh, I alredy compiled that example for testing. It seems to expect the old ggml .bin files, which can be created using the convert program in the same example directory. It doesn't run the gguf files that are built using the convet-hf-to-gguf.pyi script. Right now, there is no code that can run these gguf files converted from gpt-neo-x models.

It's much easier to add new arches to llama.cpp now (I hope) - PRs welcome

This issue was closed because it has been inactive for 14 days since being marked as stale.

I'm still interested in this.

This issue was closed because it has been inactive for 14 days since being marked as stale.

Navigation Menu

Repository navigation

GPT-NeoX has only minimal inference support #3293

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

GPT-NeoX has only minimal inference support #3293

Description

Activity

cebtenzzre commented on Sep 21, 2023

jackie1218 commented on Sep 22, 2023

cebtenzzre commented on Sep 22, 2023

ggerganov commented on Sep 28, 2023

maddes8cht commented on Nov 12, 2023

maddes8cht commented on Nov 22, 2023

Galunid commented on Nov 22, 2023

maddes8cht commented on Nov 22, 2023

ggerganov commented on Nov 23, 2023

github-actions commented on Apr 3, 2024

cebtenzzre commented on Apr 3, 2024

github-actions commented on May 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions