This repository was archived by the owner on Jun 24, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Jun 24, 2024. It is now read-only.
Does it support the new GGMLv3 quantization methods? #286
Description
Tried using the cli application to see how far it had come from being llama-rs, and noticed that an error popped up using one of the newer WizardLM uncensored models using the GGMLv3 method,
llm llama chat --model-path .\Wizard-Vicuna-7B-Uncensored.ggmlv3.q5_1.bin
⣾ Loading model...Error:
0: Could not load model
1: invalid file format version 3
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Am I using it the wrong way or is it not supported yet?
Reacted by Felipe S. S. Schneider and Lee Jongwhan
Reacted by Felipe S. S. Schneider
Activity
philpax commentedon May 29, 2023
Hi there! Yes, it's supported, but only on the latest version (
main) - we haven't cut a new release yet. Hope to have that sorted soon!Exotik850 commentedon May 30, 2023
My apologies, should've tried the main branch instead of just trying the release 😅
philpax commentedon May 31, 2023
No worries - I'll keep this up for now and pin it for people's reference until we get it out the door :)
arctic-hen7 commentedon Aug 19, 2023
@philpax have you considered making some
0.2.0-beta.1etc. releases on crates.io? This pattern has worked very well for some of my own projects in the past.philpax commentedon Aug 21, 2023
Hi there! Yeah, I've considered it, but the main blocker is #221 - I don't want to cut a release where the interface is going to be radically different in the next release. I'm hoping to have this all closed out within the next week or two, especially with GGUF on the horizon, but I've been quite busy.