Repository navigation
(possible) Misc. bug: convert_lora_to_gguf.py ignores --outtype #15890
Description
Name and Version
Tested on latest pull.
Problem description & steps to reproduce
I'm not sure if this is a bug or intended:
- Is
--outtypesupposed to convert to the type, eg:--outtype f16on af32LoRA adapter in PEFT format is supposed to create anf16LoRA in GGUF format (ie: the same command semantics asconvert_hf_to_gguf_update.py)? - or is it just a "type hint" incase it can't infer the type of the PEFT format LoRA?
I also tried running llama-quant on the resulting LoRA in f32 GGUF format, but it doesn't like being given these files...
Are non-f32 LoRAs even allowed?
If it is a bug then I'll do some more investigation and see if I can find out what is happening.
Activity
CISC commentedon Sep 9, 2025
It's because the lora tensor names don't end in
.weight:llama.cpp/convert_hf_to_gguf.py
Lines 337 to 339 in 233d773
Arguably this can be said to be intentional as you generally do not want to deteriorate LoRA weights (and they are usually fairly small).
jukofyork commentedon Sep 12, 2025
Thanks.
Yeah, I'm not sure if only F32 makes sense, but for very large rank LoRAs (256+) the files can be quite substantial.
I'll close this now as agree it's not clearly a bug and just a little confusing why it ignores the setting.
CISC commentedon Sep 12, 2025
I suppose it would make sense to use F16 if the original weights were that, but I doubt there are too many of those? Most are BF16 nowadays, and that's not a good idea to use as it will generally perform badly on a lot of hardware.
Allowing Q8 is probably perfectly viable and can be added as a special case for lora weights in
prepare_tensorsif you wish to make a PR?jukofyork commentedon Sep 12, 2025
I'll do some experiments over the next couple of weeks to see how useful it is first.