Repository navigation
QaLora #986
Description
Feature request
This came out
https://github.com/yuhuixu1993/qa-lora/blob/main/qalora.py
would like to use it with transformers
btw, I also got mezo integrated with transformers. Maybe we can swap notes to get that integrated.
Motivation
more efficient
Your contribution
I'll test, contribute code if necessary
Reacted by Hyeonseo Yang
Reacted by Allan
Activity
BenjaminBossan commentedon Oct 2, 2023
Yes, this is something we surely want to integrate into PEFT. The linked repo already uses PEFT, so hopefully this should not be too difficult to add. Let us know if you want to work on adding the feature.
Also, feel free to share your notes on MeZO, though I assume this is for the transformers
Trainer, so please share them with the transformers folks.github-actions commentedon Nov 1, 2023
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
thistleknot commentedon Nov 1, 2023
mezo:
https://github.com/thistleknot/TrainLLMv3/blob/main/functions.py#L1284
transformers suggested I submit mezo here
huggingface/transformers#24264
github-actions commentedon Nov 27, 2023
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
datavistics commentedon Dec 8, 2023
Id definitely be interested in this. It seems just as promising as LoftQ if not better.
github-actions commentedon Jan 1, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
gapsong commentedon Mar 17, 2025
Does someone have an update on that one?
gapsong commentedon Jun 4, 2025
Here is the PR for QA-Lora with GPTQ:
#2571
thistleknot commentedon Jun 4, 2025
thistleknot commentedon Jul 3, 2025
thank you!
now training with qa-lora =D