Add support for Quantization-Aware Low-Rank Adaptation (QALoRA) by gapsong · Pull Request #2571 · huggingface/peft

gapsong · 2025-06-04T12:41:50Z

This pull request introduces QALoRA (Quantization-Aware Low-Rank Adaptation), a new fine-tuning technique for quantized large language models, along with its implementation in the PEFT library. The changes include updates to documentation, configuration, and core logic to support QALoRA's memory-efficient and performance-preserving features.

Documentation Updates

examples/qalora_finetuning/README.md: Added detailed documentation for QALoRA, including its introduction, implementation details, usage examples, command-line instructions, and comparison with other techniques like LoRA and DoRA.

Configuration Enhancements

src/peft/tuners/lora/config.py: Introduced two new configuration parameters: use_qalora to enable QALoRA and qalora_group_size to control the pooling group size for memory-performance tradeoffs.

Core Logic for QALoRA

src/peft/tuners/lora/gptq.py: Updated the GPTQ LoRA implementation to support QALoRA, including logic for resolving QALoRA variants and passing group size parameters. [1] [2] [3]
src/peft/tuners/lora/layer.py: Enhanced the layer update logic to initialize QALoRA-specific parameters and handle adapter-specific configurations. [1] [2]
src/peft/tuners/lora/model.py: Incorporated QALoRA-specific parameters into the model creation and replacement process.

QALoRA Variant Implementation

src/peft/tuners/lora/variants.py: Added the QALoraLinearVariant class, implementing QALoRA-specific logic for initialization, delta weight computation, merging, unmerging, and forward propagation. This includes pooling input features and scaling them for efficient adaptation.

…raLayer and GPTQLoraLinear

BenjaminBossan

Thanks a lot for picking up this old request. Really well done to use the newly added LoRA variant abstraction to implement this.

I checked the PR but haven't done an in-depth review yet. The reason for that is that LoRA variant support has only been added to vanilla LoRA layers (i.e. the layers defined in lora/layers.py). The quantized layers, including GPTQ, don't have any code that would take LoRA variants into account. Therefore, as is, the GPTQ layer would still use the normal forward call and not QALoraLinearVariant.forward. Even worse, the GPTQ layer does not support merging and unmerging, so all of that code QALoraLinearVariant is dead code. So unless I'm missing something, there is still some work required:

Update GPTQLoraLinear.forward to account for LoRA variants (should be easy).
Implement merging and unmerging for GPTQLoraLinear (could be difficult, it depends), or scrap it for now.

To avoid 2., QA LoRA could be implemented for another quantization method that already supports merging and unmerging, like bitsandbytes, but even there, LoRA variant support has yet to be added. Also, I'm not sure how specific your code is to GPTQ.

Anyway, it's a really nice PR and I'd be happy to see it merged. LMK what you think.

… unsupported operations.

gapsong · 2025-06-09T10:57:28Z

Hi @BenjaminBossan ,

Thank you for your review and the helpful feedback!

I've addressed the main points you raised:

I have scraped the merge/unmerge logic completely from QALoraLinearVariant for the GPTQLoraLinear context, as you correctly pointed out it's not supported there and was effectively dead code.
Thank you especially for pointing out that the QALoRA variant's forward function was never actually being called within GPTQLoraLinear. This was a critical oversight, and I've now updated GPTQLoraLinear.forward to correctly dispatch to the QALoRA variant's forward pass when QALoRA is active.

Ready for another look when you have a moment!

BenjaminBossan

Thanks a lot for the updates, I did another review, where I focused on the PEFT integration itself (not checking the example or the details of the QALoRA paper). There are still a few issues but they're not big, please take a look.

Once these issues are resolved, I'll review the example. We should also think about adding a test before merging.

BenjaminBossan · 2025-06-11T10:29:08Z

+        default=False,
+        metadata={
+            "help": (
+                "Enable <a href='https://huggingface.co/papers/2309.14717'>Quantization-Aware Low-Rank Adaptation (QALoRA)</a>. This technique combines quantization-aware training "


Let's mention that it is only implemented for GPTQ for now. Also, please update the docstring (can use the same text).

BenjaminBossan · 2025-06-11T10:29:38Z

+                "Enable <a href='https://huggingface.co/papers/2309.14717'>Quantization-Aware Low-Rank Adaptation (QALoRA)</a>. This technique combines quantization-aware training "
+                "with LoRA to improve performance for quantized models. This can improve the performance of LoRA, "
+                "especially at low ranks. Right now, QALoRA only supports linear layers. QALoRA introduces a bigger "
+                "overhead than pure LoRA, so it is recommended to merge weights for inference."


This recommendation is a bit moot as merging is not supported for GPTQ. Let's remove this sentence.

BenjaminBossan · 2025-06-11T10:32:20Z

+        if use_qalora:
+            from .variants import QALoraLinearVariant
+
+            return QALoraLinearVariant()
+        if not use_dora:
+            return None
+
+        from .variants import DoraLinearVariant
+
+        return DoraLinearVariant()


Let's change the check a bit for completeness to basically:

if use_dora and use_qalora: NotImplementedError elif use_dora: variant = ... elif use_qalora: variant = ... else: variant = None return variant

BenjaminBossan · 2025-06-11T10:32:46Z

            return result

        lora_A_keys = self.lora_A.keys()
+        torch_result_dtype = result.dtype


This is not needed, right?

BenjaminBossan · 2025-06-11T10:33:25Z

-
-            if requires_conversion:
-                output = output.to(expected_dtype)
+            # requires_conversion = not torch.is_autocast_enabled()


remove the comment?

BenjaminBossan · 2025-06-11T10:39:44Z

+
+        # Create and store pooling factor for scaling
+        if not hasattr(module, "qalora_scaling_factor"):
+            module.qalora_scaling_factor = {}


Same comment as above regarding other_param_names. But do we even need qalora_scaling_factor, as it can be calculated on the fly based on module.in_features and qalora_group_size anyway?

BenjaminBossan · 2025-06-11T10:41:47Z

+            module.qalora_scaling_factor[adapter_name] = module.in_features / qalora_group_size
+        else:
+            # No special scaling if dimensions don't align
+            module.qalora_scaling_factor[adapter_name] = 1.0


Would this not lead to very different results than qalora_scaling_factor = module.in_features / qalora_group_size?

I wonder if perhaps it makes more sense to raise an error here and require that module.in_features % qalora_group_size == 0? I think this would simplify the code and when module.in_features % qalora_group_size != 0 we just get standard LoRA?

BenjaminBossan · 2025-06-11T10:42:51Z

+            torch.Tensor: The calculated delta weight.
+        """
+        if (
+            not hasattr(module, "qalora_group_size")


Is it even possible to hit this condition? Same question about not hasattr(module, "qalora_scaling_factor").

You are right. We will never reach in here

…gement in variants

gapsong · 2025-06-12T06:26:11Z

@BenjaminBossan I have integrated all your feedback into the code, except for the tests.
I'm currently working on those. I will structure the tests similarly to the dora logic and integrate them in the same way.

BenjaminBossan

Thanks a lot for all the updates. I still found a couple of issues, please check my comments. I also checked and ran the script, where I also found some areas for improvement.

As for testing, I think it would be enough to have something very similar to this one with QALoRA enabled.

Also, before pushing, don't forget to run make style to make the linter happy.

BenjaminBossan · 2025-06-12T12:31:04Z

+    --qalora_group_size 8
+```
+
+QALoRA also works with different quantization methods (GPTQ, EETQ, AWQ, etc.):


Currently, that's not true, is it?

Question still open

BenjaminBossan · 2025-06-12T12:32:38Z

+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## QALoRA vs. LoRA vs. DoRA


I don't think there is a specific reason to compare QALoRA to DoRA, is there? I think the more interesting question for users would be: What's the difference between QALoRA and QLoRA? When should I use which, what trade offs are there?

BenjaminBossan · 2025-06-12T12:34:46Z

+2. The QALoRA adapter weights are then merged with the dequantized model
+3. The merged model must be re-quantized if quantization is still desired
+
+This implementation choice was made because **it yielded better performance in practice**, despite being less memory-efficient than the direct modification approach described in the paper.


Out of curiosity: Is this your personal experience or is there a reference for this?

Question still open

BenjaminBossan · 2025-06-12T13:33:31Z

+            elif optimizer_name.lower() == "sgd":
+                opt_size = param_size  # SGD with momentum keeps 1 extra state
+
+        print(


Again, I think printing this for each layer is a bit of information overkill.

BenjaminBossan · 2025-06-12T13:36:16Z

+
+Run the finetuning script with a GPTQ quantized model:
+```bash
+python examples/qalora_finetuning/qalora_gptq_finetuning.py \


When I run this locally, I get a train loss of 0.0, can you replicate?

Question still open

BenjaminBossan · 2025-06-12T13:41:20Z

+        task_type="CAUSAL_LM",
+        use_dora=use_dora,
+        use_qalora=use_qalora,
+        qalora_group_size=8,  # Explicitly set group size for QALoRA


Would it make sense to expose this argument to the CLI?

BenjaminBossan · 2025-06-12T13:42:17Z


+    def resolve_lora_variant(self, *, use_dora: bool, use_qalora: bool, **kwargs) -> Optional[LoraVariant]:
+        if use_dora and use_qalora:
+            NotImplementedError


raise NotImplementedError(...)

BenjaminBossan · 2025-06-12T13:45:02Z

+            from .variants import DoraLinearVariant
+            variant = DoraLinearVariant()
+        elif use_qalora:
+            if self.in_features % kwargs["qalora_group_size"] == 0:


Let's not perform the check here. Instead, let's move it inside QALoraLinearVariant.init.

This check can now be removed, right?

…and refactor training tests - Added detailed error message for unsupported simultaneous use of Dora and QA_LoRA in GPTQLoraLinear. - Refactored QALoraLinearVariant to streamline pooling and scaling operations, improving clarity and performance. - Consolidated multiple training test cases for VeRA and RandLora into a more organized structure, ensuring consistency across single and multi-GPU tests. - Updated training configurations to include new parameters for QALoRA and improved handling of token embeddings. - Ensured that model checkpoints are correctly validated and that training loss assertions are in place for all tests.

gapsong · 2025-06-14T17:23:48Z

More tests are still needed for comprehensive coverage and some of the existing comments are a work in progress and will be refined.

BenjaminBossan · 2025-06-16T09:57:14Z

More tests are still needed for comprehensive coverage and some of the existing comments are a work in progress and will be refined.

Okay, ich warte dann, bis diese fertig sind bevor ich erneut reviewe ;)

…ction, enhancing gradient checks in tests, and improving adapter handling in model training.

gapsong · 2025-06-16T21:16:04Z

@BenjaminBossan du kannst wieder reinschauen.

Ich habe die Initialisierung nochmal überarbeitet. Jetzt wird lora_A direkt beim Init durch eine neue Instanz mit der gepoolten Eingabedimension ersetzt. Vorher hatten wir lora_A noch in der alten Shape und haben das Pooling erst im Forward angewendet, so wie es auch ursprünglich im Paper gemacht wurde.

Mit der neuen Variante übernimmt lora_A das Pooling bereits beim Init, wodurch die Dimensionen und damit die Anzahl der trainierbaren Parameter bei QALoRA tatsächlich kleiner werden als bei QLoRA. Vorher waren die "trainable params" bei QLoRA und QALoRA identisch, jetzt ist QALoRA sparsamer.

Eine interessante Beobachtung: Der Speicherverbrauch ist trotzdem sehr ähnlich zu QLoRA, obwohl weniger trainierbare Parameter vorhanden sind. Das scheint aber durch die zusätzlichen Pooling-Tensoren im Forward zustande zu kommen. Vielleicht kannst du mir nochmal bestätigen, ob das so korrekt und zu erwarten ist.

BenjaminBossan

Let's switch back to English.

Thanks you for the update, they look good overall. I noticed that some of my comments seem yet to be unaddressed, could you please double-check? Also, something seems to have gone wrong when adding the GPU tests, let's mostly revert those changes and only add the new test.

ch habe die Initialisierung nochmal überarbeitet. Jetzt wird lora_A direkt beim Init durch eine neue Instanz mit der gepoolten Eingabedimension ersetzt. Vorher hatten wir lora_A noch in der alten Shape und haben das Pooling erst im Forward angewendet, so wie es auch ursprünglich im Paper gemacht wurde.

Mit der neuen Variante übernimmt lora_A das Pooling bereits beim Init, wodurch die Dimensionen und damit die Anzahl der trainierbaren Parameter bei QALoRA tatsächlich kleiner werden als bei QLoRA. Vorher waren die "trainable params" bei QLoRA und QALoRA identisch, jetzt ist QALoRA sparsamer.

Nice optimization.

Eine interessante Beobachtung: Der Speicherverbrauch ist trotzdem sehr ähnlich zu QLoRA, obwohl weniger trainierbare Parameter vorhanden sind. Das scheint aber durch die zusätzlichen Pooling-Tensoren im Forward zustande zu kommen. Vielleicht kannst du mir nochmal bestätigen, ob das so korrekt und zu erwarten ist.

Hmm, this is very hard to say without further details. How did you test this and how large was the difference? In general, for big base models, the majority of memory will be consumed by the base weights and the amount of memory used by LoRA is relatively small. Quantization helps of course but the general pattern still holds. Therefore, if LoRA is made a little bit more memory efficient, the total effect may still be negligible. And as you mentioned, depending on the dataset (mainly the sequence length), activations/hidden states can take up a large portion of memory and those are also mostly unaffected by the LoRA parameter count.

BenjaminBossan · 2025-06-17T14:12:15Z

+
+Run the finetuning script with a GPTQ quantized model:
+```bash
+python examples/qalora_finetuning/qalora_gptq_finetuning.py \


Question still open

BenjaminBossan · 2025-06-17T14:12:19Z

+    --qalora_group_size 8
+```
+
+QALoRA also works with different quantization methods (GPTQ, EETQ, AWQ, etc.):


Question still open

BenjaminBossan · 2025-06-17T14:12:33Z

+2. The QALoRA adapter weights are then merged with the dequantized model
+3. The merged model must be re-quantized if quantization is still desired
+
+This implementation choice was made because **it yielded better performance in practice**, despite being less memory-efficient than the direct modification approach described in the paper.


Question still open

BenjaminBossan · 2025-06-17T14:14:11Z

+        default=False,
+        metadata={
+            "help": (
+                "It is only implemented in GPTQ for now. Enable <a href='https://huggingface.co/papers/2309.14717'>Quantization-Aware Low-Rank Adaptation (QALoRA)</a>. This technique combines quantization-aware training "


Let's ensure that each line is 120 chars max.

BenjaminBossan · 2025-06-17T14:14:47Z

+            from .variants import DoraLinearVariant
+            variant = DoraLinearVariant()
+        elif use_qalora:
+            if self.in_features % kwargs["qalora_group_size"] == 0:


BenjaminBossan · 2025-06-17T14:16:13Z

-
-            # assert loss is not None
-            assert trainer.state.log_history[-1]["train_loss"] is not None
+    # In der PeftGPTQGPUTests Klasse hinzufügen:


BenjaminBossan · 2025-06-17T14:17:46Z

        assert not weights_not_cpu

-    @pytest.mark.single_gpu_tests
-    def test_causal_lm_training_vera(self):


Hmm, it looks like a bunch of unrelated tests where moved around. I don't think that should be necessary and it also makes reviewing a lot harder. Could you please ensure that the existing tests remain in place and only new tests for GPTQ-QALoRA are added?

…ion and structure for better readability and maintainability.

BenjaminBossan · 2025-06-18T14:06:08Z

@gapsong Please ping me once the PR is ready for the next review.

… tests

…y breaking long lines, update dataset loading to use provided data path, and remove unused arguments.

…cessary parameters for clarity and conciseness.

…ter, update dataset loading, and remove unused parameters for improved clarity and functionality.

gapsong · 2025-06-22T12:48:21Z

@BenjaminBossan I have adjusted the code expect for the resolve_lora_variant
Could you convince me that if self.in_features % kwargs["qalora_group_size"] == 0:
belongs into the init()?

BenjaminBossan · 2025-06-23T09:18:23Z

Thanks for the updates. I tried the example script again for 5 steps, using TheBloke/Llama-2-7B-GPTQ, and found that the train_loss is 0.0. Can you confirm that? It doesn't matter if use_qalora is enabled or not.

Could you convince me that if self.in_features % kwargs["qalora_group_size"] == 0: belongs into the init()?

For, mostly for these reasons:

These types of checks are typically done during init methods in PEFT, so it is consistent.
It is a QALoRA related piece of code, thus it makes sense to put in the QALoRA specific code.
resolve_lora_variant should focus on what the name says, deciding which LoRA variant is used.

…2 in README and add validation for divisibility by group size in QALoraLinearVariant.

gapsong · 2025-06-23T13:15:56Z

@BenjaminBossan I’m done and I found the “bug”:
The qalora_group_size was set to 8, which was too small. I’ve changed the default value to 32.

I also adjusted the resolve_lora_variant function:
Training with "lora" is no longer the default if the dimensions do not fit—specifically, when module.in_features % kwargs["qalora_group_size"] != 0. Instead, an error is now thrown if the parameters do not match.

I hope this time works out :)

BenjaminBossan

Thanks for the updates, not much is missing for the PR.

The qalora_group_size was set to 8, which was too small. I’ve changed the default value to 32.

Good find. Do you know why that resulted in a loss of 0? I wonder if there is some kind of check (say, some hidden states not containing nan) that could be performed to prevent this in the future. Depending on how train logging is set up, users may otherwise waste hours of training time.

BenjaminBossan · 2025-06-23T14:42:26Z

+    --learning_rate 3e-4 \
+    --cutoff_len 512 \
+    --use_qalora \
+    --qalora_group_size 32 \


Thanks for updating to a better default. Do you think that qalora_group_size=16 is still a good choice for the default value in LoraConfig?

BenjaminBossan · 2025-06-23T14:43:31Z

+            from .variants import DoraLinearVariant
+            variant = DoraLinearVariant()
+        elif use_qalora:
+            if self.in_features % kwargs["qalora_group_size"] == 0:


This check can now be removed, right?

BenjaminBossan · 2025-06-23T14:45:43Z

+        """
+        if "qalora_group_size" not in kwargs:
+            raise ValueError(
+                "QALoraLinearVariant.init expects 'qalora_group_size' to be provided in kwargs."


Suggested change

"QALoraLinearVariant.init expects 'qalora_group_size' to be provided in kwargs."

"`use_qalora=True` requires 'qalora_group_size' to be provided in kwargs."

BenjaminBossan · 2025-06-23T14:46:09Z

+
+        if module.in_features is not None and module.in_features % kwargs["qalora_group_size"] != 0:
+            raise ValueError(
+                f"QALoraLinearVariant.init expects module.in_features ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"


Suggested change

f"QALoraLinearVariant.init expects module.in_features ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"

f"`use_qalora=True` requires `module.in_features` ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"

…cy; streamline argument handling in training script.

gapsong · 2025-06-23T15:52:08Z

@BenjaminBossan included your suggestions!
Thank you

gapsong · 2025-06-24T11:52:26Z

@BenjaminBossan something seems off with the GPU memory, when using QA_Lora I am investigating the problem at the moment.

…ess, improve pooling logic, and enhance LoRA computation for clarity and efficiency. Memory peaks were really high

gapsong · 2025-06-24T13:35:34Z

@BenjaminBossan GPU problem was resolved. I removed torch.nn.functional.avg_pool1d since the peak memory needed for the unsqueezed layers were to high

BenjaminBossan

Thanks for the latest update. I just have a small comment, otherwise the PR LGTM.

I also ran the example locally with and without QALoRA and the difference in memory usage was negligible, which I think is what we hoped to see.

BenjaminBossan · 2025-06-24T15:24:03Z

            "base_model.model.lin1.lora_B.adapter1.weight",
        )

+    def test_requires_grad_qalora_same_targets(self):


Hmm, for some reason I haven't really noticed this test. IMO it is not necessary and can be removed. This is because QALoRA does not change the handling of multiple adapters, so there is not really a reason to believe there could be anything wrong there. We also don't check other quantization methods here. Unless you had a specific reason to add this test, I'd suggest to just remove it.

…proved clarity and maintainability.

HuggingFaceDocBuilderDev · 2025-06-24T16:10:47Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan

I did a final pass and discovered two small things that need fixing, otherwise LGTM. Thanks for your patience.

BenjaminBossan · 2025-06-25T08:57:40Z

+
+        if module.in_features is not None and module.in_features % kwargs["qalora_group_size"] != 0:
+            raise ValueError(
+                f"`use_qalora=True` requires `module.in_features` ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"


Could you please add line breaks to this long line?

BenjaminBossan · 2025-06-25T08:58:12Z

        return result


+class QALoraLinearVariant(LoraVariant):


Please move this class to the bottom of the file, as is it sits between different DoRA classes.

gapsong · 2025-06-25T09:56:11Z

@BenjaminBossan what do I have to do here to resolve the errors or is it something out of my scope?

BenjaminBossan · 2025-06-25T10:22:53Z

what do I have to do here to resolve the errors or is it something out of my scope?

You mean the MacOS errors? Yeah, they're unrelated and should be fixed with the next transformers release.

gapsong · 2025-06-25T12:30:49Z

@BenjaminBossan shall we merge it then?

BenjaminBossan · 2025-06-25T13:08:42Z

shall we merge it then?

Did you see my last comment?

gapsong · 2025-06-26T06:19:53Z

shall we merge it then?

Did you see my last comment?

I have missed it. It should be ready now @BenjaminBossan

BenjaminBossan

Thanks for contributing QALoRA support with GPTQ to PEFT. The PR LGTM, nicely done using LoRA variants for this. Failing tests are unrelated.

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

The function signature was missing **kwargs, which results in a failure after merging #2571.

thistleknot · 2025-07-03T16:24:15Z

thank you, now training with qa-lora =D

Support for Quantization-Aware Low-Rank Adaptation (QALoRA) for GPTQ.

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

Support for Quantization-Aware Low-Rank Adaptation (QALoRA) for GPTQ.

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

Add support for Quantization-Aware Low-Rank Adaptation (QALoRA) in Lo…

edc88cb

…raLayer and GPTQLoraLinear

gapsong mentioned this pull request Jun 4, 2025

QaLora #986

Closed

BenjaminBossan reviewed Jun 5, 2025

View reviewed changes

gapsong added 2 commits June 9, 2025 12:54

Refactor QALoraLinearVariant methods to raise NotImplementedError for…

c5f4bf9

… unsupported operations.

Fix dtype handling and use foward in GPTQLoraLinear forward method

af27857

BenjaminBossan requested changes Jun 11, 2025

View reviewed changes

Enhance QALoRA support by refining group size handling and error mana…

6ebf4e5

…gement in variants

BenjaminBossan requested changes Jun 12, 2025

View reviewed changes

Refactor QALoRA implementation by removing unused memory printing fun…

cad05f5

…ction, enhancing gradient checks in tests, and improving adapter handling in model training.

BenjaminBossan requested changes Jun 17, 2025

View reviewed changes

gapsong added 2 commits June 17, 2025 21:11

Refactor test for QALoRA with GPTQ quantization by improving indentat…

45c0c65

…ion and structure for better readability and maintainability.

Changed structure

4f8d336

gapsong added 4 commits June 21, 2025 11:38

Update README and config for QALoRA clarity; remove redundant line in…

1d76b08

… tests

Refactor README and training script for QALoRA: improve readability b…

75b0c56

…y breaking long lines, update dataset loading to use provided data path, and remove unused arguments.

Refactor README for QALoRA: remove outdated command examples and unne…

170eca8

…cessary parameters for clarity and conciseness.

Refactor train_model function in QALoRA script: add data_split parame…

268259b

…ter, update dataset loading, and remove unused parameters for improved clarity and functionality.

Update QALoRA examples and validation: change default group size to 3…

6cdb5cd

…2 in README and add validation for divisibility by group size in QALoraLinearVariant.

BenjaminBossan requested changes Jun 23, 2025

View reviewed changes

Refactor QALoraLinearVariant error messages for clarity and consisten…

8d4f9dd

…cy; streamline argument handling in training script.

Refactor forward method in QALoraLinearVariant: streamline weight acc…

2fd3f77

…ess, improve pooling logic, and enhance LoRA computation for clarity and efficiency. Memory peaks were really high

BenjaminBossan reviewed Jun 24, 2025

View reviewed changes

Remove redundant QALoRA gradient tests from RequiresGradTester for im…

8a08a10

…proved clarity and maintainability.

BenjaminBossan requested changes Jun 25, 2025

View reviewed changes

Moved QALoraLinearVariant to the bottom of variants.py

8d85f75

BenjaminBossan approved these changes Jun 26, 2025

View reviewed changes

BenjaminBossan merged commit e34852f into huggingface:main Jun 26, 2025

BenjaminBossan added a commit to BenjaminBossan/peft that referenced this pull request Jun 27, 2025

FIX Update signature for resolve_lora_variant

85df7b8

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

BenjaminBossan mentioned this pull request Jun 27, 2025

FIX Update signature for resolve_lora_variant #2618

Merged

BenjaminBossan added a commit that referenced this pull request Jun 27, 2025

FIX Update signature for resolve_lora_variant (#2618)

4562926

The function signature was missing **kwargs, which results in a failure after merging #2571.

thistleknot mentioned this pull request Jul 7, 2025

Qa-Lora integration axolotl-ai-cloud/axolotl#2878

Open

3 of 5 tasks

efraimdahl pushed a commit to efraimdahl/peft that referenced this pull request Jul 12, 2025

ENH Support Quantization-Aware LoRA with GPTQ (huggingface#2571)

e936202

Support for Quantization-Aware Low-Rank Adaptation (QALoRA) for GPTQ.

efraimdahl pushed a commit to efraimdahl/peft that referenced this pull request Jul 12, 2025

FIX Update signature for resolve_lora_variant (huggingface#2618)

f8fbc05

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

ved1beta mentioned this pull request Aug 4, 2025

qa-lora integration axolotl-ai-cloud/axolotl#3013

Open

iambogeumkim pushed a commit to iambogeumkim/peft that referenced this pull request Jun 21, 2026

ENH Support Quantization-Aware LoRA with GPTQ (huggingface#2571)

ea236d5

Support for Quantization-Aware Low-Rank Adaptation (QALoRA) for GPTQ.

iambogeumkim pushed a commit to iambogeumkim/peft that referenced this pull request Jun 21, 2026

FIX Update signature for resolve_lora_variant (huggingface#2618)

cc6240d

The function signature was missing **kwargs, which results in a failure after merging huggingface#2571.

	"QALoraLinearVariant.init expects 'qalora_group_size' to be provided in kwargs."
	"`use_qalora=True` requires 'qalora_group_size' to be provided in kwargs."

	f"QALoraLinearVariant.init expects module.in_features ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"
	f"`use_qalora=True` requires `module.in_features` ({module.in_features}) to be divisible by 'qalora_group_size' ({kwargs['qalora_group_size']})"

Conversation

gapsong commented Jun 4, 2025

Documentation Updates

Configuration Enhancements

Core Logic for QALoRA

QALoRA Variant Implementation

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

gapsong commented Jun 9, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gapsong commented Jun 12, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gapsong commented Jun 14, 2025

Uh oh!

BenjaminBossan commented Jun 16, 2025

Uh oh!

gapsong commented Jun 16, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!