Repository navigation
QLora broken - bnb.py resolve_variant should accept QALoRA params for compatibility with LoraLayer #2671
Description
System Info
python: 3.11.13 (main, Jun 5 2025, 13:12:00) [GCC 11.2.0]
Platform: Linux-6.14.0-24-generic-x86_64-with-glibc2.39
peft: 0.15.2.dev0
trl: 0.19.1
accelerate: 1.9.0
transformers: 4.53.2
Who can help?
Reproduction
When running with the following QLoRA training setup:
print("🔧 Setting up QLoRA training...")
# Configure 4-bit quantization for QLoRA
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load model with 4-bit quantization
model = transformers.AutoModelForCausalLM.from_pretrained(
script_args.model_name_or_path,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
# Prepare model for k-bit training (essential for QLoRA)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)
# Configure LoRA for QLoRA
lora_config = LoraConfig(
task_type="CAUSAL_LM",
r=4
lora_alpha=8,
target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05, # Slightly higher dropout for QLoRA
bias="none",
init_lora_weights=True,
)
model = get_peft_model(model, lora_config)The bnb layer inherits from LoraLayer, and LoraLayer calls self.resolve_lora_variant with the following arguments:
lora_variant = self.resolve_lora_variant(
use_dora=use_dora, use_qalora=use_qalora, qalora_group_size=qalora_group_size
)However, resolve_variant in bnb.py does not accept use_qalora and qalora_group_size, causing a break.
Expected behavior
The resolve_variant function in bnb.py should accept use_qalora and qalora_group_size as arguments for compatibility with LoraLayer, but safely ignore them internally if not needed, preventing errors when called via a LoraLayer interface.
Activity
BenjaminBossan commentedon Jul 25, 2025
I think this is because you're using an outdated from source PEFT install before #2618 was merged, is that possible? At least on the latest PEFT, I can't reproduce the error. This is what I used:
gapsong commentedon Jul 25, 2025
Ohhh true. Sorry for wasting your time. Yes that fixed the problem.
BenjaminBossan commentedon Jul 25, 2025
No problem, it happens :)