Repository navigation
High Initial Loss When Fine-Tuning Gemma Model #38
Description
Dear authors,
Thans for your great works!
I am currently trying to fine-tune the Gemma-7B model using PiSSA, but I am encountering an issue where the initial loss and grad norm are extremely high.
This doesn't seem to be cuased by the pissa algorithm, since using LoRA to fine-tune Gemma-7B also has similar problem.
Do you have encounted this question, or have any ideas on how to solve it? Thanks a lot!
Activity
Peng-YM commentedon Feb 11, 2025
+1, I have also encountered the same issue when fine-tuning the Qwen-2.5-7B model. The initial loss is approximately an order of magnitude higher than that of the standard LoRA method. For instance, while the standard LoRA achieves an initial loss of around 0.6, PiSSA exhibits an initial loss of approximately 6 or higher.
gapsong commentedon Jul 29, 2025
@zhangzc21 @Peng-YM did you find some futher answers? I am facing the same problem...
Peng-YM commentedon Sep 1, 2025
We ultimately couldn't resolve this issue and ended up abandoning the PiSSA fine-tuning approach
gapsong commentedon Sep 3, 2025
Me too. Even the answer of them did not help me at all.
I have used the same Hyperparameters as they recommend, but still not working out.
@Peng-YM
fxmeng commentedon Sep 4, 2025
Hello, we trained Qwen2.5-7B and Gemma-7B using PiSSA with the default hyperparameters, and we did not observe the phenomenon of higher loss. Attached are the training logs.
There may be two possible reasons for the issue you encountered:
1. When performing the PiSSA initialize, you used lora_alpha == lora_r, but during training, you used lora_alpha == 2 * lora_r. In this case,
W != 2 * U[:r] S[:r]V[:r] + U[r:]S[r:]V[r:].
2. Environment-related issues.
If you could share your environment file and the training script, we can further analyze the possible cause of this problem.
trainer_state_metamath_pissa_gemma_7b.json
trainer_state_metamath_pissa_qwen2.5_7b.json