ZeRO3 vs ZeRO2 Memory Comparison #146

Activity

1. SystemErrorWang commented on Jul 15, 2024

   

   SystemErrorWang

   on Jul 15, 2024

   More actions

   Hi, Thanks for your great work. Have you compare the gpu memory usegae of difference partition strategy, e.g, ZeRO2 and ZeRO3

   Hello，my training always got stuck when using deepspeed, have you succeeded in training with deepspeed? would you please share the code/command/experiences?