Hi,
The engine of ZeroQuant inference is not released yet. The code example in DeepSpeed-Example is only to help verify the accuracy of ZeroQuant.
The kernel/engine released is on our calendar and we are actively working on it to make it compatible for various models. Please stay tuned.
For LKD, we will also release it soon.
For the last question, the code for training or accuracy testing is different than the final inference engine. Here, everything is simulated, so we can do quantization aware training or other things
Originally posted by @yaozhewei in #2207 (comment)
hi ，when the ZeroQuant inference (for GPT model) will be released?
Hi,
The engine of ZeroQuant inference is not released yet. The code example in DeepSpeed-Example is only to help verify the accuracy of ZeroQuant.
The kernel/engine released is on our calendar and we are actively working on it to make it compatible for various models. Please stay tuned.
For LKD, we will also release it soon.
For the last question, the code for training or accuracy testing is different than the final inference engine. Here, everything is simulated, so we can do quantization aware training or other things
Originally posted by @yaozhewei in #2207 (comment)
hi ，when the ZeroQuant inference (for GPT model) will be released?