hey @RezaYazdaniAminabadi I have been waiting for this PR for a long time and recently tried out your branch for ZeroQuant (with GPT-J). I found a couple of issues: `quantizer` [here](https://github.com/microsoft/DeepSpeed/blob/84e0d03bc5ef9339d5bac6d8e784c50aaf098377/deepspeed/module_inject/replace_module.py#L574) seems to have been used without initializing. I tried using [GroupQuantizer](https://github.com/microsoft/DeepSpeed/blob/84e0d03bc5ef9339d5bac6d8e784c50aaf098377/deepspeed/module_inject/replace_module.py#L139) and it works but during forward pass, why is it not doing `selfAttention_int8` [here](https://github.com/microsoft/DeepSpeed/blob/84e0d03bc5ef9339d5bac6d8e784c50aaf098377/deepspeed/ops/transformer/inference/transformer_inference.py#L467). I get `!!!! kernel execution error.` at that point.
Originally posted by @sdpmas in #2217 (comment)