[BUG] OPT inference fp32 predictions are wrong

[rahul003]

Describe the bug
Fp32 predictions are incorrect, but fp16 predictions are good.

Logits comparision: HF, Deepspeed

  tensor([[[-7.3910, -7.7681,  4.7013,  ..., -7.0938, -7.1234, -7.1775],
         [-6.5882, -6.6528,  2.2693,  ..., -6.7766, -6.4634, -6.7310],
         [-6.3671, -6.7828,  2.7458,  ..., -6.8141, -6.5591, -6.5313],
         ...,
         [-3.6584, -3.5718,  4.9058,  ..., -3.8075, -3.8717, -3.6752],
         [-2.7994, -3.0189,  3.8580,  ..., -3.0059, -2.8154, -3.0953],
         [-4.6610, -4.1895,  3.7045,  ..., -4.5697, -4.5012, -4.3216]]],
       device='cuda:0', grad_fn=<UnsafeViewBackward0>) tensor([[[-7.3958, -7.7729,  4.6978,  ..., -7.0986, -7.1283, -7.1822],
         [-5.6408, -5.6727,  1.1509,  ..., -5.6369, -5.5548, -5.8536],
         [-5.6177, -5.6468,  3.2333,  ..., -5.6963, -5.8001, -5.5175],
         ...,
         [-6.7400, -6.7934,  6.3651,  ..., -6.3616, -6.4490, -6.3501],
         [-8.1459, -8.1146,  6.3947,  ..., -7.5899, -7.8562, -7.7664],
         [-7.5022, -7.2970,  4.8574,  ..., -7.1109, -6.9802, -7.1456]]],
       device='cuda:0', grad_fn=<UnsafeViewBackward0>)

Predictions with model.generate

tensor([[ 5625,    16,    10,  2721,   183,     8,    38,   236,     7,   458,
            19,    47,    10,   367,     9,   127,  2674,   383,     4, 50118]],
       device='cuda:0') tensor([[5625,   16,   10, 2721,  183,    8,   38,  236,    7,   16,   10,  205,
         1246,    9,    5, 2136,   16,   10,  205, 1246]], device='cuda:0')
['Today is a beautiful day and I want to share with you a few of my favorite things.\n'] ['Today is a beautiful day and I want to is a good example of the word is a good example']

To Reproduce

import os
import torch
import deepspeed
import transformers

from deepspeed import module_inject
from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer

# Get local gpu rank from torch.distributed/deepspeed launcher
local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))

print(
    "***************** Creating model in RANK ({0}) with WORLD_SIZE = {1} *****************"
    .format(local_rank,
            world_size))
#generator = pipeline('text-generation',
#                     model='facebook/opt-1.3b',
#                     device=local_rank)
model = AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b')
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-1.3b')
model.cuda()

use_pipe = False
use_fp16 = False

if use_pipe:
    generator = pipeline('text-generation',
                     model='facebook/opt-1.3b',
                     device=local_rank)
else:
    model = AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b')
    tokenizer = AutoTokenizer.from_pretrained('facebook/opt-1.3b')
    model.cuda()

if use_fp16:
    model.half()
    dt = torch.half
else:
    dt = torch.float

prompts = "Today is a beautiful day and I want to"

if not use_pipe:
    inputs = tokenizer(prompts, return_tensors="pt", add_special_tokens=False)
    inputs.to(local_rank)

    hf_logits = model(inputs.input_ids, attention_mask=inputs.attention_mask, output_hidden_states=True)['logits']
    hf_tokens = model.generate(inputs.input_ids)
    model = deepspeed.init_inference(model, mp_size=world_size, dtype=dt, replace_method='auto', replace_with_kernel_inject=True)
else:
    hf_output = generator(prompts)
    generator.model = deepspeed.init_inference(generator.model, mp_size=world_size, dtype=dt, replace_method='auto', replace_with_kernel_inject=True)


if not use_pipe:
    logits = model(inputs.input_ids, attention_mask=inputs.attention_mask, output_hidden_states=True)['logits']

    print(hf_logits, logits)
    tokens = model.generate(inputs.input_ids)

    print(hf_tokens,tokens)
    print(tokenizer.batch_decode(hf_tokens), tokenizer.batch_decode(tokens))
else:
    ds_output = generator(prompts)
    print(hf_output, ds_output)

Expected behavior
Matching outputs. With fp16 outputs do match.

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.11.0+cu113
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.3
deepspeed install path ........... ['/fsx/inference/deepspeed/deepspeed']
deepspeed info ................... 0.7.3+c84bca37, c84bca37, master
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3

Screenshots
NA

System info (please complete the following information):

• OS: [e.g. Ubuntu 18.04] 20.04
• GPU count and types [e.g. two machines with x8 A100s each] A100 GPU 40GB
• Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB] 1 node
• Python version: 3.8
• Any other relevant info about your setup

Launcher context
Single process

Docker context
Are you using a specific docker image that you can share?
Can't share but nothing special.

Additional context
Add any other context about the problem here.

[arashb]

Thanks @rahul003 for reporting this issue. We have fixed this issue in the PR #2285 which is not merged yet. Feel free to switch to branch ds-inference/opt-FP32-accuracy-fix in the meantime, and also please let us know if you face any other issue regarding OPT models with FP32 precision.

[rahul003]

PR looks good