Pytorch T5 does not run on GPU

[nreimers]

🐛 Bug

When I try to run T5 from the latest transformers version (and also from the most recent git version) on the GPU, I get the following error:

Traceback (most recent call last):
  File "T5_example.py", line 32, in <module>
    outputs = model(input_ids=input_ids)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 780, in forward
    
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 616, in forward
    encoder_decoder_position_bias=encoder_decoder_position_bias,
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 422, in forward
    self_attention_outputs = self.layer[0](
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 373, in forward
    attention_output = self.SelfAttention(
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 338, in forward
    raise ValueError("No position_bias provided and no weights to compute position_bias")
  File "/home/reimers/sbert/transformers/src/transformers/modeling_t5.py", line 289, in compute_bias
   values = self.relative_attention_bias(rp_bucket)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/reimers/anaconda3/envs/sbert/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

This is the example code to reproduce the problem:

from transformers import T5Model, T5Tokenizer
import torch

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5Model.from_pretrained('t5-small')
model = model.to('cuda')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute"), device='cuda').unsqueeze(0) 
outputs = model(input_ids=input_ids)
last_hidden_states = outputs[0]

The error is in the file modeling_t5.py at line 284-289:

rp_bucket = self._relative_position_bucket(
            relative_position,  # shape (qlen, klen)
            bidirectional=not self.is_decoder,
            num_buckets=self.relative_attention_num_buckets,
        )
values = self.relative_attention_bias(rp_bucket)  # shape (qlen, klen, num_heads)

rp_bucket is a tensor on the CPU, which causes the above error.

If I move rp_bucket to the GPU, the code works correctly on the GPU:

rp_bucket = self._relative_position_bucket(
            relative_position,  # shape (qlen, klen)
            bidirectional=not self.is_decoder,
            num_buckets=self.relative_attention_num_buckets,
        )
 rp_bucket = rp_bucket.to('cuda') #Dirty quick fix
 values = self.relative_attention_bias(rp_bucket)  # shape (qlen, klen, num_heads)

I'm not sure why rp_bucket is on the CPU.

[moscow25]

I can also confirm that T5 runs on CPU but not GPU -- thanks for the hack fix, will use that until GPU tensor is fixed.

[mohammedayub44]

Hi,
I was planning to run some examples with T5 on GPU. Is this already been fixed on GPU ?

[ArashHosseini]

@mohammedayub44, in v2.5.0 it works without any issue, I guess yes.

print (last_hidden_states)
tensor([[[ 9.2098e-02,  1.1048e-01,  2.6714e-02,  ...,  1.2918e-02,
           6.1260e-05,  9.5352e-02],
         [ 8.7042e-02,  8.3914e-02,  6.9337e-02,  ..., -3.9229e-02,
           3.3525e-04,  1.4291e-01],
         [ 9.6290e-02, -4.8915e-03,  5.5687e-02,  ..., -1.0703e-01,
           6.4940e-04, -2.1393e-01],
         [-3.0119e-03,  1.1048e-01,  3.0696e-03,  ..., -5.1768e-02,
           3.5166e-04,  1.5510e-01],
         [-6.3620e-02,  5.4474e-02, -1.8415e-02,  ..., -8.4559e-02,
           6.1696e-04,  5.8805e-02],
         [-6.0232e-02,  1.3885e-01,  7.9865e-03,  ..., -4.9981e-02,
           4.3370e-04,  4.4865e-02]]], device='cuda:0', grad_fn=<MulBackward0>)

[mohammedayub44]

Great I'll check it out. Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[go-dustin]

Has this been corrected? I'm on version 2.8.0 on a GCP AI Platform Notebook using the PyTorch:1.4 image and I'm still getting this error.

`
cuda0 = torch.device('cuda:0')
tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors="pt", max_length=512).to(cuda0)

summary_ids = model.generate(tokenized_text,
num_beams=2,
no_repeat_ngram_size=2,
min_length=50,
max_length=100,
early_stopping=True, )

output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select
`

[PeterAJansen]

I can confirm also having this issue on 2.10.0

[zzj0402]

RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

    model.to(DEVICE)
    model.train()
    input_ids.to(DEVICE)
    lm_labels.to(DEVICE)
    loss = model(input_ids=input_ids, lm_labels=lm_labels)[0]
    loss.backward()

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.