example for is next sentence

[charlesmartin14]

Can you make up a working example for 'is next sentence'

Is this expected to work properly ?

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized input
text = "Who was Jim Morrison ? Jim Morrison was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# Load pre-trained model (weights)
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
model.eval()

# Predict is Next Sentence ?
predictions = model(tokens_tensor, segments_tensors)

[thomwolf]

I think it should work. You should get a [1, 2] tensor of logits where predictions[0, 0] is the score of Next sentence being True and predictions[0, 1] is the score of Next sentence being False. So just take the max of the two (or use a SoftMax to get probabilities).
Did you try it?
The model behaves better on longer sentences of course (it's mainly trained on 512 tokens inputs).

[thomwolf]

Closing that for now, feel free to reopen if there is another issue.

[Alexadar]

Guys, are [CLS] and [SEP] tokens mandatory for this example?

[dirkgr]

This is not super clear, even wrong in the examples, but there is this note in the docstring for BertModel:

`pooled_output`: a torch.FloatTensor of size [batch_size, hidden_size] which is the output of a
    classifier pretrained on top of the hidden state associated to the first character of the
    input (`CLF`) to train on the Next-Sentence task (see BERT's paper).

That seems to suggest pretty strongly that you have to put in the CLF token.

[hackable]

from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM,BertForNextSentencePrediction

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized input
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# Load pre-trained model (weights)
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
model.eval()

# Predict is Next Sentence ?
predictions = model(tokens_tensor, segments_tensors )




print(predictions)

tensor([[ 6.3714, -6.3910]], grad_fn=<AddmmBackward>)

How do i infer this as true or false

[dariocazzani]

Those are the logits, because you did not pass the next_sentence_label.

My understanding is that you could apply a softmax and get the probability for the sequence to be a possible sequence.

Sentence 1: How old are you?
Sentence 2: The Eiffel Tower is in Paris
tensor([[-2.3808, 5.4018]], grad_fn=<AddmmBackward>)
Sentence 1: How old are you?
Sentence 2: I am 193 years old
tensor([[ 6.0164, -5.7138]], grad_fn=<AddmmBackward>)

For the first example the probability that the second sentence is a probable continuation is very low.
For the second example the probability is very high (I am looking at the first logit)

[pengjiao123]

predictions = model(tokens_tensor, segments_tensors )
I try the code more than once，why I have the different result？
sometime predictions[0, 0] is higher ，however， the same sentence pair，predictions[0, 0] is lower.

[thomwolf]

Maybe your model is not in evaluation mode (model.eval())?
You need to do this to desactivate the dropout modules.

[pengjiao123]

It is OK.THANKS A LOT.

[AIGyan]

error: --> 197 embeddings = words_embeddings + position_embeddings + token_type_embeddings 198 embeddings = self.LayerNorm(embeddings) 199 embeddings = self.dropout(embeddings) The size of tensor a (21) must match the size of tensor b (14) at non-singleton dimension 1

The above issues get resolved, when I added few extra 1's and 0's to make the shape similar tokens_tensor and segments_tensors. Just wondering am I using in a right way.

My predictions output is a tensor array of size 21 X 30522 .
And what I believe the example is to predict the word which is [MASK] . Can you also please guide how to predict the next sentence?

[yuchuang1979]

Maybe your model is not in evaluation mode (model.eval())?
You need to do this to desactivate the dropout modules.

@thomwolf Actually even when I used model.eval() I still got different results. I observed this when I use every model of the package (BertModel, BertForNextSentencePrediction etc). Only when I fixed the length of the input (e.g. to 128), I can get the same results. In this way I have to pad 0 to indexed_tokens so it has a fixed length.

Could you explain why is like this, or did I make any mistake?

Thank you so much!

[Alexadar]

Maybe your model is not in evaluation mode (model.eval())?
You need to do this to desactivate the dropout modules.

@thomwolf Actually even when I used model.eval() I still got different results. I observed this when I use every model of the package (BertModel, BertForNextSentencePrediction etc). Only when I fixed the length of the input (e.g. to 128), I can get the same results. In this way I have to pad 0 to indexed_tokens so it has a fixed length.

Could you explain why is like this, or did I make any mistake?

Thank you so much!

Make sure

1. input_ids, input_mask, segment_ids have same length
2. vocabulary file for tokenizer is from the same config dir as your bert_config.json

I had symilar symptoms when vocab and config was from diferent berts