Original subtitles

How to batch inputs together? In this video, we will see how to batch input sequences together.

In general, the sentences we want to pass through our model won't all have the same

lengths. Here we are using the model we saw in the sentiment analysis pipeline and want to classify

two sentences. When tokenizing them and mapping each token to its corresponding input IDs,

we get two lists of different lengths.

Trying to create a tensor or a NumPy array from those two lists will result in an error, because

all arrays and tensors should be rectangular. One way to overcome this limit is to make the

second sentence the same length as the first by adding a special token as many times as necessary.

Another way would be to truncate the first sequence to the length of the second, but we

would them lose a lot of information that might be necessary to properly classify the sentence.

In general, we only truncate sentences when they are longer than the maximum length the

model can handle. The value used to pad the second sentence should not be picked randomly: the model

has been pretrained with a certain padding ID, which you can find in tokenizer.pad_token_id.

Now that we have padded our sentences, we can make a batch with them.

If we pass the two sentences to the model separately and batched together however,

we notice that we don't get the same results for the sentence that is padded (here the second one).

If you remember that Transformer models make heavy use of attention layers, this should

not come as a total surprise: when computing the contextual representation of each token,

the attention layers look at all the other words in the sentence. If we have just the sentence or

the sentence with several padding tokens added, it's logical we don't get the same values.

To get the same results with or without padding, we need to indicate to the attention layers

that they should ignore those padding tokens. This is done by creating an attention mask,

a tensor with the same shape as the input IDs, with zeros and ones. Ones indicate the tokens the

attention layers should consider in the context and zeros the tokens they should ignore. Now

passing this attention mask along with the input ids will give us the same results as when we sent

the two sentences individually to the model! This is all done behind the scenes by the tokenizer

when you apply it to several sentences with the flag padding=True. It will apply the padding with

the proper value to the smaller sentences and create the appropriate attention mask.