Original subtitles

So in this lecture, we are going to look at another example of tax classification, which is slightly

more complicated.

In this example, we are going to build a tax classifier where the input consists of not just one but

two sentences.

This is very practical and we can think of examples where we might want to have a model that knows how

to process multiple input sentences at the same time.

One example is answering multiple choice questions as input.

We might want to pass in the question along with all the possible answers, and we want our model to

select the right answer.

Another example is with chat bots.

With earlier chat bots, a simple way to train them was with single prompt and response pairs.

So you would give the chat bot a prompt and the chat bot would memorise appropriate responses.

But this doesn't take into account the history of the conversation, which might make the conversation

awkward.

Another example is question answering where we pass in a question along with a passage of text which

contains the answer.

We would then like the model to select the portion of the passage where the answer resides.

Now you might be wondering how are we going to build a transformer that can handle multiple sentences

as inputs?

If you have experience with machine learning or neural networks, you might have some sense that this

would be a difficult task.

Normally, with transfer learning, we only have to change the head of the Pre-Trained network while

keeping the inputs in the middle layers of the same.

But how can we change the number of inputs?

The answer is we don't have to.

It turns out that with the single input already in place, we can train the transformer to understand

having multiple sentences concatenated into the same inputs.

In fact, conceptually, this could work with almonds as well.

I'll leave it as an exercise for you to think about how that might work.

You'll recall that with Bert, which is the main model we are using in this section, is pre-trained

on unsupervised NLP tasks.

Well, it turns out that one of these tasks involves having multiple sentences in the same input.

So it makes sense that Bert should be able to handle multiple sentences.

In particular, the task is called next sentence prediction.

To build the data for this task, we take two sentences from our training corpus and the label is whether

or not the second sentence follows the first.

In other words, we build a binary classifier.

For Bird, we use special formatting to help the model understand where the first and second sentences

are located.

In particular, we always start with the class token.

We then follow it with the first sentence.

We then add the SEP token.

We then follow that with the second sentence.

And finally, we add another token at the end.

So hopefully you now understand that the utility of these special tokens, when we only had one sentence

as input, they may have seemed superfluous, but now they actually have practical utility.

The task we'll be looking at next is known as textual intelligence.

This might sound complicated, but in fact it is quite simple.

Thanks to my rule, all data is the same.

It is no different from the next sentence prediction task I just described.

In particular, we're still going to have to input sentences and the target will still be binary.

The only difference is the meaning of the task.

In this case, we want to know whether one sentence entails another.

For example, consider the input sentence pair.

Bob buys a car and Bob owns a car.

This is an example of where the first sentence entails the second.

Now consider Bob purchased cheese and Bob doesn't have cheese.

This is an example of where there is no entitlement.

Okay.

So now let's discuss what this will look like in code.

If you think about it, we know that the model has to be the same.

What changes are the inputs?

The input is still text and the output is still a binary prediction.

What changes are the inputs?

In other words, we should pay attention to the data set and tokenize her.

So the data set will look something like this.

Recall that you can think of it like a tabular data set with different columns for different things,

just like a csv previously.

We only had one input sentence and a label.

This time we'll have two sentences and a label in our dataset.

These are simply called sentence one incentives to.

Luckily, the tokenized brand hugging face is well equipped to handle two sentences at once.

You simply need to pass them in a separate arguments.

This will conceptually convert the inputs into a string formatted as we described earlier, where we

have the class token in the first sentence.

The SEP token.

The second sentence.

And another SEP token.

In reality, these of course, will be converted into tokens and then token IDs.

Now there's one important note to make at this point.

You'll recall that for the bird's organizer, we generate this input called token type IDs.

Having two input sentences will help us understand that the role of this input when we use Bird and

we have two sentences as input, the Birds organizer will use zeros for the positions of the first sentence

and ones for the positions of the second sentence.

So now you know that this input actually has a purpose and isn't totally superfluous.

However, this is an interesting point, and the next example will be using a slightly different model

called Distil Birds.

For this model, there is no input called token type IDs.

It is simply not necessary as an exercise.

After seeing the following notebook, you should double check that this is the case.

When we use Bird, there will be an input called token type IDs, but when we use two Silbert, there

won't be.

Also compare the results to see which model performs better along with their speed of training.

As you recall, we would like to apply that tokenisation to every sample in our data set object.

So we write a tokenized function that we can pass into the map method.

This will also take care of truncation.

The next step is to call the map method passing in the tokenize function, as we did before.

This gives us back our tokenized data.

Of course, at this point, the format of our data is exactly the same as it was in the previous lectures,

which means there is no more work to do.

As an exercise.

Since this is quite simple.

You might want to complete the following notebook on your own.

Before looking at the next lecture.

We'll be using the RC Easy to set up, which is also part of the glue benchmark.

So good luck and I'll see you in the next lecture.