Original subtitles

So in this lecture we will move on to the next step of our question answering notebook.

Now, as you recall, typically after we load in the data, we apply the token user to convert the text

into a numerical format for the neural network.

At a high level.

That seems simple, but as you've seen many times already, the devil is in the details.

Firstly, let's note that the checkpoint will be using for this notebook will be either Bert or distiller

Bert.

This might surprise you since question answering seems like a pretty complicated task.

So you think it might require an encoder decoder setup?

But this is not the case.

This will be easier to understand once we discuss the model outputs and perhaps some theory about how

transformers actually work.

For now, it suffices to know that Bert and Bird like models are the correct choice for the token.

This means that we'll be using the same token ISAR that we used earlier in the course.

As you know, this token ISAR can already handle two input sentences such as you saw in the textual

entitlement task.

Okay.

So just to review what happens when we use the tokenization to tokenize two pieces of text and how do

we do it?

Well, you'll recall that this works by simply passing in the two pieces of text as two separate arguments

by convention will pass in the question first and then the context.

If we decode the outputs from the tokenized, meaning turn the token IDs back into words, we will get

a big, long string containing both the question and the context.

Concatenate it together.

In particular, this always starts with the special CLS token, followed by the first sentence followed

by the SEP token followed by the second sentence.

And finally, one more step token.

Now, please note that I'm using the word sentence loosely.

In fact, the context may contain more than one sentence.

In fact, it likely will most, if not all of the time.

Now, one challenging aspect of question answering is that the context part of the input can be really

long.

This is unlike the textual entitlement or next sentence prediction examples, since for those the two

inputs are just two actual sentences.

In this case, the context will contain many sentences.

Now, as you recall, Bert can only handle a limited number of tokens, but it wouldn't be a good idea

to truncate the context, since the part we leave out may contain the answer to the question.

And it also wouldn't be a good idea to truncate the question.

Since then, we wouldn't know the question.

So what is the solution to this?

The solution is to split the context into multiple windows.

As a result, one data sample will turn into multiple data samples, some of which will contain the

answer and some of which will not.

So by doing this, we introduce a case where no answer can be found.

Now you might be concerned for those of you very keen students, you might notice that we have a problem

when the answer crosses the boundary between these windows.

If half of the answer is at the end of one context window and the other half of the answer is at the

beginning of the next context window, then no answer will be valid and the model won't really be learning

the answers to the questions.

We solve this problem by using overlapping windows instead.

In the hugging face library, the amount of overlap is called the stride.

So the full tokenized call looks like this as before we pass in the context and the question.

The next argument is maxlength where we specify the maximum length of the entire input.

This includes the question and the context and the special tokens.

The next step is to specify special instruction only seconds.

This means, since the context is the second input text, only truncate this but do not truncate the

question which is the first input text.

The next argument is the stride, which defines how much overlap there is between the context windows

when they are split up.

And finally we set return overflowing tokens to true a better name for this would probably be overlapping

tokens since this corresponds to the tokens that overlap when we set the stride to a positive number.

Importantly, note that when we call the tokenizing like this, it expands the data.

If we have one question in context pair, this might be converted into multiple input samples depending

on how long the context is.

One thing that will become important later on is how we know which sample from the tokenized output

corresponds to which sample from the tokenized input.

Imagine for instance, that we enumerated the original input samples.

So we have sample zero, sample one, sample two and so forth.

At the output, we might have 00011, two, three, three, three and so forth.

This means that Sample zero got expanded into three separate model inputs.

Sample one got expanded into two model inputs and so forth.

Luckily we do get access to this data when we call the token user as described above.

This will return a new key we haven't seen before called overflow to sample mapping.

It contains exactly these integers, specifically the original input sample indices.

Now there's one more important argument into the token riser that I've left out until now.

This is the argument return offsets, mapping.

We will be setting this true now like the previous overflow to sample mapping.

This probably seems very random and unnecessary, but don't worry, I feel the same way.

It will all come together when we look at our next task.

But this is just to set up the preliminaries, so let's just assume that it's useful.

Basically, when we set this argument to true, we will get back an additional token as your output

in addition to the usual input IDs, attention mask and so forth.

This will be called the offset mapping.

What this does is for each model input, it's going to give us a list of tuples.

Each of these tuples corresponds to a token in the input sequence.

Specifically, each tuple will contain the start and end character positions of that token.

So as an example, suppose the question is where is Bob?

And the context is Bob is at home.

The first tuple is zero zero, which corresponds to the special CLS token because technically this doesn't

take up any space.

The second tuple goes from 0 to 5 because the word aware contains five letters.

The next tuple goes from 6 to 8 because the word is contains two letters and so forth.

Note that after the question is complete, we have another zero zero tuple which corresponds to the

SEP token which does not take up any space.

After this we have the tuples for the context which importantly start counting from zero.

Now again, it might be totally unclear why we would even need this information.

To give you some intuition for why we might need this.

Consider that when we finally want to represent the answer to our question, the answer will be given

as start and end positions of each token.

So, for example, token a 2 to 4, which corresponds to the phrase at home, but in order to convert

this back into a string to present to the user, we must know where these tokens begin and end.

That is what is the first character and what is the last.

So hopefully that gives you some idea of why this information is needed.