All language subtitles for 005 Using the Tokenizer in Python_en

So in this lecture, we will be continuing our notebook from earlier in the session.

As you know, the next step will be to load up our tokenize.

In this notebook, we'll be using the stillbirth as our checkpoint, since it trains much faster than

Bert.

But if you're doing this at home by yourself, I would recommend trying Bert instead.

In addition, a two to Bert.

As usual we call auto tokenize or from pre-trained passing in the checkpoint.

The next step is to test our token isare on a question and context pair.

We'll choose the sample at index one again.

In order to understand the results, we'll take the input IDs and call tokenize or decode to get the

result back in English.

So as expected, we get both the question and context.

Concatenate it into one string.

Note that by convention we'll always put the question first.

As you can see, the string starts with the special CLS token as usual.

Then we have the question, which is what is in front of the Notre Dame main building?

Then we have the SEP token, then we have the context, and then it ends with another set.

So as you've just seen, the context can be quite long.

So long, in fact, that the combination of the context and question might be too long for the model

to handle the accepted strategy.

For this is to split the context into multiple samples.

In order to do this, we need to specify a few additional arguments when the tokenizing is called.

Specifically, we set max length to 100, specifying that the number of input tokens for any given sample

should be in most 100 tokens long.

We set truncation into only seconds, which means that we will only truncate the context but not the

question.

As you recall, the context is the second input string.

We wouldn't want to truncate the question since then.

We wouldn't know what the question is.

And because we split the context into multiple windows, we don't want to accidentally cut off the answer.

Meaning we don't want part of the answer to be in one window and the other part of the answer to be

in another window.

So in order to avoid that situation, we set the stride which in this case has been set to 50.

Note that 150 are just toI values for this experiment.

Later on, we'll be using different values.

Finally we said return overflowing tokens the true meaning.

We want to return the tokens in the overlapping parts of the context windows.

So once we've called the token Pfizer note that it will give us multiple samples.

This is because the context is split into multiple windows and it's combined with the question each

time.

So after we've called the token isare, we're going to loop through each of the resulting samples and

print the decoded input IDs.

So as you can see, we get four new samples.

This means that the context was split into four different parts.

Note that the question has not been truncated, which is what we specified.

Instead, the context in this case is just a part of the full context which we saw above.

And again, note that we have the CLS and SEP tokens in their usual places.

Now you know that when we call the organizer, we don't just get back the token IDs.

So let's check the keys of the result to see what we actually got.

So in addition to the input IDs and attention mask, which you've already seen, we get another item

called overflow to sample mapping.

So let's print this out just to see what it is.

So it's just a list of zeroes, which is not that informative.

What I'm going to do is tell you right now what this means and then do a more meaningful demonstration

in the next step.

So what does this mean?

Well, as you recall, when we split up the context, we get back multiple samples.

However, we may want to know what is the original sample index it came from?

That's what this tells us.

Since we only passed in one original sample, this is going to be zero each time because they all came

from the zero original sample.

So just to get a better feel for this overflow to sample mapping, we're going to call the tokenizing

once again, but this time on the first three samples instead of just one, we're also going to add

one additional argument return offsets mapping which has been set to true.

Again, we print out the overflow to sample mapping results.

So hopefully these results make more sense.

As you can see, all of our input samples have been split into four parts.

This tells us that the first four outputs correspond to the original sample at index zero.

The next four outputs correspond to the original sample at index one, and the final four outputs correspond

to the original sample at index two.

Now to really confirm that this is the case, we can also decode the input IDs.

So as you can see, this confirms that each of the first three samples really was split into four parts.

This is why we see the same three questions repeated four times.

So for the next step, we're going to go back to looking at a single context question pair with the

same arguments.

As you recall, we just added a new argument called the return offsets mapping.

This will return something that will be very useful throughout this notebook.

We'll follow the tokenized call by checking the keys of the output once again.

So as you can see, we get one new output, which is called offset mapping.

So let's take a look at what this offset mapping actually is.

So as you can see, it appears to be a list of lists of tuples.

Note that I've also made a note in the comments describing what you see.

So check that out as well if you need.

Basically these tuples show us the character positions of each token in the model input.

As you recall, the model input can really be conceptualized as a string, and a string is really an

array of characters.

So if we think of the input as an array of characters, we can describe where each token is located

by its positions in that array.

Note that special tokens like CLS and CEP always have the tuples zero zero.

Otherwise, consider the question for this input.

Recall that it's what is in front of the Notre Dame main building.

The first tuple other than CLS is zero four.

This makes sense because the first token of the question starts at position zero and it's four characters,

long as in the word of what has four characters in it.

The next tuple is five seven.

This makes sense because the next word which is is starts at the next position and is two characters

long.

The next tuple is 810, which also makes sense.

The next word is in which is two characters long.

The next tuple is 1116, which again makes sense.

This corresponds to the word front, which is five characters long.

So hopefully you get the idea.

Please feel free to go through the rest of the words in the question to make sure that this makes sense

to you.

Note that the final token is just one character which corresponds to the question mark.

Another important detail to notice is that when we look at the context, the indices start back at zero.

This will be useful to know for any computations you want to do later on.

Now, as you recall, although we only have one question and context pairs input, the tokenize are

still produced multiple samples because the context was split up.

So let's have a look at the offset mapping corresponding to the next sample.

So here we can see that it goes to the next sample because a new list starts.

Now, as you can see, it starts at the same.

We have zero zero for CLS.

Then we have other tuples for the question, which again is what is in front of the Notre-Dame main

building question mark.

We again have zero zero for Sep.

But then we have something interesting.

The first context tuple starts with 174 180.

Importantly, it does not start from zero.

What this means is that these two tell us the positions of the characters in the original full context.

Again, these facts may seem kind of random, but you'll need to know them in order to properly understand

the code.

Later on in this notebook.

Now, just to make things more concrete, let's check the size of the offset mapping list.

So as you can see, it contains four elements because the original question context pair was split into

four different model inputs.

What may also be useful is checking the length of the elements inside the parent list.

As you can see, it contains 100 tuples since we truncated each input to be size 100.

Now you should note that not all of the inputs will be of this size.

As an exercise, try to find one that isn't and think about why.