Original subtitles

Let's take a look at word-based tokenization. Word-based tokenization is the idea of splitting

the raw text into words, by splitting on spaces or other specific rules like punctuation. In this

algorithm, each word has a specific number, an "ID", attributed to it. In this example, "Let's"

has the ID 250, do has ID 861, and tokenization followed by an exclamation point has the ID 345.

This approach is interesting, as the model has representations that are based on entire words.

The information held in a single number is high

as a word contains a lot of contextual and semantic information in a sentence.

However, this approach does have its limits. For example, the word dog and the word

dogs are very similar, and their meaning is close. However, the word-based tokenization

will attribute entirely different IDs to these two words, and the model will therefore learn

different meanings for these two words. This is unfortunate, as we would like the model

to understand that these words are indeed related and that dogs is the plural form of the word dog.

Another issue with this approach is that there are a lot of different words in a language.

If we want our model to understand all possible sentences in that language,

then we will need to have an ID for each different word, and the total number of words,

which is also known as the vocabulary size, can quickly become very large.

This is an issue because each ID is mapped to a large vector that represents the word's meaning,

and keeping track of these mappings requires an enormous number of weights when the vocabulary

size is large. If we want our models to stay lean, we can opt for our tokenizer to ignore

certain words that we don't necessarily need. For example, when training our tokenizer on a text,

we might want to take the 10,000 most frequent words in that text

to create our basic vocabulary, instead of taking all of that language's words.

The tokenizer will know how to convert those 10,000 words into numbers,

but any other word will be converted to the out-of-vocabulary word, or the "unknown" word.

This can rapidly become an issue: the model will have the exact same representation

for all words that it doesn't know, which will result in a lot of lost information.