Original subtitles

Let's take a look at subword-based tokenization. Understanding why subword-based tokenization

is interesting requires understanding the flaws of word-based and character-based tokenization.

If you haven't seen the first videos on word-based and character-based tokenization,

we recommend you check them out before looking at this video.

Subword-tokenization lies in between character-based and word-based tokenization

algorithms. The idea is to find a middle ground between very large vocabularies, large quantity of

out-of-vocabulary tokens, loss of meaning across very similar words, for word-based tokenizers,

and very long sequences, less meaningful individual tokens for character-based tokenizers.

These algorithms rely on the following principle: frequently used words should not

be split into smaller subwords, but rare words should be decomposed into meaningful subwords.

An example is the word dog: we would like to have our tokenizer to have a single ID for the word

dog, rather than splitting it into characters: d, o, and g. However, when encountering the word

dogs, we would like our tokenizer to understand that at the root, this is still the word dog,

with an added s while slightly changes the meaning while keeping the original idea. Another example

is a complex word like tokenization, which can be split into meaningful subwords. The root of

the word is token, and ization completes the root to give it a slightly different meaning.

It makes sense to split the word into two: token, as the root of the word (labeled as the "start" of

the word). ization as additional information (labeled as a "completion" of the word).

In turn, the model will now be able to make sense of token in different situations.

It will understand that the words token, tokens, tokenizing, and tokenization are linked and have

a similar meaning. It will also understand that tokenization, modernization, and immunization,

which all have the same suffixes, are probably used in the same syntactic situations.

Subword-based tokenizers generally have a way to identify which tokens are start of words, and

which tokens complete start of words: token as the start of a word. ##ization as completing a word.

Here the ## prefix indicates that ization is part of a word rather than the beginning of it.

The ## comes from the BERT tokenizer, based on the WordPiece algorithm. Other tokenizers use other

prefixes, which can be placed to indicate part of words like seen here, or start of words instead!

There are a lot of different algorithms that can be used for subword tokenization, and most models

obtaining state-of-the-art results in English today use some kind of subword-tokenization

algorithm. These approaches help in reducing the vocabulary sizes by sharing information

across different words, having the ability to have prefixes and suffixes understood as such.

They keep meaning across very similar words, by recognizing similar tokens making them up.