Original subtitles

In this video, we'll study the decoder architecture. An example of a popular decoder-only architecture

is GPT-2. In order to understand how decoders work, we recommend taking a look at the video

regarding encoders: they're extremely similar to decoders. One can use a decoder for most

of the same tasks as an encoder, albeit with, generally, a little loss of performance. Let's

take the same approach we have taken with the encoder to try and understand the architectural

differences between an encoder and a decoder. We'll use a small example, using three words.

We pass them through the decoder. We retrieve a numerical representation of each word. Here,

for example, the decoder converts the three words “Welcome to NYC” in these three

sequences of numbers. The decoder outputs exactly one sequence of numbers per input

word. This numerical representation can also be called a "Feature vector", or "Feature

tensor". Let's dive in this representation. It contains one vector per word that was passed

through the decoder. Each of these vector is a numerical representation of the word

in question. The dimension of that vector is defined by the architecture of the model.

Where the decoder differs from the encoder is principally with its self-attention mechanism.

It's using what is called "masked self-attention". Here for example, if we focus on the word

"to", we'll see that its vector is absolutely unmodified by the "NYC" word. That's because

all the words on the right (also known as the right context) of the word is masked.

Rather than benefitting from all the words on the left and right, I.e., the bidirectional

context, decoders only have access to the words on their left. The masked self-attention

mechanism differs from the self-attention mechanism by using an additional mask to hide

the context on either side of the word: the word's numerical representation will not be

affected by the words in the hidden context. So when should one use a decoder? Decoders,

like encoders, can be used as standalone models. As they generate a numerical representation,

they can also be used in a wide variety of tasks. However, the strength of a decoder

lies in the way a word has access to its left context. The decoders, having only access

to their left context, are inherently good at text generation: the ability to generate

a word, or a sequence of words, given a known sequence of words. In NLP, this is known as

Causal Language Modeling. Let's look at an example. Here's an example of how causal language

modeling works: we start with an initial word, which is "My". We use this as input for the

decoder. The model outputs a vectors of dimension 768. This vector contains information about

the sequence, which is here a single word, or word. We apply a small transformation to

that vector so that it maps to all the words known by the model (mapping which we'll see

later, called a language modeling head). We identify that the model believes the most

probable following word is "name". We then take that new word, and add it to the initial

sequence. From "My", we are now at "My name". This is where the "autoregressive" aspect

comes in. Auto-regressive models re-use their past outputs as inputs in the following steps.

Once again, we do that the exact same operation: we cast that sequence through the decoder,

and retrieve the most probable following word. In this case, it is the word "is". We repeat

the operation until we're satisfied. Starting from a single word, we've now generated a

full sentence. We decide to stop there, but we could continue for a while; GPT-2, for

example, has a maximum context size of 1024. We could eventually generate up to 1024 words,

and the decoder would still have some memory of the first words of the sequence! If we

go back several levels higher, back to the full transformer model, we can see what we

learned about the decoder part of the full transformer model. It is what we call, auto-regressive:

it outputs values that are then used as its input values. We repeat this operations as

we like. It is based off of the masked self-attention layer, which allows to have word embeddings

which have access to the context on the left side of the word. If you look at the diagram

however, you'll see that we haven't seen one of the aspects of the decoder. That is: cross-attention.

There is a second aspect we haven't seen, which is it's ability to convert features

to words; heavily linked to the cross attention mechanism. However, these only apply in the

"encoder-decoder" transformer, or the "sequence-to-sequence" transformer (which can generally be used interchangeably).

We recommend you check out the video on encoder-decoders to get an idea of how the decoder can be used

as a component of a larger architecture!