Original subtitles

In this video, we'll study the encoder-decoder architecture.

An example of a popular encoder-decoder model is T5. In order to understand how the encoder-decoder

works, we recommend you check out the videos on encoders and decoders as standalone models.

Understanding how they behave individually will help understanding how an encoder-decoder behaves.

Let's start from what we've seen about the encoder. The encoder takes words as inputs,

casts them through the encoder, and retrieves a numerical representation

for each word cast through it. We now know that the numerical representation holds information

about the meaning of the sequence. Let's put this aside and add the decoder to the diagram.

In this scenario, we're using the decoder in a manner that we haven't seen before.

We're passing the outputs of the encoder directly to it! Additionally to the encoder outputs,

we also give the decoder a sequence. When prompting the decoder for an output with no

initial sequence, we can give it the value that indicates the start of a sequence.

And that's where the encoder-decoder magic happens. The encoder accepts a sequence as input.

It computes a prediction, and outputs a numerical representation. Then, it sends

that over to the decoder. It has, in a sense, encoded the sequence. And the decoder, in turn,

using this input alongside its usual sequence input, will take a stab at decoding the sequence.

The decoder decodes the sequence, and outputs a word. As of now, we don't need to make sense of

that word, but we can understand that the decoder is essentially decoding what the encoder has

output. The "start of sequence word" indicates that it should start decoding the sequence.

Now that we have both the feature vector and an initial generated word, we don't need the

encoder anymore. As we have seen before with the decoder, it can act in an auto-regressive manner;

the word it has just output can now be used as an input. This, in combination with the

numerical representation output by the encoder, can now be used to generate a second word.

Please note that the first word is still here; as the model still outputs it. However, it is greyed

out as we have no need for it anymore. We can continue on and on; for example until the decoder

outputs a value that we consider a "stopping value", like a dot, meaning the end of a sequence.

Here, we've seen the full mechanism of the encoder-decoder transformer: let's go over it one

more time. We have an initial sequence, that is sent to the encoder. That encoder output is then

sent to the decoder, for it to be decoded. While we can now discard the encoder after a single use,

the decoder will be used several times: until we have generated every word that we need.

Let's see a concrete example; with Translation Language Modeling; also called transduction;

the act of translating a sequence. Here, we would like to translate this English sequence "Welcome

to NYC" in French. We're using a transformer model that is trained for that task explicitly. We use

the encoder to create a representation of the English sentence. We cast this

to the decoder and, with the use of the start of sequence word, we ask it to output the first word.

It outputs Bienvenue, which means "Welcome".

We then use "Bienvenue" as the input sequence for the decoder. This, alongside the feature vector,

allows the decoder to predict the second word, "à", which is "to" in English.

Finally, we ask the decoder to predict a third word; it predicts "NYC",

which is, once again, correct. We've translated the sentence! Where the encoder-decoder really

shines, is that we have an encoder and a decoder; which often do not share weights.

We, therefore, have an entire block (the encoder) that can be trained to understand the sequence,

and extract the relevant information. For the translation scenario we've seen earlier, for

example, this would mean parsing and understanding what was said in the English language; extracting

information from that language, and putting all of that in a vector dense in information.

On the other hand, we have the decoder, whose sole purpose is to decode the feature output by

the encoder. This decoder can be specialized in a completely different language, or even modality

like images or speech. Encoders-decoders are special for several reasons. Firstly,

they're able to manage sequence to sequence tasks, like translation that we have just seen.

Secondly, the weights between the encoder and the decoder parts are not necessarily shared. Let's

take another example of translation. Here we're translating "Transformers are powerful" in French.

Firstly, this means that from a sequence of three words, we're able to generate

a sequence of four words. One could argue that this could be handled with a decoder;

that would generate the translation in an auto-regressive manner; and they would be right!

Another example of where sequence to sequence transformers shine is in summarization.

Here we have a very long sequence, generally a full text,

and we want to summarize it. Since the encoder and decoders are separated,

we can have different context lengths (for example a very long context for the encoder which

handles the text, and a smaller context for the decoder which handles the summarized sequence).

There are a lot of sequence to sequence models. This contains a few examples of

popular encoder-decoder models available in the transformers library.

Additionally, you can load an encoder and a decoder inside an encoder-decoder

model! Therefore, according to the specific task you are targeting,

you may choose to use specific encoders and decoders, which have proven their worth

on these specific tasks. This wraps things up for the encoder-decoders. Thanks for watching!