Original subtitles

So, let's get started.

So I'll be talking about building LLMs today.

So I think a lot of you have heard of LLMs before, but just

as a quick recap.

LLMs standing for large language models

are basically all the chat bots that you've

been hearing about recently.

So, ChatGPT, from OpenAI, Claude, from Anthropic, Gemini

and Llama, and other types of models like this.

And today we'll be talking about how do they actually work.

So it's going to be an overview because it's only one lecture

and it's hard to compress everything.

But hopefully, I'll touch a little bit

about all the components that are needed

to train some of these LLMs.

Also, if you have questions, please interrupt me

and ask if you have a question.

Most likely other people in the room or on Zoom have other.

Have the same questions.

So, please ask.

Great.

So what matters when training LLMs.

So there are a few key components that matter.

One is the architecture.

So as you probably all LLMs are neural networks,

and when you think about neural networks,

you have to think about what architecture you're using.

And another component, which is really important

is the training loss and the training algorithm.

So, how you actually train these models, then it's data.

So, what do you train these models on.

The evaluation, which is how do you

know whether you're actually making progress

towards the goal of LLMs and then, the system component.

So that is like how do you actually

make these models run on modern hardware, which

is really important because these models are really large.

So now more than ever, systems are actually

really an important topic for LLMs.

So those five components, you probably all know that LLMs.

And if you don't know LLMs are all

based on transformers or at least some version

of transformers.

I'm actually not going to talk about the architecture today.

One, because I gave a lecture on transformers a few weeks ago

and two, because you can find so much information online

on transformers.

There's much less information about the other four topics.

So, I really want to talk about those.

And another thing to say is that most of academia

actually focuses on architecture and training

algorithm and losses as academics

and I've done that for a big part of my career,

is simply we like thinking that this is like we make

new architectures, new models, and it

seems like it's very important.

But in reality, honestly, what matters in practice is mostly

the three other topics.

So, data, evaluation and systems, which is what most

of industry actually focuses on.

So, that's also one of the reasons

why I don't want to talk too much about the architecture,

because really the rest is super important.

Great.

So, overview of the lecture, I'll

be talking about pretraining.

So, pretraining, you probably heard that word.

This is the general word.

This is kind of the classical language modeling paradigm where

you basically train your language model to essentially

model all of internet.

And then, there's a post training,

which is a more recent paradigm which

is taking these large language models

and making them essentially AI assistants.

So, this is more of a recent trend since ChatGPT.

So, if you ever heard of GPT3 or GPT2,

that's really pretraining land.

If you heard of ChatGPT, which you probably have,

this is really post training land,

so I'll be talking about both, but I'll start with pretraining

and specifically I'll talk about what

is the task of pretraining LLMs and what is the loss that people

actually use.

So, language modeling, this is a quick recap.

Language models at a high level are simply

models of probability distribution over sequences

of tokens or of words.

So it's basically some model of p of x1

to XL, where x1 is basically what

one and XL is the last one in the sequence or in the sentence.

So, very concretely, if you have a sentence like the mouse

ate the cheese, what the language model gives

you is simply a probability of this sentence being uttered

by a human or being found online.

So, if you have another sentence like "The the mouse ate cheese."

Here, there's grammatical mistakes.

So, the model should know that this should

have some syntactic knowledge.

So, it should know that this has less likelihood

of appearing online.

If you have another sentence like the cheese ate the mouse,

then the model should hopefully know about the fact

that usually cheese don't eat mouse.

So, there's some semantic knowledge

and this is less likely that the first sentence.

So, this is basically at a high level what language models are.

One word that you probably have been hearing a lot in the news

are generative models.

So, this is just something that can generate.

Models that can generate sentences

or can generate some data.

The reason why we say language models are generative models

is that once you have a model of a distribution,

you can simply sample from this model.

And now we can generate data.

So we can generate sentences using a language model.

So the type of models that people are all currently using

are what we call autoregressive language models.

And the key idea of autoregressive language models

is that you take this distribution over words

and you basically decompose it into the distribution

of the first word, multiply by the distribution of

or the likelihood of the distribution of the second word

given the first word, and multiply it

by P of the third word given the first two words.

So, there's no approximation here.

This is just the chain rule of probability, which you

hopefully you all know about.

Really no approximation.

This is just one way of modeling a distribution.

So, slightly more concisely, you can write it

as a product of P's of the next word, given everything which

happened in the past.

So, of the context.

So, this is what we call autoregressive language models.

Again, this is really not the only way

of modeling distribution.

This is just one way.

It has some benefits and some downsides.

One downside of autoregressive language models

is that when you actually sample from this autoregressive

language model, you basically have

a for loop, which generates the next word, then conditions

on that next word.

And then we generate in other words.

So, basically if you have a longer sentence

that you want to generate, it takes more time to generate it.

So, there are some downsides of this current paradigm,

but that's what we currently have.

So, I'm going to talk about this one.

Great.

So, autoregressive language models.

At a high level, what a task of autoregressive language model

is simply predicting the next word, as I just said.

So, if we have a sentence like she likely prefers,

one potential, next word might be dogs.

And the way we do it is that we first tokenize.

So, you take these words or subwords you tokenize them

and then you give an ID for each token.

So here you have one, two, three.

Then, you pass it through this black box.

As I already said, we're not going

to talk about the architecture.

You just pass it through, pass it through a model,

and you then get a distribution, a probability distribution

over the next word or over the next token.

And then you sample from this distribution,

you get a new token and then you detokenize.

So, you get a new ID, you detokenize

and that's how you basically sample from a language model.

One thing which is important to note

is that the last two steps are actually

only needed during inference.

When you do training, you just need

to predict the most likely token and you can just

compare to the real token which happened in practice,

and then, you basically change the weights

of your model to increase the probability of generating

that token.

Great.

So, autoregressive neural language models.

So to be slightly more specific, still,

without talking about the architecture,

the first thing we do is that we have all of these.

Sorry, yes.

On the previous slide.

Predicting the probability of the next token,

does this mean that your final output vector has

to be the same dimensionality as the number of tokens

that you have?

Yes.

How do you deal with if you have more token.

Adding more token to your [INAUDIBLE]?

Yeah so we're going to talk about tokenization

actually later so you will get some sense of this.

You basically can deal with adding new tokens.

I'm kind of exaggerating.

There are methods for doing it, but essentially people

don't do it.

So it's really important to think about

how you tokenize your text, and that's why

we'll talk about that later.

But it's a very good point to note

is that you basically-- the vocabulary size, so

the number of tokens that you have is essentially

the output of your language model.

So it's actually pretty large.

So autoregressive neural language models.

First thing you do is that you take every word or every token.

You embed them so you get some vector representation

for each of these tokens.

You pass them through some neural network, as we said,

it's a transformer.

Then you get a representation for all the word

and all the words in the context.

So it's basically a representation

of the entire sentence.

You pass it through a linear layer,

as you just said, to basically map it to the number

so that the output-- the number of outputs

is the number of tokens.

You then pass it through some softmax

and you basically get a probability distribution

over the next words given every word in the context.

And the last that you use is basically--

it's essentially a task of classifying the next token.

So it's a very simple, kind of, machine learning task.

So you use the cross-entropy loss.

Where you basically look at the actual target that happened,

which is the target distribution, which

is a one hot encoding, which in this case says,

I saw the real word that happened is cat.

So that's a one hot distribution over cat.

And here this is the actual--

do you see my mouse?

Oh, yeah.

This is the distribution that you generated.

And basically you do cross entropy,

which really just increases the probability of generating cat

and decreases all the probability of generating

all the other tokens.

One thing to notice is that, as you all know again,

this is just equivalent to maximizing the text log

likelihood because you can just rewrite

the max over the probability of this autoregressive language

modeling task as just being this minimum of I just

added the log here and minus, which

is just the minimum of the loss, which is the cross entropy loss.

So basically minimizing the loss is

the same thing as maximizing the likelihood of your text.

Any question?

Questions?

OK, tokenizer.

So this is one thing that people usually

don't talk that much about.

Tokenizers are extremely important.

So it's really important that you understand at least what

they do at a high level.

So why do we need tokenizers in the first place?

First, it's more general than words.

So one simple thing that you might think

is we're just going to take every word that we will have.

You just say every word is a token in its own.

But then what happens is if there's a typo in your word?

Then you might not have any token associated

with this word with a typo.

And then you don't know how to actually pass

this word with a typo into a large language model.

So what do you do next?

And also, even if you think about words, words is a very--

words are fine with Latin-based languages.

But if you think about a language like Thai,

you won't have a simple way of tokenizing

by spaces because there are no spaces between words.

So really, tokens are much more general than words.

It's the first thing.

Second thing that you might think

is that you might tokenize every sentence, character

by character.

You might say A is one token, B is another token.

That would actually work and probably very well.

The issue is that then your sequence becomes super long.

And as you probably remember from the lecture

on transformers, the complexity grows quadratically

with the length of sequences.

So you really don't want to have a super-long sequence.

So tokenizers basically try to deal with those two problems

and give common subsequences a certain token.

And usually how you should be thinking about it is around

an average of every token is around 3-4 letters.

And there are many algorithms for tokenization.

I'll just talk about one of them to give you a high level, which

is what we call Byte Pair Encoding, which is actually

a pretty common.

One of the two most common tokenizers.

And the way that you train a tokenizer

is that first you start with a very large corpus of text.

And here, I'm really not talking about training a large language

model yet, this is purely for the tokenization step.

So this is my large corpus of text with these five words.

And then you associate every character

in this corpus of text a different token.

So here, I just split it up every character

with a different token, and I just

color coded all of those tokens.

And then what you do is that you go through your text,

and every time you see pairs of tokens that are very common,

the most common pair of token, you just merge them.

So here you see three times the tokens t and o

next to each other.

So you're just going to say this is a new token.

And then you continue, you repeat that.

So now you have tok, tok which happens three times.

Toke with an E that happens 2 times and token,

which happens twice, and then ex which also happens twice.

So this is the-- if you were to train a tokenizer on this corpus

of text, which is very small, that's

how you would finish with a token--

with like trained tokenizer.

In reality, you do it on much larger corpus of text.

And this is the real tokenizer of--

actually, I think this is GPT3 or ChatGPT.

And here you see how it would actually separate these words.

So basically you see the same thing

as what we gave in the previous example.

Token becomes its own token.

So tokenizer is actually split it up

into two tokens token and -izer.

So yeah, that's all about tokenizers.

Any questions on that?

Yeah.

How do you deal with spaces, and how do you

deal with [INAUDIBLE].

Yeah so actually there's a step before tokenizers,

which is what we call pre-tokenizers, which

is exactly what you just said.

So this is mostly--

in theory, there's no reason to deal with spaces and punctuation

separately.

You could just say every space gets its own token,

every punctuation gets its own token,

and you can just do all the merging.

The problem is that-- so there's an efficiency question.

Actually, training these tokenizers takes a long time.

So you better-- because you have to consider every pair of token.

So what you end up doing is saying if there's a space,

this is very-- like pre-tokenizers

are very English specific.

You say if there's a space, we're

not going to start looking at the token that came before

and the token that came afterwards.

So you're not merging in between spaces.

But this is just like a computational optimization.

You could theoretically just deal with it

the same way as you deal with any other character.

And--

Yeah.

When you merge tokens to delete the tokens that you merged away

or do you keep the smaller tokens that emerge?

You actually keep the smaller tokens.

I mean, in reality, it doesn't matter much because usually

on a large corpus of text, you will have actually everything.

But you usually keep the small ones.

And the reason why you want to do that

is because if-- in case there's, as we said before, you have

some grammatical mistakes or some typos,

you still want to be able to represent

these words by character.

So, yeah.

Yes.

Are the tokens unique?

So I mean, say in this case T-O-K-E-N is there only one

occurrence or could--

do you need to leave multiple occurrence so they could have--

take on different meanings or something?

Oh I see what you say.

No, it's every token has its own unique ID.

So a usual-- this is a great question.

For example, if you think about a bank, which

could be bank for like money or bank like water,

it will have the same token.

But the model will learn, the transformer

will learn that based on the words that are around it,

it should associate that--

I'm saying-- I'm being very handwavy here,

but associate that with a representation that

is either more like the bank money side or the bank water

side.

But that's a transformer that does that.

It's not a tokenizer.

Yes.

Yes.

So you mentioned during tokenization,

keep the smaller tokens you started with, right.

Like if you start with a T you keep the T

and then you build your tokenize out to

[INAUDIBLE] allow input token.

So let's say maybe you didn't train on token, but in your data

you are trying to encode token.

So how does the tokenizer know to encode it with token or to

[INAUDIBLE]?

Yeah.

The great question.

You basically when you-- so when you tokenize,

so that's after training of the tokenizer

when you actually apply the tokenizer

you basically always choose the largest token

that you can apply.

So if you can do token, you will never do T,

you will always do token.

But there's actually-- so people don't usually

talk that much about tokenizers, but there's

a lot of computational benefits or computational tricks

that you can do for making these things faster.

So I really don't think we-- and honestly, I

think a lot of people think that we should just get away

from tokenizers and just kind of tokenize character

by character or bytes by bytes.

But as I said, right now there's this issue of length,

but maybe one day, like in five or 10 years,

we will have different architectures

that don't scale quadratically with the length of the sequence.

And maybe we'll move away from tokenizers.

So can you share with us the drawback?

Why do people want to move away from the tokenizer?

Yeah.

So I think one good example is math.

If you think about math, actually numbers right now

are not tokenized.

So for example, 327 might have its own token, which

means that models, when they see numbers,

they don't see them the same way as we do.

And this is very annoying because I mean,

the reason why we can generalize with math

is because we can deal with every letter separately

and we can then do composition.

Where you know that basically if you add stuff,

it's the same thing as adding every one separately

plus like whatever the unit that you add.

So they can't do that.

So then you have to do special tokenization.

And, like, one of the big changes that GPT4 did

is changing the way that they tokenize code.

So for example, if you have code, you know you have often,

in Python, these four spaces at the beginning.

Those were dealt with strangely before.

And as a result, like, the model couldn't really

understand how to deal with code.

So tokenize actually matter a lot.

OK, so I'll move on right now, but we can come back later

on tokenizers.

Great.

So we talked about a task the loss the tokenizer,

let's talk a little bit about evaluation.

So the way that LLMs are usually evaluated

is what we call-- is using what we call perplexity.

At a high level it's basically just your validation loss.

The slight difference with perplexity

is that we use something that is slightly more interpretable,

which is that we use the average per token loss,

and then you exponentiate it.

And the reason why you exponentiate it

is because you want--

I mean, the loss has a log inside and you--

like one humans are actually pretty

bad at thinking in log space.

But two logs depend on the base of the log

while when you exponentiate you basically have everything

in the vocabulary size unit.

And the average per token is just so

that your perplexity is independent of the length

of your sequence.

So perplexity is just two to the power average

of the loss of the sequence.

So perplexity is between one and the length of the vocabulary

of your tokenizer.

One it's simply well, if you predict perfectly

the thing which every word, then every word

will have basically products of ones.

So the best perplexity you can have is one.

If you really have no idea, you basically

predict with one divided by size of vocabulary

and then you do simple math and you basically

get perplexity of size of vocabulary.

So the intuition of perplexity is

that it's basically the number of tokens

that your model is, kind of, hesitating between.

So if your model is perfect, it doesn't hesitate.

It know exactly the word.

If it really has no idea, then it

hesitates between all of the vocabulary.

So perplexity really improved.

That's perplexity on a standard data set between 2017 and 2023.

It went from a kind of 70 tokens to less than 10 tokens

over these five, six years.

So that means that the models were previously

stated between 70 words every time it was generating a word,

and now it's hesitating between less than 10 words.

So that's much better.

Perplexity is actually not used anymore

in academic benchmarking, mostly because it depends

on the tokenizer that you use.

It depends on the actual data that people are evaluating on.

But it's still very important for development of LLMs.

So when you actually train your own LLM people

will still really look at the perplexity.

One common other way and now more common in academia

of evaluating these LLMs is just by taking all the classical NLP

benchmarks, and I'll give you a few examples later and just,

kind of, aggregating everything.

So collect as many automatically evaluatable benchmarks

and just evaluate across all of them.

So one such-- or actually two such

benchmarks are what we call HELM, which is from Stanford.

And another one is the Hugging Face open leaderboard,

which are probably the two most common ones right now.

So just to give you an idea, in HELM,

all of these type of tasks, which

are mostly things that can be easily evaluated

like question answering.

So think about many different question answering tasks.

And the benefit with question answering

is that you usually know what is the real answer.

So you can-- the way that you evaluate these models

and I'll give you a concrete example in one second,

is that you can just look at how likely the language model is

to generate the real answer compared to some other answers.

And that's essentially, at a high level,

how you evaluate these models.

So to give you a specific example,

MMLU is probably the most common academic benchmark for LLMs.

And this is just a collection of many question

and answers in all of those domains.

For example, college medicine, college physics,

astronomy and these type of topics.

And the questions are things like, so this is in astronomy.

What is true for type-1a supernova?

Then you give four different potential answers

and you just ask the model which one is more likely.

So there are many different ways of doing it.

Either you can look at the likelihood of generating

all these answers, or you can ask the model

which one is the most likely.

So there are different ways that you can prompt the model,

but at a high level, you know which one is correct.

And there are three other mistakes.

Yes.

Creating unconstrained text as an output.

Yeah.

How do you evaluate a model if it

gives something that's semantically completely

identical, but is not the exact tokens that you expect?

Yeah.

So that's a great question.

I'll talk more about that later.

Here, in this case, we don't do unconstrained.

So the way you would evaluate MMLU is basically either

you ask the first question, and then you

look at the likelihood of the model generating A,

the likelihood of the model generating B, C, and D

and you look at which one is the most likely.

Or you can ask the model out of A, B, C, D,

which one is the most likely.

And you look at whether the most likely next token is A, B,

C, or D. So you constrain the model

to say it can only answer these four things.

You say you constraint--

Yeah.

You constrain the prompt or do you

mean of its whole probability distribution

that it outputs you only comparing

the outputs of like-- you're only comparing the A token the

[INAUDIBLE].

Yeah.

So in the second case I gave you, you would do exactly the--

actually would do both.

You would prompt the model saying A, B, C, or D

plus you would constrain to only look at these four tokens.

In the first case, you don't even need to generate anything.

So in the first case, you literally just

look, given it's a language model,

it can give a distribution over sentences.

You just look at what is the likelihood of generating

all of these words?

What is the likelihood of generating the second choice?

And you just look at whether the most likely sentence is actually

the real answer.

So you don't actually sample from it,

you really just use P of X1 to XL.

Does that make sense?

That being said, evaluation of open-ended questions

is something we're going to talk about later,

and it's actually really important

and really challenging.

Yes.

Earlier you mentioned [INAUDIBLE] metrics

like perplexity are not I usually

use because it depends on how you do

your tokenization, some design choices.

I was wondering if you could speak more to that.

Yeah.

So think about perplexity.

I told you perplexity is between 1 and vocabulary size.

So now imagine that ChatGPT uses a tokenizer that has 10,000

tokens but Gemini from Google uses a tokenizer that had

100,000 potential tokens.

Then actually the Gemini one will have the upper bound

of the perplexity that you can get is actually worse for Gemini

than for ChatGPT.

Does that make sense?

So that's just an idea.

It's actually a little bit more complicated than that,

but that's just one festival with a bit

of where you can see that the tokenizer actually matters.

Great.

OK, so evaluation challenges.

There are many.

I'll just talk about two really briefly.

One, as I told you, there are two ways of doing evaluation

for these MMLUs.

Actually, there are many more than two

but I gave you two examples.

And it happens that for a long time,

even though that was a very classical benchmark

that everyone uses actually different companies

and different organizations were actually

using different ways of evaluating MMLU.

And as a result, you get completely different results.

For example, Llama-65b, which was the first model of meta

in the llama series, had on HELM 63.7 accuracy

but on this other benchmark had like 48.8.

So really the way that you evaluate, and this is not even

talking about prompting this is really just the way

that you evaluate the models.

Prompting is another issue.

So really, there are a lot of inconsistencies.

It's not as easy as it looks.

First thing.

Yeah, sorry.

How can we make sure that all these models

are trained on the benchmark?

Second thing.

This is a great question.

Train test contamination.

This is something which I would say

is really important in academia in--

given that the talk is mostly about training large language

models, for companies, it's maybe not that important

because they know what they trained on.

For us, we have no idea.

So, for us, it's a real problem.

So there are many different ways of trying

to test whether the test set--

or sorry, whether the test set was actually

in the training set.

One, kind of, cute trick that people in the lab,

in [? Tatsuo's ?] lab have found, is that what you can do

is that given that most of the data set online

are not randomized, you can just look at--

and that language models, what they do is just

predict the next word.

You can just look at the entire test set.

What if you generate all the examples

in order versus all the examples in a different order.

And if it's more likely to generate a thing in order, given

that there's no real order there,

then it means that probably it was in the training set.

Does that make sense?

So there are many-- that's like one of them.

There are many other ways of doing it.

Train test contamination, again, not

that important for development, really important for

academic benchmarking.

Great.

So there are many other challenges,

but I'll move on for now.

Great.

Data.

So data is another really big topic.

At a high level people just say you basically

train large language models on all of internet.

What does that even mean?

So people sometimes say, well, of clean internet,

which is even less defined.

So internet is very dirty and really not representative

of what we want in practice.

If I download a random website right now,

you would be shocked at what is in there.

It's definitely not your Wikipedia.

So I'll go really briefly on what people do.

I can answer some questions, but I mean,

data is on its own it's a huge topic.

Basically, first what you do is download all of internet.

What that means is that you use web crawlers that

will go on every web page, on internet or every web page that

is on Google.

And that is around 250 billion pages right now.

And that's around 1 petabyte of data.

So this is actually a Common Crawl is one web crawler.

So people don't usually write their own web crawlers

what they do is that they use standard web crawlers,

and Common Crawl is one of them that basically every month adds

all the new websites that were added on internet that are found

by Google, and they put it in a big basically a big data set.

So that's-- on Common Crawl, you have around 250 billion pages

right now.

So 1E6 gigabytes of data.

Once you have this--

so this is a random web page.

Like literally random from this Common Crawl.

And what you see is that one, it really

doesn't look at type of things that you would usually see,

but actually-- so this is an HTML page.

It's hard to see, but if you look through

will see some content.

For example, here, Test King World

is your ultimate source for the system x high performance

server.

And then you have three dots.

So you don't even-- the sentence is not even finished.

That's how random internet looks like.

So, of course, it's not that useful

if you just train a large language model

to generate things like this.

So what are some of the steps that are needed?

First one, you extract the text from the HTML.

So that's what I just tried to do by looking

at basically the correct tags.

There are a lot of challenges through this.

For example, extracting math is actually

very complicated, but pretty important for training

large language models.

Or for example, boilerplates.

A lot of your forums will have the same type of headers,

the same type of footers.

You don't want to repeat all of this in your data,

and then you will filter undesirable content.

So not safe for work, harmful content, PII.

So usually every company has basically

a blacklist of websites that they don't

want to train their models on.

That blacklist is very long and you basically

say if it comes from there, we don't train on this.

There are other ways of doing these things.

Is that you can train a small model for classifying what

is PII, removing these things.

It's hard.

Every point here that I'm going to show you

is a hard amount of work, but I'm just

going to go quickly through it.

So filter undesirable content.

Second or fourth is de-duplication.

As I said, you might have things like headers and footers

in forums that are always the same.

You want to remove that.

Another thing that you might have

is a lot of URLs that are different, but actually show

the same website.

And you might also have a lot of paragraphs that come from common

books that are basically de-duplicated 1,000 times

or 10,000 times on internet.

So you have to de-duplicated.

Also very challenging because you have to do that at scale.

Once you do the de-duplication, you

will do some heuristic filtering.

You will try to remove low-quality documents.

The way you do that are things like rules-based filtering.

For example, if you see that there are some outlier tokens.

If the distribution of tokens in the website

is very different than the usual distribution of tokens,

then it's probably some outlier.

If you see that the length of the words in this website

is super long, there's something strange going on that website.

If you see that the website has only three words,

maybe, is it worth training on it.

Maybe not.

If it has 10 million words, maybe there's something also

wrong going on that page.

So a lot of rules like this.

Yes.

Why do we filter out undesirable content

from our data set instead of putting it in as,

like, a supervised loss?

Can we not just say, here's this like, hate speech website,

let's actively try to--

let's actively penalize the model for getting it.

We'll do exactly that, but not at this step.

That's why the post-training will come from.

Pretraining the idea is just to say

I want to model, kind of, how humans speak, essentially.

And I want to remove all these headers, footers

and menus and things like this.

But it's a very good idea that you just had.

And that's exactly what we'll do later.

Next step, model-based filtering.

So once you filter a lot of data, what you will do--

that's actually a very cute trick.

You will take all of Wikipedia and you

will look at all the links that are

linked through Wikipedia pages.

Because probably if something is referenced by Wikipedia,

it's probably some high-quality website.

And you will train a classifier to predict whether something

comes from-- whether a document comes from one

of these references from Wikipedia

or whether it's from the random web.

And you will try to basically say,

I want more of the things that come from Wikipedia references.

Does that make sense?

So yeah.

So you will train a machine learning model.

Usually also very simple models because you

need to do that really at scale.

I mean, just think about the 250 billion pages.

Next one, you will try to classify your data

into different domains.

You will say, OK, this is entertainment, this is books,

this is code, this is like these type of domains.

And then you will try to either up or down weight

some of the domains.

For example, you might say--

you might see that actually if you train more on code, then

actually your model becomes better on reasoning.

So that's something that people usually say in

a very hand-wavy way.

If you train your model more on code,

actually it helps reasoning.

So you want to update the coding distribution

because that helps for general language modeling skills.

Books is usually also another one that people usually update.

Entertainment, they usually down weight.

So things like this.

Of course, you want to do it-- so people used to do it, maybe

kind of heuristically.

Now there's entire pipelines that we'll

talk about of how to do these things slightly

more automatically.

And then at the end of training, you usually train--

after training on all of this data that we saw

you usually train on very high quality data

at the end of training your large language model where you

decrease your learning rate.

And that basically means that you're,

kind of, overfitting your model on a very high quality data.

So usually what you do there is Wikipedia.

You basically overfit on Wikipedia

and you overfit on, like, human data that was collected.

The other thing is like continual pretraining

for getting longer context.

I'm going to skip over all of these things.

But that's just to give you a sense of how hard it

is when people just say I'm going to train on internet,

that's a lot of work.

And, really, we haven't figured it out yet.

So collecting well data is a huge part

of practical, large language model.

Some might say that it's actually the key.

Yes.

[INAUDIBLE] about data.

So basic question.

So usually when you start with like a petabyte of data,

after you go through all the steps,

what's the typical amount of data you have remaining.

And then how large a team does it typically

take to go through all the data steps you talked about?

Sorry how la-- is your question how large

is the data after you filter?

Yeah.

After you filter and then you go through all the steps.

How large a team do you need to go through, like,

all the filtration steps you mentioned.

How slow is it or--

How many people would you need to be

able to do this [INAUDIBLE]?

OK that's a great question.

I'm going to somewhat answer about the data.

How large is the data set at the end of this slide.

For number of people that work on it, that's a good question.

I'm actually not quite sure, but I would say, yeah,

I actually don't quite know but I

would say it's probably even bigger than the number of people

that work on the tuning of the pretraining of the model.

So the data is bigger than the modeling aspect.

Yeah, I don't think I have a good sense.

I would say probably in LLAMA's team, which have 70-ish people,

I would say maybe 15 work on data.

Yeah.

All these things, you don't need that many people,

you need a lot of compute also.

Because for data you need a lot of CPUs.

So, yeah.

And I'll answer the second question

at the end of this slide.

So as I just, kind of, alluded to really,

we haven't solved data at all for pretraining.

So there's a lot of research that has to be done.

First, how do you process these things super efficiently?

Second, how do you balance kind of all

of these different domains?

Can you do synthetic data generation?

That's actually a big one right now.

And because we don't have--

we'll talk about that later, but we don't have

enough data on the internet.

Can you use multimodal data instead of just text data?

And how does that improve even your text performance?

There's a lot of secrecy because, really, this

is the key of most of the pretraining large language

models.

So for competitive dynamics, usually these companies

don't talk about how they do the data collection.

And also there's a copyright liability issue.

They definitely don't want to tell you

that they've trained on books even though they did

because if not can sue them.

Common academic benchmarks.

So that will, kind of, answer what you asked.

It started-- so those are the smaller ones.

The names are not that important,

but it started from around $150 billion tokens, which are

around 800 gigabytes of data.

And now it's around 15 trillion--

15 trillion tokens, which is also

the size of the models that are-- right now the best models

are probably trained on that amount of data.

So 15 trillion tokens, which is probably,

I guess, two orders of magnitude bigger than that.

So 80E3 gigabyte.

So that would be around 100 to 1,000 times filtering

of the Common Crawl, if I'm not mistaken.

So, yeah.

One very famous one is the Pile.

So this is an academic benchmark, the Pile.

And we can just look at what distribution of data they have.

It's things like archive, PubMed Central,

which is all the biology stuff.

Here it's Wikipedia, you see Stack Exchange, some GitHub

and some books and things like this.

Again, this is on the smaller side.

So this is-- if we look at here, this is on 280B so, in reality,

it's like 100 times bigger so you cannot have that much

of GitHub and of Wikipedia.

In terms of closed source models.

Just to give you an idea, Llama 2

it was trained on 2 trillion tokens,

Llama 3 15 trillion tokens, which is currently

the best model that we know on how much it was trained on,

which is the same thing as is the best academic or the biggest

academic benchmark, which is 15 trillion tokens.

GPT4 we don't really but it's probably

in the same order of magnitude or it's probably around that.

Actually, it's probably around 13 from leaks.

If the leaks are true.

Great.

So scaling laws.

Any other questions on data before we go to scaling laws?

Sorry I know I'm giving you a lot of information,

but there's a lot into training, large language models.

Great scaling laws.

So the idea is that what people saw around 2020, or at least

from a long time, but they've been able to theoretically show

it or empirically show it since 2020,

is that the more data you train your models on

and the larger the models, the better the performance.

This is actually pretty different than what

you've seen in this class.

In this class we teach you about overfitting.

Overfitting doesn't happen with large language models.

Larger models, better performance.

It's something that really took a long time

for the community who took this type of class to realize.

But for the exam, overfitting exists.

So, OK, the idea of scaling loss is that if-- given that more

data and larger models will always

give you better performance, can we

predict how much better your performance will

be if you increase the amount of data and the size of your model?

And surprisingly, it works.

So here you see three plots from a very famous paper called

Scaling Laws from OpenAI.

Here you see on the x-axis compute.

So how much did you train--

like, how much compute did you spend for training?

And here you see test loss.

So this is essentially, I mean, perplexity,

but it's your validation loss.

So it's a log of the perplexity.

And if you put these two on log scale,

then you see that the performance or the--

sorry, the scaling law is linear.

That means that if you increase your compute

by a certain amount, you can say by how much your test loss will

actually decrease.

Same thing with data and same thing for parameters.

If you increase the data set size,

your loss will decrease by an amount

that is somewhat predictable.

If you increase the number of parameters,

the loss will decrease by an amount,

which is somewhat predictable.

This is really amazing.

Very surprising.

I mean, it looks innocuous when you look at these type of plots,

but that's crazy because it means that you can predict

how well we're going to perform in two or three years,

depending on how much compute we will add,

assuming that these things will hold.

There's nothing theoretical about it.

Yes.

Two things.

One, what is the loss that they're using here.

Is this perplexity?

So it's-- I said perplexity was like 2 to the power of the loss.

So this is the power of the perplexity.

And then the second thing is, when

you increase the number of parameters

or you increase the data set size [INAUDIBLE] data

[INAUDIBLE] times, doesn't that just inherently

increase your compute?

Like does all of this [INAUDIBLE] come to just how

[INAUDIBLE] you [INAUDIBLE]?

Yes.

--or something specific [INAUDIBLE]?

No, this is a great question.

So the compute here is actually a factor of two things, the data

and the parameter.

What I'm showing here is that you can--

well, actually, we're going to talk about that in details.

But basically, if you increase the number of parameters,

you should increase the number of data that you have.

So you actually don't go multiple times

to the same data set.

No one does epochs in at least not yet

because we haven't still kind of enough data.

So yeah, this is all the same trend,

which is increase compute decrease loss.

Yes.

Have we seen the numbers for the last two years or this

is still holding?

It is still holding.

I don't have good numbers to show you,

but it is still holding, surprisingly.

Yes.

Is there no evidence that control quality density

will ever plateau?

In theory, we would expect it plateau, [INAUDIBLE]?

No empirical evidence of plateauing anytime soon.

Why?

We don't know.

Will it happen?

Probably.

I mean, it doesn't need to because it's actually

in log scale.

So it's not like as if it had to go.

It had to plateau.

Like mathematically, it could continue decreasing like this.

I mean, most people think that it will probably

plateau at some point.

We don't know when.

So that's-- I'll talk more about scaling laws now.

So why are scaling laws really cool?

Imagine that I gave you--

you're very fortunate I gave you 10,000 GPUs for this month.

What model will you train?

How do you even go about answering that question?

And I mean, this is a hypothetical,

but that's exactly what these companies are faced with.

The old pipeline, which was basically

tune hyperparameters on the big models.

So let's say I have 30 days, I will train

30 models for one day each.

I will pick the best one and that will be the final model

that I will use in production.

That means that the model that I actually used

was only trained for one day.

The new pipeline is that you first find a scaling recipe.

So you find something that tells you, for example,

like one common thing is that if you increase

the size of your model, you should decrease your learning

rate.

So you find a scaling recipe such

that you know if I increase the size of my model,

here's what I should do with some hyperparameters.

Then you tune your hyperparameters

on smaller models of different sizes.

Let's say I will say for three days, of my 30 days,

I will train many different models.

And I will do hyperparameter tuning

on these small models, each of different sizes.

Then I will fit a scaling law and try

to extrapolate from these smaller models, which

one will be the best if I train it for much longer--

or sorry if I train it for a larger model.

And then I will train the final huge model

for 27 days instead of just one day.

So the new pipeline is not train things

or do hyperparameter tuning on the real scale of the model

that you're going to use in practice,

but do things on smaller ones at different scales.

Try to predict how well they will perform

once you make them bigger.

I will give-- I will give you a very concrete example right now.

Let's say transformers versus LSTMs.

Let's say you have these 10,000 GPUs,

you are not sure which one you should be using.

Should I be using a transformer-based model

or LSTM-based model.

What I will do is I will train transformers

at different scales.

So here you see different parameters on the x-axis,

y-axis is my test source.

I will then train different LSTMs at different scales.

Once I have these points, I will see oh it, kind of,

fits a scaling law.

I will fit my scaling law and then

I will be able to predict if I had 10 times more compute,

here's how well I would perform for the LSTM.

It's actually slightly less linear for the LSTM,

but you can probably try to predict where you would end up.

And clearly from this plot, you would see

that transformers are better.

One thing to notice when you read these type of scaling laws

is that there are two things that are important.

One is really your scaling rate, which

is the slope of the-- the slope of the scaling law.

The other thing is your intercept,

you could start worse, but actually

become better over time.

It just happens that LSTMs are worse for both.

But I could show you another one where things--

you can predict that actually after a certain scale

you're better off using that type of model than others.

So that's why scaling laws are actually really useful.

Any questions on that?

Yeah.

So these are all, kind of, very--

how sensitive are these to small differences in the architecture.

Like one like transformer architecture

versus another transformer architecture.

Do you think we have to fit your own curve

and, basically, say like oh scaling laws tell me this should

be some logarithmic function.

Like, let me extrapolate that for

my own specific architecture.

Yeah, so usually, for example, if you're an academic

and you want to-- now at least that's pretty recent

and you want to propose a new activation.

That's exactly what you will do.

You will fit a scaling law, show another scaling law

with the standard like, I don't GELU

and you will say that it's better.

In reality, once you start thinking about it in scaling

laws terms, you really realize that actually

all the architecture differences that we

can make, like the small, minor ones, all they do

is maybe change a little bit the intercept.

But really that doesn't matter because just

train it for 10 hours longer or like wait for the next computer

GPUs and these things are really secondary.

Which is exactly why I was telling you originally,

people spend too much time on the architecture and losses.

In reality, these things don't matter as much.

Data though.

If you use good data, you will have much better scaling laws

than if you use bad data.

So that really matters.

Another really cool thing you can do with scaling laws

is that you can ask yourself, how to optimally allocate

training resources.

Should I train larger models.

Because we saw that it's better when you train larger models,

but we saw that it's also better when you use more data.

So which one should I do?

Should I just train on more data, a smaller model,

or should I train a larger model on less data?

So Chinchilla is a very famous paper that first showed this.

The way they did it, I want to give you

a little bit of a sense of what these plots are.

Here you see training loss again on the x-axis,

you see parameter differences, sorry, parameter size--

number of parameters.

So the size of the model.

And here all these curves are what

we call ISO flops, which is that all the models on this curve

have been trained with the same amount of compute.

The way that you do that is that you train--

you change.

Sorry, you vary the number of tokens that were trained on

and the size of the models, but you vary in such a way

that the total compute is constant, OK.

So all these curves that you see with different colors

have different amount of compute that were trained on.

Then you take the best one for each of those curves.

Once you have the best one for each of those curves,

you can ask-- you can plot how much flops it was

and which curve were you on and how much parameters

did you actually use for training that specific point.

You put that on the log log scale again and now

you fit a scaling law again.

So now I have something which tells me

if I want to train a model of 10 to the power 23 flops, here is

exactly the number of parameters that I should be using.

100 B.

And you can do the same thing with flops and tokens.

So now you can predict--

if I tell you exactly I have one month of compute,

what size of model should I be training?

Fit the scaling law, and I tell you.

Of course that all looks beautiful.

In reality like there's a lot of small things of like,

should you be counting, like, embedding parameters,

there's a lot of complexities.

But if you do things well, these things actually do hold.

So the optimal number of parameters that Chinchilla paper

have found is to use 20 tokens for every parameter

that you train.

So if you add one more parameter,

you should train your thing on-- your model on 20 more tokens.

So one caveat here is that this is optimal training resources.

So that is telling me if you have 10 to the power, 23 flops

or if you have 100, I don't know how much that is, $100 million

or 10-- no, that's much less, actually.

Let's say I have $5 million to train

my best model that gets the lowest

loss what would I train on?

In reality, these companies need to think about inference also.

If you have a smaller model, they will spend less over time.

So actually, if you consider the inference cost,

you have other papers that try to show that, it's

around 150 parameters, sorry--

tokens per parameters, because you prefer having a smaller

model because over time you're going

to actually spend less money on inference of these models.

So 150 to 1, that's around what the best models are trained

on right now, at least the ones that are

used in practice in production.

Great.

Any questions on Chinchilla?

Great.

Oh sorry.

In practice, how expensive is inference for these models

relative to training?

Actually, very expensive.

I will not talk about inference because that would

be another entire lecture.

But just think about ChatGPT where

they have I don't know how much it is now,

like 600 million people that use it.

Like, that's a lot.

Yeah.

So it's actually very expensive.

There's a lot of optimization you can do for inference though.

And that's an entire other lecture.

I'm going to skip that this time, but it's very interesting.

OK tunings.

As I said, there are many things that you

can answer with scaling laws.

I just try to give you two examples,

but really there are many things.

What data do you use.

What mixture-- what data mixing weighting you use.

The mixtures, that's what we talked about before.

What architecture you use, whether you should make

your models wider or deeper?

Should you be paying for more GPUs

or actually collecting more data?

All these things are things you can try

to answer with scaling laws.

One thing I want to say is the bitter lesson.

If you ever heard of Richard Sutton,

very famous blog post in 2019, what he realized,

which I think not enough people realize,

I didn't-- definitely did not realize at that time,

is that once you see these type of scaling laws you know that

the more compute you have, the better models you will get.

So with scale, you will get better model.

And you also know by Moore's law or these type

of variants of Moore's law that you will always

have better compute.

Then the only thing that matters is just

to have architectures that can leverage computation.

So what matters is basically systems data and less

so the architecture, like the small architecture

differences like, your activation and things like this.

So I think that's one of the reasons why most of research

focuses on some things that for industry matters less.

And I was one of those researchers

for a large part of my career.

So don't spend time over complicating.

Do the simple things, do it well.

See all them.

That's really what OpenAI taught us with ChatGPT and with all

the GPTs before.

OK, I want to give you some back of the envelope computation.

So I might be off by a few factors here,

but I just want to give you a sense of how costly it is

to train some of these models.

I'll give us an example.

llama3 400b which is currently the best open source model that

you can get.

It was trained on 15.6 tokens.

It has 405 billion parameters.

So just now that you know what is

like this optimal tokens per parameter, that's around 40.

So that's a little bit more than Chinchilla,

but less than this like inference optimal model.

So they went for training optimallity

Flops for this model.

So one simple way to compute flops

is 6 times the number of parameters,

times the number of data that you train on.

So if you do the simple calculation here,

it's 3.8 e25 flops.

The reason why this is important is

that if you follow it a little bit, the news,

there's an executive order from Biden that basically

says that once you have one e26 parameters, sorry, flops, then

you have special scrutiny on your models.

So they went to 2X less than that.

So they really went right below this

to not have special scrutiny.

So 3.8.

I might be off by a little bit, but it's definitely

under the 1 e26

So parameter p is parameters n is data, number of tokens.

This is just an approximation.

Yeah.

OK.

Compute and we know that they trained on 16,000 h100s and we

know the throughput they set it to.

So if you do the computation, it takes around 70 days

or 26 million GPU hours.

At least that's what my back of the envelope computation.

They actually said that they use 30 million

instead of 26 million GPU hours.

So maybe they had some challenges.

I don't really know.

But if you follow the simple computation,

it's around 70 days.

Cost.

I mean this it's hard to approximate,

but I'm just going to say it's, kind of, the rent.

Like, what if I wanted to rent H100, that many H 100

for that many days, how much will I pay?

H100 a lower bound on the renting costs of H100

is around two hours--

$2 per hour.

So if you multiply this by 26,000,000 hours,

you get $52 million.

So they probably pay less than that,

but not actually much less because all these services

that actually rent GPUs, they don't make that much money.

So it's probably slightly less, but not that much less.

Now salary I said 50 employees, 500k per year.

Yeah it's probably the right ballpark.

$25 million.

So if you put altogether around $75 million

for training this llama model.

I'm probably off by like 10 million,

but that's kind of right ballpark.

Carbon emitted.

A lot of people might ask like also the cost is not

the only thing that is important.

So I did the computation.

It's around 4000 tons of CO2 equivalent.

That is actually only 2000 return tickets

from JFK to London.

So right now carbon emitted is actually not--

I mean, it's huge, but it's not meaningful yet.

I think in maybe GPT6, GPT7, once you multiply this

by 100, that might become a real issue.

Right now it's still not, I think,

an issue in the grand scheme of things.

Next model the way you should be thinking about these models is

that every new generation, the number of flops essentially

multiplies 10x, or at least that's what they try if they

have enough energy.

And if they can buy enough GPUs.

Great.

Any question on these back of the envelope math.

No.

OK.

So now we talked about pretraining,

I wanted to also chat about systems

because now we know compute is really important so there's

a question of how do you optimize the--

how do you optimize the compute?

I will leave that for the end because I'm not

sure how much time we will have.

I think it's important, but hopefully I'll

be able to talk about it later.

It's slightly different than what we've

been talking about right now.

So I'll move on to post-training for now.

So the task of post-training, the reason why

we need to do post training is, as I told you

before, it's to make AI assistants.

So language modeling is not really the thing

that you want when you have an AI assistant.

For example, if you ask to GPT3, which

is a purely language model--

a pure language model, not a non-aligned one.

If you ask a question explain the moon landing

to a six-year-old, the completion that you would get

is something explain the theory of gravity to a six-year-old.

Because what it learned is that on internet,

if you have one question, you usually

have maybe another bullet point of other similar questions

you don't usually have question and then answer later.

This is not what you want from an AI assistant.

So how do we do this alignment, which

is this post training and making these models assistants?

So the goal of this alignment is to basically get

LLMs follow the instructions that

are given by users and maybe some designers,

kind of, desires.

So think about motivation.

You don't want the model-- like OpenAI

doesn't want the model to say stuff that is very toxic.

So here you see on the left-hand side

that when you ask a question, it actually provides a real answer.

So it's not like before the LLM.

And on the right-hand side, you see that it would--

if you ask to write a tweet describing how a certain part

of the population are evil, it will say that it cannot do that.

So that's kind of this alignment.

The background here is that basically the data

that you want for training some of these models is--

like, we know what we want.

Which is just asking humans, this is a question,

this is the answer that you want.

But the thing is that it's very expensive to collect that data,

and it's hard to find it online.

In contrast, pretraining data is not what you want,

but there's a lot of it.

So what we will do, or the main idea is simply

take a pretrained large language model

pretrained on all of internet and then just fine tune.

So you just change a little bit the weights on the type of data

that you actually want.

And hopefully given it, you already

pretrained it on all of internet,

it basically learns or knows how to speak in English

and knows standard language syntax

then you can really fine tune it with very little data.

OK, SFT.

So Supervised Fine Tuning is really exactly what I just said.

Which is the idea of fine-tuning the large language

model on basically the desired answers that

are collected from humans.

So why is it called supervised fine tuning?

Because you basically want to do language modeling on the real

answers.

So language modeling is this like next word prediction,

and that's the fine tuning part.

And then you want to do it on desired answers given by humans

so that's why we call it supervised.

So how do we collect this data?

Well, I just said it.

You just ask humans to tell you this

is a question this is the answer that you would

want from some of these models.

So this is an example.

I can't read very well on my computer,

but my kid needs to do a science--

no let's read this one.

Can you write a short introduction

about the relevance of the term monopsony?

And then it says monopsony refers to a market

structure, blah blah, blah.

And that's a human network there.

So, actually, this is Open Assistant,

which was a way to collect data online by humans.

So this type of supervised fine tuning or alignment

is really the key of ChatGPT.

This is what made the big jump from GPT 3, which was mostly

something that was known by AI researchers

to ChatGPT, which became known by basically everyone.

So the problem with human data is

that it's very slow to collect and very expensive.

So one possible simple idea is to use

LLMs to scale data collection.

So that's exactly what we did with Alpaca one year ago.

What we did is that we asked humans,

so we use a data set of human question answers.

So there were 175 question answers here,

and we asked the best model at the time,

so text-davinci 003 to basically generate many more of these

question and answers.

So all we did is, this is what humans would write now,

write similar answers and similar questions.

And we collected 52,000 LLM-generated question answers.

And then what we did is simply we took llama 7B,

which was the best pre-trained model at the time.

And we just fine tuned this with supervised fine tuning,

as I told you.

And that's how we got the Alpaca 7B model.

And this is the type of data that we collected.

So things like what does algorithm mean?

And algorithm is a step by step set of instructions

you use to solve a problem or achieve a goal, blah, blah,

blah, blah.

So the data is not actually-- it's actually pretty good,

given that it was LLM generated by LLMs from essentially two

generations ago.

So that really started at least for us

as an academic replication of ChatGPT.

Now it really-- there's a big field

of synthetic data generation of how

to use LLMs to basically make development of LLMs faster.

And basically by decreasing the amount of human hours that

you need.

Quantity of data.

So we talked about what type of data and how we collect it.

One thing which is surprising with SFT

is that you don't need that much data.

So what this paper showed this is called LIMA,

is that if you scale the amount of data that you use from

supervised fine tuning from 2000 to 32,000,

it really doesn't help much.

So here scaling laws definitely don't help.

And so the intuition here is that all you learn

is you learn how to format your desired answers.

Another way of saying it is that your pre-trained models, they

essentially model the distribution of every user

on internet, one that might write bullet points,

another one that might answer question-- answer

question with an answer.

So all you tell your model is like, wait,

you should actually be optimizing

more for this type of user than another one.

So you're not actually teaching it--

you're not teaching anything through this SFT, so

supervised fine tuning, all you do

is you tell the model to optimize for one type of user

that it saw already in a pretrained data set.

So the knowledge is already in the pretrained LLM

and you basically just specialize to one type of user.

Great.

Any question on SFT?

Yes.

So I know it's a big issue with synthetic data

where if you keep generating data from the same distribution,

eventually you're not learning a new distribution,

you're essentially playing with it.

Just bootstrapping that.

Yeah.

Surely you can't scale that forever, right.

You can't keep going on and generating

from the same distribution.

You hope to learned something new.

Yeah.

So are there-- it's an active area of research

but any thoughts that you have around

how people are maybe thinking around this and better ways

to bootstrap?

Or to give up on this idea and realize that the chart shows

you don't need that many so just get humans to generate

2000 really good prompts.

Yeah.

So that's a very good question.

So for the data stuff, so I'm saying

it's not that important for SFT, but there

will be another thing we'll talk about right after where actually

data does matter.

My intuition based on not that much empirical results

is that you can still get, even though you use your LLMs,

if you use purely LLM generated text

and you do that for like three or four generations of LLMs,

I agree with you that probably you won't improve much.

But for me what is important is how do you use human in the loop

with LLMs?

Not purely LLMs, not purely humans,

but maybe what you can do is just

have the model regenerate some new text

and just humans write a few edits.

Edits are much faster than writing the entire text.

And I think that if you have that type of collaboration,

then from an information theoretical point of view,

you still get additional information,

but you're still much faster than if you use humans.

And I think that as a field we'll

probably move towards these type of things, which is really

just finding the examples that are important and asking humans.

It's kind of active learning, just

asking humans exactly when you need to get their inputs.

Yes.

Do we train with the same loss function

and the same general training algorithm

for the supervised fine tuning bit

as we do for the pretraining?

Because the examples you showed, I

think the important thing of the good examples

is like super factually accurate.

Like there's these more complex things

and it's still just like [INAUDIBLE].

Same loss.

So that's why here--

yeah, I didn't-- maybe didn't emphasize enough.

This is just language modeling.

Fine tune the LLM with language model and the desired answers.

So this is literally the same loss.

It will be different in two seconds,

but the first step of SFT is literally

the same loss where you just say, OK, I

want to actually specialize on that type of data.

So there's even a question of what is pretraining,

what is post-training?

Because, in reality, it's just like a different data

that you use.

The reason why we usually call it post-training is that the way

we collect that data is very different.

Great, great questions.

Yes.

Maybe it's the same question, but why would

these 2000 examples have such a overweighted influence

on fine tuning?

So that's why we--

also that's another reason why we call it post-training

is that we use different type of hyperparameters.

So, I told you basically at the end

of pretraining you essentially end up

with a learning rate of 0.

Here, you're going to increase your learning rate.

So like 1e minus 5, 1e minus-- yeah.

And so the way that you give to them is actually different.

OK.

Second step or second part of this post training

is what we call reinforcement learning

from human feedback or RLHF.

Some of you might have heard of that.

The idea is that SFT has a problem, namely that you

do behavioral cloning, which means that you just try to clone

what the humans would say.

And that has many issues.

One of them is that you're bound by human abilities.

So if-- humans actually humans won't generate the things

that they think is actually the best thing to generate.

So if you ask me to write a book,

I mean, I can definitely enjoy your book.

I can probably say one book is better than another,

but I'm definitely not going to be as good as writing the book

that I want to read.

So you're going to be bound by the human ability

to generate things, even though the humans might be better

at distinguishing between things.

That's one issue.

Issue number two, I find that actually pretty interesting

is that it--

if you ever heard of the word hallucination. so this

is LLMs generating fake-- like false information.

Hallucination might-- at least people

have hypothesized that can come from the supervised fine tuning

even if you do supervised fine tuning on data that is correct.

And the reason why that is is that if--

given I told you that basically SFT is with very little data.

And it's with data that the model

doesn't learn anything new.

So what if the human gives an answer that the model didn't

know was true.

From the model perspective, the human basically

is telling the model generate this thing that seems plausible

but actually have no idea if it's true or not.

So just to give you a very concrete example,

if we go back to this monopsony example,

can you write blah blah blah about monopsony?

Imagine that the human wrote a reference on this type of book.

And that book might exist.

That might be a correct reference,

but what if the LLM never saw this reference

during pretraining.

Then it doesn't know that it's a correct reference.

So really what you tell the model

is to generate or make up some plausible sounding reference

rather than actually tell the real reference

that it saw during pretraining.

So hallucination might be caused by this SFT.

So that's problem number two.

Does that all make sense?

Great.

Problem number 3, price.

Generating the ideal answers is very pricey.

And that comes back to your question

of humans writing the entire answer is actually

pretty expensive.

So that's why RLHF comes in.

The idea is that instead of cloning the behaviors of humans,

we're going to maximize human preference.

And the way we're going to do that, so the pipeline,

is that for a certain-- for every instruction,

you're going to ask a model to generate two answers

and usually use a pretty good model.

So you usually don't use an LLM here, you use a SFT fine tune,

you use a fine tuned LLM already to give pretty good answers.

And then you ask labelers which of these two answers was better?

So select the preferred one.

And then with different types of algorithms,

we're going to talk about the algorithms, you just fine

tune the model to generate more of the green thing

than the red thing.

So more of the good stuff.

So now the question is how and we're

going to talk about that right now.

So there are two ways that we're going to talk about

and two that are mainly use in the community.

The first one is simply the idea of using reinforcement learning.

So hopefully you all know what reinforcement learning is now.

So when you think about using reinforcement learning,

one important question is like, what is the reward

that we're optimizing.

So in this case, there are really two options

that I could think about.

The first one, you could just say,

I'm going to compare the output generated by some baseline,

the output generated by my model.

And I'm just going to ask the human to say which one is better

and I'm going to use this as a reward.

So if I'm better than the baseline,

this is a plus 1, if not, it's a minus 1.

So now it's binary reward.

The problem with binary reward is that it's very sparse

and you don't get much information out of it.

Like maybe your answer was slightly better,

maybe it was like way better and you don't really

know from this how much better it was.

So option 2 is that you can train

what we call a reward model, which is simply a classifier.

So you use machine learning to classify

how much better two outputs are from the preference--

from the perspective of the human.

So this is a little bit meta, but what you basically

do is that you train--

you take a reward model, which is just a large la-- also

a large classifier, and you basically ask this reward model,

you give it the input and the actual output

that you have, one of the two outputs.

And you just exponentiate that so that's the softmax loss

that you all know about.

And now you divide by the exponentiated reward

on the first example--

I'm sorry, on the first output and this

is on the second output.

And you basically train--

so the reason why you do that is that you train your model,

you train this reward model to be

able to classify how much better one output is to another one.

So another slightly less convoluted way of saying it

is that your reward model will output

some reward that will be used as the logits of your softmax.

So now if you have high logits in your softmax,

it means that you highly likely this output is better.

So that's what we call Bradley-Terry model.

Yes.

Will this reward model [INAUDIBLE]

lower the entire output, or is it going to [INAUDIBLE]?

So this takes the entire--

yeah, this takes the entire output at once.

So it takes all the input and all the output

and it gives one number.

Yes.

So [INAUDIBLE] reward model, where would the human be then?

Sorry.

With the reward model, where would the human be?

Like--

I see.

OK sorry.

Maybe I wasn't clear.

You train this reward model to fit this green and red

preference from humans.

So basically you train a classifier

to say whether the humans prefer red or green.

But instead of using the binary reward, which

is what the human would tell you you basically use

the logits of the softmax.

And the thing with the logits is that logits are continuous.

So now you know that if your reward model said

it has high logits, then, in some ways,

the human highly preferred this answer to some other answer.

Great.

So as I just said, continuous information is better.

So that's what people use in practice or at least

used to use in practice.

I'll tell you about the other algorithm later.

So what do you do at the end is that you basically

try to just use reinforcement learning that you know about.

Now we know we have a reward.

What you sample through is the generation

from your large language model.

And then you just use some regularization term.

So the reason why we do this regularization term

is for avoiding what we call overoptimization.

So this reward model might not be

really represent-- might not perfectly

model human preferences.

So you don't want to maximize this thing

to essentially infinity.

And you do it using a PPO, which is a common reinforcement

learning algorithm.

One thing to note here, because it will be important for later,

is that when we use maximum likelihood--

sorry, now the large language models

are actually a policy for your reinforcement learning.

It's not maximizing maximum likelihood anymore.

Which means that you're not modeling any distribution

anymore.

And the reason why this is important

is that models that went through this type of PPO

actually don't give you likelihoods

of text that are meaningful.

Because what you optimize them to do

is basically just optimize for generating

the most likely thing, not optimize for modeling,

all the answers that humans might say.

Another way of saying that is that there's

nothing that incentivizes here the model to not give

a single possible generation.

Nothing here says it's good if you have some distribution

with some entropy.

If you haven't followed, it's not that important but just good

to know.

Great.

So PPO is exactly what ChatGPT did originally.

So here is on their blog post on what

they have is step one do supervised fine tuning, which

now you all know about.

Step two, train a reward model on human preferences.

Step three, do PPO multiple steps,

which is where you see this blue arrow.

So you continue-- you train the model once with the PPO,

you collect new data, you continue.

And that's why-- and that's exactly what ChatGPT did.

And that was the big breakthrough

between GPT 3 and ChatGPT.

One thing to note is that PPO has many challenges.

Reinforcement learning is something that

is super nice theoretically.

In practice, anyone who ever worked

with reinforcement learning knows it's such a mess.

There's a lot of things like rollouts, outer loops,

clipping so many complications.

So it's messy.

This is the idealized PPO used for LLM settings,

so that's already much more complicated

than this expectation we saw before.

And in practice it's actually much more complicated.

So we have one implementation of it that we had to do,

and I'm not going to go through it.

But basically have so much stuff that you

have to think about when you implement

that type of PPO algorithm.

So you have clipping everywhere, you have a lot of complexities

and things are not well documented.

All this to say that we're going to there was a new method that

was proposed also from Stanford one year ago

called DPO, which is essentially a simplification of PPO.

And the way-- what they did or the idea that they have

is that instead of using reinforcement learning,

you can just maximize the probability of generating

the stuff that you like and minimizing

the probability of the stuff that you don't like.

So if you think about the human preference, the red and green,

maximize green, minimize red.

So the loss is actually this one where what you see

this is simply some log of the model.

So this is the likelihood of a model generating the things

that the human preferred, given the inputs.

And what you try to do is basically

maximize the likelihood of generating the things that you

like, minimize the likelihood of the things that you don't like.

All the rest of the terms here it's not too important.

It's actually really not that complicated to understand.

But at a high level, it's really just maximizing the things

you like, minimizing the rest.

And one thing to note, which I was going to say just here,

is that actually all the rest is chosen such

that the global minima of PPO and the global minima

of like this DPO, under some assumptions,

are essentially equivalent.

So this is the right thing to do mathematically.

I'm not going to go through the derivations,

but that's the right thing to do.

It's pretty different with PPO in the sense that now--

with PPO, what you had to do is collect the human preferences,

then train a reward model with maximum likelihood,

then use reinforcement learning.

Now all you do is basically maximum likelihood.

Much simpler.

Yes.

I mean, yeah.

So it seems like this is A, much simpler and B, like,

what you would just intuitively do with [INAUDIBLE]?

Why did they start with this reward model.

Like what led them doing that?

I think it's a great question.

I don't really know.

What I can tell you is that.

At ChatGPT the people who did basically

this PP-- sorry, who did ChatGPT initially

are the ones who actually wrote PPO.

And I think they were just-- like,

there are a lot of reinforcement learning people.

And I think that for them it was very intuitive.

So there's also some additional potential benefits.

For example, I don't want to--

yeah, for example, if you use the reward model,

the cool thing here with reinforcement learning

is that you can use unlabeled data with the reward model.

So here you can only use the labeled data for doing DPO--

For PPO-- for PPO, you first train your reward model

and then you can use unlabeled data

where the reward model will basically

label this unlabeled data.

So this additional, kind of, potential--

there could be potential improvements.

In practice it happens that there are none.

And I think just that a lot of people in this team

were reinforcement learning experts, including

the main author of PPO, John Schulman.

So much simpler than PPO, and it's basically performs as well.

So now this is the standard thing that people use.

At least in the open source community,

I believe it's actually the standard also in industry.

So that's called DPO.

Gains so those are all the papers on the left.

Here this is on the summarization task.

You see, all I want to show you is

that basically the pretrained models were OK

and they improve of scale.

If you do supervised fine tuning,

you improve them a little bit more,

if you do PPO or something with RLHF human feedback,

you get performance that are, oftentimes

depending on a benchmark, even better than humans.

So this is the human reference summaries.

Same thing.

This is on a paper that we have Alpaca farm where

we see the evaluation here is not too important

but basically see pretrained model.

You jump to SFT and then you jump to PPO, DPO and PPO,

DPO have the exact same performance.

So basically RLHF helps.

That's, kind of, the conclusion and DPO is simple.

Data.

The way that you collect that type of data.

First idea is just use humans as we already talked about.

Guidelines are very complicated for what

humans should be labeling, and it's really not that easy.

And actually, if you ever do some of the labeling,

you will see that it's extremely complicated.

Like if I Zoom in to this.

Here, I have a question tell me about self-driving cars.

And you read both self-driving cars

are vehicles that are capable of detecting

the surroundings, blah, blah blah, blah.

Self driving cars are cars that are equipped

with sensors, blah blah, blah to navigate

without the need for a driver.

I mean, both seem OK.

Which one is better?

It's actually hard to say at a glance.

And as a result, the problem with humans

is that you will start optimizing

a lot of high-level features.

For example, the second one is longer.

I can guarantee you that most humans will choose

the second one, even though I mean,

maybe the first one is better.

I don't know.

I haven't read it carefully.

So challenges of humans.

First, slow and expensive.

Second, as I just mentioned, it's hard to focus on things

that matter, like correctness.

And people usually look at things

that don't matter as much like the form, like length.

And as a result, so what I show here

is that when you do RLHF, the more you do RLHF,

the longer the output of the models become.

So if you've ever been annoyed at ChatGPT

answering you super long sentences,

this is because of RLHF.

Annotator distribution shift.

Like the distribution of annotators

that you use matters a lot, and you have to think,

like, what is even the humans that we want

to represent in these models?

Another question is crowdsourcing ethics.

Like usually these-- basically a lot

of the labeling that is done, the people who do them

are not paid well and they have to go

through a lot of toxic data because you basically

want the model to avoid saying the toxic data.

So crowdsourcing ethics too.

So many challenges with human data.

So what we did, also last year, is again,

the same thing as Alpaca, just the idea of like oh well, there

are challenges with humans, maybe

we can just replace them with LLMs.

So what we did is simply replace--

I see that.

I'm just realizing that the slides are not centered.

Anyways you replace a human preference with preferences.

So here, on this figure, you see on the x-axis, the price

that we paid for collecting human data.

It's around $300 for 1,000 examples.

And this is on mechanical Turkers which are usually

like cheaper than maybe some of the other companies

that you could go through.

And on the y-axis, it's basically

the agreement with other humans, with the mode of other humans.

And what you see is that actually, as I told you before,

labeling is really complicated.

Humans agree with themselves only around 66%

of the time on a binary task.

And it's not that the humans are not good

here because we were five main authors on this paper.

We tried to label this data ourselves,

and we only had, like, 67 or 68% accuracy, even though we

talked-- like we talked for like three hours of how

we should be doing labeling.

But really, it's complicated.

It's not an easy task.

And here I just showed many different models.

And, basically, you see that models are much cheaper,

and they can actually get higher agreement

with the mode of humans than humans themselves.

And the reason why is because humans have a lot of variance,

models have no variance.

So there might be a little bit more biased

but have less variance.

So it works surprisingly well.

And now it's, kind of, the standard

in open source community.

I think even in industry a lot of people

use both humans and LLMs for improving

the collection of RLHF data.

And this is like-- this is the paper from last year,

but honestly, now it's more like the LLMs would be around this

agreement, and this costs around,

I would say 50 50x than humans and better agreement with human

than humans themselves.

OK.

So that gets us to evaluation of post training.

That goes back to your initial question

at the beginning of the lecture.

How do you evaluate something like ChatGPT?

The answers that GPT could give are basically unbounded.

And it's not that there's one right answer,

there are many answers that are just as good.

So there are many challenges.

One, you can't use validation loss

because one method might use PPO,

the other one might use DPO.

Validation loss is not comparable.

Second, you can't use--

sorry, perplexity.

That's the thing I told you before.

These models are not calibrated.

They don't give distributions.

They just optimize for one thing.

So you can't use perplexity for actually evaluating these type

of models once they aligned--

sorry, once they're aligned.

Third, there's a large diversity of questions

that humans might ask to these models.

Generation open QA some question answering some summarization

and all of these things.

So there's so many things you have to cover.

Then the tasks are really open ended,

so it's very hard to automate.

So that's what you were alluding to before.

So the idea is that instead of trying

to come up with really easily automated benchmarks,

it's just we're going to ask questions that users actually

ask to these models in practice.

And we're just going to ask annotators

to say between these two models, which one is better.

What's the better output.

So basically the exact same thing

as basically the data from RLHF but you

use it now for evaluation.

Yes I'm not sure I understand what

you mean by can't use perplexity not calibrated.

Like RLHF still doing like next token prediction.

So--

Why can't perplexity be used then?

So think about the optimal solution

after doing PPL is basically one model that

gives you essentially a delta.

Like basically it says that there's only one sentence

that is--

that could be generated for that question.

So now if you use it on something

that is slightly semantically differently different,

it would actually give a likelihood of 0 for that answer.

So in reality, it's not that extreme because as you say,

it's still a distribution, but it just

shows you that there's a fundamental issue

with perplexity.

Once these models are not LLMs anymore,

they were not trained, at least with PPO

they're not trained to do maximum likelihood anymore,

they were trained to be policies.

So probably the most common or the most--

yeah, the most common benchmark or the most trusted one

is what we call ChatBotArena, which is basically

go on internet, have random users on the internet,

blindly talk with two chatbots, just ask many questions,

see the two answers and rate, which one is better.

And you do that over hundreds of thousands of users and then

you get the actual preferences and you get rankings of models.

So you can go right now on ChatBotArena

and actually interact with these models.

One potential issue just to highlight

is that while people who want to do these type of things

are usually more like tech-driven or like tech savvy.

So a lot of the questions that you will ask

are more like tech stuff discussing

software errors, inquiries about AI tools

and all of these things.

So another issue is cost and speed.

If you really want to use something

like this for development process,

it will be too costly because you will need to basically pay

a lot of humans to do that.

So one simple idea is, again, as we said many times,

just use LLM instead of humans.

You probably know the drill at this point.

Steps for every instruction generate outputs

by some baseline and the model that you want to evaluate.

So here you imagine that I'm comparing an answer

from ChatGPT and from Misrule.

I'm just asking a model, another model, which one is better.

And I just basically average that out.

Yeah.

I asked ChatGPT 4, which one is better.

I averaged that out over my entire distribution,

over my entire benchmark or data set,

and that gives me a win rate.

So a win probability for one model compared to another one.

And now you can rank models.

And this is the AlpacaEval leaderboard.

So the benefits of this is that actually we

show-- we get 98% correlation with ChatBotArena.

So very high correlation with humans.

So this is yeah, comparison with correlation

with other benchmarks.

And it takes less than three minutes and less than $10

to run.

So it's pretty cheap.

And there are downsides though.

One of them is poor correlation.

So as we already saw before, LLMs prefer,

this is one spurious correlation, not many.

I'll just talk about one.

LLMs prefer longer outputs.

Actually humans also prefer longer outputs.

But the problem or the issue once you use LLMs

is that once there is bias, you will continue optimizing that.

Humans at some point, I can guarantee you

if I ask a simple question, and you give me

five pages of answers, I'll be like,

no, I don't like that answer.

But LLMs if they have this bias and they were trained for that,

they will continue preferring longer outputs.

So here we see the preference just showing

that humans and models prefer longer outputs.

And here is another view of the initial AlpacaEval data set

benchmark, where when we asked--

when we rank GPT4, when we look at the win rate of GPT4

versus actually GPT4 itself, if we use the standard GPT4,

it gets 50%, kind of, by definition because we're

comparing GPT4 versus GPT4.

But if we ask a GPT4 to be slightly more verbose,

so we just say in the prompt, be verbose in your answers,

then it gets a win rate of 64.4%.

So really there's a huge variance.

And if we ask it to be concise, it

gets 20% so there's a huge variance

depending on whether you ask it to be concise or verbose.

That's very annoying.

So one possible solution, which is what we did,

is just use some regression analysis.

I'm not going to go into details,

but basically use causal inference

tools to control for length.

And right now actually length matters much less.

So if you ask it to be verbose, you still get some gains,

but much less.

Great.

So that's all about post training.

And now for the next eight minutes,

I might talk about systems or just answer questions.

Yes.

Can you go back to your post training, internal post

training.

How did we tune those parameters using

the small body of fine-tuning data

and have such big effect on the model?

You mentioned earlier that there's a different set

of hyperparameters.

Are we changing just some of the weights, the later weights

or other weights.

What's actually happening?

Yeah.

Yeah, I, kind of, skimmed through all of this.

You change all the weights.

Actually, industry will change all the weights.

In open source land, you might have

heard of Laura, which is going to change basically only

some of the weights or it actually, to be more specific,

it's going to add some differences

to the output of every layer.

But in industry, you're going to just fine tune all the weights.

And also to say something else about the data, actually,

this last step, RLHF you usually going

to collect a lot more data than with SFT.

So if FSFT is like 5,000, 10,000, maybe 50,000 with,

RLHF I think you're going to be more around like the one million

order of magnitude.

It's still much less than pretraining though.

Yeah.

Because pretraining is 15 trillion tokens.

I mean, this is like-- that's not even a drop

and yet you influence the weight a lot.

So because you do it--

I mean, you have to think that how you do it is you use--

I mean, as I said, the learning rate that you're going to use

is going to be different, but also you only do that.

So just imagine if I trained--

even if I trained on one sentence,

but over and over again at some point

my model will only generate that sentence

even if it was just one sentence instead of

the 15 trillion tokens.

So if you use a large enough learning

rate and for enough time, you will basically

overfit that sentence.

So the key thing to remember is that the data is not--

it's not as if you mix some post-training data

and some pretraining data.

You do pretraining, and then you just start fine-tuning only

on the post-training.

So another way, maybe another perspective

is that the pretraining is just the initialization

of your model.

And once you view it that way, that this is just

initialization of weights, then there's nothing special.

Like you don't need to remember that you train on a lot of data

before.

The only thing that matters is that you had an initialization

and now I actually train the model.

So maybe you think about it that way.

Like this is a Markov property in some ways.

It's just like you had your weights.

This is my initialization.

Now I'm training that one.

Does that answer your question?

Kind of but you said something just now about it's

almost the equivalent of just rerunning the fine tuning

data many times.

Is it actually-- is that what actually happens in order

to give so much more preference?

You might-- I actually don't know right now how they do it

in industry.

When we did our packet, we had to do three epochs.

So you did run it three times through it.

But I mean, even the number of times

that you run it through, it's actually not important.

The only thing-- the only thing is the effective learning rate

that what matters.

So yeah.

Great.

So I think I have five minutes.

OK I might try to give a high-level overview at least

from one of the systems trick.

Systems, as we said, for everyone bottleneck is--

sorry compute is the huge bottleneck.

One question you might ask is, why not buy more GPUs?

GPUs are expensive, but also are scarce.

Even if you have $10 million right now,

you cannot buy the best GPUs.

[INAUDIBLE]

There's also some physical limitations.

When you have multiple GPUs, you have

to communicate between them.

That takes time.

So just buying more GPUs is not that easy.

So it's really important to think about

how do you allocate resources and how do you optimize

your pipeline, so system?

101 on GPUs, I'm sorry, I'm going slightly faster.

I hope that some of you at least can follow.

GPUs are basically optimized for throughput.

CPUs are optimized for latency.

So GPUs, the way you have to think about it

is that there's one--

there's one command that is run on many, many cores

at the same time on different type of data.

So this is how you see a GPU.

You see there are many different codes.

We call them streaming multiprocessors,

which is very different than the usual CPU architecture.

So just think high throughput parallelization for GPUs.

GPUs are optimized for fast matrix multiplication.

So every time you will do-- you will do something on GPU.

If you can do it with a matrix multiplication,

it's going to be 10 times faster than with anything else.

That is a little bit annoying because it

means that we are, kind of, bottlenecked

to doing anything with matrix multiplications.

Another thing to note with GPUs is

that compute has been improving faster

than memory and communication.

So right now GPUs usually are hard to keep--

Like the data that you sent to GPUs

is actually hard to keep up with the processes.

So most of your GPUs are actually

going to be idle if you just run normal code,

if you don't optimize your code.

So communication-- and this will continue over time.

Another thing to know about GPUs is that there's

a memory hierarchy.

This is the same thing actually with CPUs,

but basically the closer you are to your cores,

the less memory there is, but the faster things run.

If you are further, more memory slower.

Oh yeah I'm going to skip that.

OK actually, I'm going to say it.

I told you about this--

the fact of communication.

The metric that people usually look at

is model FLOP utilization.

So what is the theoretical maximum that GPU could run at,

number of flops that you could use per second--

divide-- sorry, the number of observed throughput

divided by this theoretical maximum.

And in general, if you reach 50% you're very happy.

Like Facebook I looked at llama was at 45

or something like this.

So that means that data doesn't come fast enough

even for these big companies.

So one simple trick, and that might

be the only one I'm going to tell you about,

is low precision.

One simple idea is that well, if I'm

going to put my floats in low precision,

then there's going to be fewer bits

that I have to send to my GPUs.

If there's fewer bits, it's faster communication,

lower memory consumption.

Things are going to go faster.

And for deep learning it just happens

that decimal is not that important.

So when you do matrix multiplication, when

you do like for example, SGD, there's already so much noise

that if you update something by 0.01 or 0.015, who cares.

So basically instead of using 32 bits per float, which

is what people used to use, or 64 for example, which

is what you would use in other domains,

you use 16 bits for matrix multiplication.

So for every float you use 16 bits.

And for training you have this type

of what we call automatic mixed precision.

Which is that some of the things are in 32 bits,

others are in 60 bit--

on 16 bits.

Generally, the way you should be thinking about

it is that your weights are stored-- of your model,

are stored in 32 bits.

But just before the computation you put everything in 16 bits.

Like this you do computation super fast.

And at the end you update your weights in 32 bits.

And the reason why you do all the updates in 32 bits is just

think that if your learning rate, for example,

is very small, you still want to be able to make

a difference in your weights.

So all the computation is done in 16 bits,

but the weights are actually stored in 32 bits.

So that's like the standard way that people are doing it.

OK, I'll actually talk just about this,

and then I'll skip all the rest, operator fusion, because I think

this is actually pretty cool.

As I just said, communication is very slow

and actually every time you use a PyTorch line,

it basically moves variable to global memory of your GPU.

So when you have something like this x dot cosine equal x1,

and then you do x1 dot cosine.

What is happening behind the scenes

is that you take the x, which is data.

You ship it to your actual processors of your GPUs.

You apply the cosine.

You ship it back to the main memory of your GPU

and then you see the next line.

You ship it back to the computer-- to the GPU processor,

you apply another cosine and you ship it back again.

So another way to see that is that you

go from your DRAM, which is your global memory and your GPU

and you ship it to compute.

You ship it back for every line.

This is a naive way of doing it.

This seems very wasteful.

So the idea, simple idea of operator fusion

is just communicate, do all the computation, ship it back once.

And this is exactly what fused kernels are.

So if you ever want to make your compute-- your computations

in PyTorch much faster, just apply torch dot

compile on your model.

This is going to make your model around 2 times faster.

And what it does is simply that it rewrites your code--

your PyTorch code basically in C++ in CUDA to do

the communication only once then do all the operations,

then ship it back.

OK I'm not going to have time to talk about tiling.

Tiling is important.

Parallelization.

Parallelization is important.

And mixture of experts.

Mixture of experts is important.

Outlook.

There are many things we haven't talked about.

We haven't talked about architectures we definitely

haven't talked about inference.

There are many other things that are important with LLMs.

What is the UI that you use?

I mean, arguably ChatGPT, the big novelty was just

have a simple UI to use it.

Multi-modality.

What are all the misuses you could have.

The fact that there might not be enough data on the internet

to train all these models.

Legality of data collection, so many other things.

If you are interested in all these topics,

I would suggest three classes.

CS224N is probably the one that touches the least on LLMs,

but it gives some background and historical context

of all the LLMs and gives some adjacent material.

CS324 I think it's called--

I think it's just called Large Language Models, more

in depth reading and lectures on everything I talked about.

CS336 which is large language model from scratch,

you actually build your own LLM.

It's an amazing class also given by my two supervisors.

Very heavy workload, so be careful.

Great.