Original subtitles

How to instantiate a Transformers model?

In this video we will look at how we can create and use a model from the Transformers library.

As we've seen before, the TFAutoModel class allows you to instantiate a pretrained model

from any checkpoint on the Hugging Face Hub.

It will pick the right model class from the library to instantiate the proper architecture

and load the weights of the pretrained model inside it.

As we can see, when given a BERT checkpoint, we end up with a TFBertModel, and similarly

for GPT-2 or BART.

Behind the scenes, this API can take the name of a checkpoint on the Hub, in which case

it will download and cache the configuration file as well as the model weights file.

You can also specify the path to a local folder that contains a valid configuration file and

a model weights file.

To instantiate the pretrained model, the AutoModel API will first open the configuration file

to look at the configuration class that should be used.

The configuration class depends on the type of the model (BERT, GPT-2 or BART for instance).

Once it has the proper configuration class, it can instantiate that configuration, which

is a blueprint to know how to create the model.

It also uses this configuration class to find the proper model class, which is combined

with the loaded configuration, to load the model.

This model is not yet our pretrained model as it has just been initialized with random

weights.

The last step is to load the weights from the model file inside this model.

To easily load the configuration of a model from any checkpoint or a folder containing

the configuration folder, we can use the AutoConfig class.

Like the TFAutoModel class, it will pick the right configuration class from the library.

We can also use the specific class corresponding to a checkpoint, but we will need to change

the code each time we want to try a different model.

As we said before, the configuration of a model is a blueprint that contains all the

information necessary to create the model architecture.

For instance the BERT model associated with the bert-base-cased checkpoint has 12 layers,

a hidden size of 768, and a vocabulary size of 28,996.

Once we have the configuration, we can create a model that has the same architecture as

our checkpoint but is randomly initialized.

We can then train it from scratch like any PyTorch module/TensorFlow model.

We can also change any part of the configuration by using keyword arguments.

The second snippet of code instantiates a randomly initialized BERT model with ten layers

instead of 12.

Saving a model once it's trained or fine-tuned is very easy: we just have to use the save_pretrained

method.

Here the model will be saved in a folder named my-bert-model inside the current working directory.

Such a model can then be reloaded using the from_pretrained method.