Shadowing Practice: Transformer Networks Explained: How ChatGPT Works - Learn English Speaking with Video

Elige un modo de práctica

Cada modo abre una sala de práctica a pantalla completa.

259 oraciones
Si las oraciones son demasiado cortas o largas, haz clic en Editar para ajustarlas.
1
In this video we'll discuss transformers, which are special types of neural networks that are mainly used in large language models,
2
such as ChatGPT.
3
This was the original structure of the transformer that was first published in 2017 in a paper entitled
4
Attention is all you need.
5
A transformer can for example be used to translate from one language to another.
6
This is the encoder, whereas this is the decoder part of the transformer.
7
Suppose that we would like to translate this
8
Swedish sentence Jag gillar kaffe, which means I like coffee.
9
The encoder then processes this
10
Swedish sentence, which goes into the decoder,
11
together with the first two English words that have already been translated from previous steps,
12
to predict the word kafir.
13
Suppose that we would like to use a transformer to only predict the next word in a sentence of the same language.
14
Then we do not need the encoder, which means that we use a decoder -only transformer.
15
ChatGPT is, for example, a decoder -only transformer.
16
This is the transformer that we will work with in this video to predict the next word in the sentence.
17
Once the model has predicted the word coffee, it can be used as input to build a whole sentence like this.
18
The transformer that I will use in this video involves word embedding and positional encoding.
19
To simplify the calculations, we'll here just use a single head attention block, and skip the feedforward layer,
20
and directly use the output from the normalization layer into the
21
final fully connected layer to predict the next word in a sentence.
22
Inside a single -head attention block, a number of steps take place, which involve the calculation of the Q, K and V matrices.
23
I will go through every step in this transformer based on a simple example.
24
But first, we will start to discuss tokenization, which is the process of splitting raw text into tokens
25
this sentence can be separated into the following tokens in this
26
case a token is the same as a word each word
27
or token is assigned an identifier which means
28
that each token is associated with a unique integer these numbers
29
are the ones the network uses to internally process the words
30
so if the network were to interpret this sentence it would write the sentence with these token ids suppose
31
that the network has predicted the following sequence of numbers that he wants to display for the user
32
it then converts the numbers to words or tokens which are then displayed on the screen i like my new car.
33
Padding is used to ensure that all input sequences have the same length during training.
34
Suppose that the expected input sequence length would be only seven words or tokens during the training.
35
If the model sees these three words as the input sequence it would add padding like this
36
and the corresponding token ID sequence would then look like this.
37
In language models, one uses masking, which is a way to prevent the model from learning from padded tokens.
38
By using masking, the model will pay no attention to these padded tokens
39
when interpreting the input sequence to predict the next word.
40
Note that a word may also be split into two or more tokens
41
to better interpret the sequence so now we know what tokens are
42
and that they can be seen as words characters
43
or parts of words we'll now discuss word embedding which is a way to explain the meaning of a word to computer
44
Suppose that we have the following words or tokens and
45
that we would like to explain the meaning of these words to a computer.
46
Let's start by explaining how mobile these things are.
47
For example, trucks and cars are mobile because they can transport things from one place to another.
48
Cats and dogs can also move, whereas fruits and houses are not mobile.
49
So, we give items that are mobile positive values.
50
More mobile items will get larger values, whereas less mobile items will get negative values on this scale.
51
This is called an embedding dimension, or an embedding vector, that describes how mobile the items are.
52
Let's add a second dimension that describes if the items are eatable.
53
Since fruits are eatable and are things that are not mobile, they will be in the top left corner in these two dimensions.
54
A house is not eatable and not mobile, so it would be down here.
55
Cats and dogs are normally not referred to as eatable, and trucks and cars are definitely not eatable.
56
what's nice with this representation is that similar items are grouped together
57
which helps the transformer to interpret the meaning of these words for example the transformer now understands
58
that trucks and cars are similar items however we need more dimensions because although bananas and apples are fruits.
59
An apple might also be associated with the company that develops technology products, such as phones and computers.
60
So, if we add a third dimension that describes how much the words or tokens are associated with technology,
61
we can now separate the two words banana and apple, where the word apple gets a high technology score,
62
whereas the word banana gets a low score the GPT -3 model had about 12 ,000 embedding dimensions
63
and 50 ,000 tokens
64
so where do these values come from well transformers are trained on large amount of text
65
and during the training they learn to associate eatable things with fruits
66
and that cars and trucks can move things these values are actually weights
67
that the network learns to optimize in order to correctly predict the next word suppose
68
that you would predict the next word in this sentence i guess you predicted the word apple because it fits the context.
69
The word that you paid most attention to was probably eating
70
because the predictive word should be associated with something that we can eat.
71
The color of the item that is eaten also helps to predict the next word.
72
If we were to change this word from she to he
73
that would not change our prediction which means that this word is less useful to predict the next word.
74
If we change the word eating to wearing the prediction would now change completely.
75
Other words may now fit better.
76
The caller does not here help us much to predict the next word.
77
So to predict the next word in a sentence we pay more attention to certain words
78
which is exactly what a transformer is all about training a transformer involves a target sequence
79
that can be seen as a true sentence from a given text
80
if the transformer happens to predict the incorrect word it will updates its trainable weights using back propagation
81
which means that it corrects its error so that it predicts the correct word or token.
82
This is done on thousands of words and sentences.
83
Suppose that these are all the words that we have to build a sentence with, and that these two embedding dimensions represent food and clothes.
84
For example, apple is highly associated with food but not with clothes, whereas the opposite is true for the word shirt.
85
Note that zero here means no association with embedding dimension.
86
So, if a transformer were to predict the next word in the following sentence,
87
it would extract these rows from the embedding matrix with the words
88
or tokens that are included in the sentence so
89
that they have the following reduced embedding matrix
90
so this will be our input to our simplified transformer
91
that has already been trained to predict the next word when we use a trained model to predict.
92
This process is called inference.
93
Next, we should add positional encoding to give the transformer information about the order of the words in the sentence.
94
A simple way to include positional encoding is to create a
95
matrix of the same size as the extracted embedding matrix and place increasing numbers in this matrix,
96
and simply add those numbers to the embedding matrix.
97
A more common way is to compute the positions based on sine and cosine functions.
98
But there are a number of ways to do this.
99
This is just one method.
100
Since we only have two embedding dimensions in our example.
101
An i starts at zero and is increased to one for the second embedding dimension.
102
The exponent will here be equal to zero which means
103
that we can simplify the equations of the positional encoding like this
104
so we plug in position zero
105
and compute the values then we plug in position one two three
106
and four we will
107
We'll now add these two matrices, so that we get position -aware embeddings.
108
For example, 0 .2 plus 0 .14 is 0 .34.
109
We'll now compute the attention matrix, which starts by computing the Q
110
and K matrices based on the position -aware embedding matrix
111
so to compute the attention matrix we multiply the position -aware embedding matrix x by a
112
so -called q -weight matrix to produce a query matrix
113
or q matrix to multiply these matrices we do a dark product which means that we multiply the values in these elements,
114
and then these elements, and then sum these two products.
115
Then we continue to the second row and do the same calculations.
116
Note that since these weights happen to be 1 and 0,
117
the first embedding dimension is identical to the new feature dimension in q
118
this new feature dimension therefore represents association to food only
119
and since these weights happen to be 0 .5 the second
120
feature dimension can be seen as an equal mixture of food
121
and clothes these values are optimized during the training on a large amount of text
122
so that the transformer predicts the next word as good as possible note
123
that these weights are fixed which means
124
that they are used for all types of input sequences during
125
inference after the network has been trained the q weight matrix
126
may actually have a different number of columns than the embedding matrix
127
which will produce a query matrix with a different size than that position -aware embedding matrix.
128
The GPT -3 model had 12 ,288 embedding dimensions,
129
so it would have had this size for our simple example.
130
And the Q -weight matrix would therefore have 12 ,288 rows.
131
The Q -weight matrix for a single head attention had 128 columns in GPT -3,
132
which would therefore produce a query matrix of 5 times 128 in our example, because our sequence length is 5,
133
and the number of columns in the Q -weight matrix is 128.
134
For example, these dimensions represent the original position -aware embedding dimensions of food and clothes,
135
because the Q weight matrix happened to include these numbers.
136
Whereas this dimension represents an equal mixture of the two embedding dimensions,
137
because the values of these two weights are the same.
138
Anyway, to keep things simple, we use just a 2x2 matrix in this example.
139
Note that by multiplying the Q -weight matrix by matrix X is the same as using a linear fully connected
140
or dense layer here, with the following weights.
141
If we plug in these two values here, and compute the weighted sums of the embedding values for the first token,
142
we'll get the values of the first row of the query matrix.
143
Then we take the next row, and so forth.
144
To produce the attention matrix, we also need to calculate the key matrix.
145
Let's use this k -weight matrix with the following made -up weights.
146
If we multiply the k -weight matrix by the position -aware embedding matrix, we will get the key matrix.
147
which again can be seen as if we use a linear fully connected or dense layer.
148
We will now multiply q by k.
149
So these are our q and k matrices.
150
To multiply these matrices, we need to take the transpose of k so that it looks like this,
151
where its previous columns are now presented as rows.
152
Multiplying Q and K gives us the following matrix.
153
For example, this value is calculated by multiplying these elements and these.
154
This follows the same concept as how a similarity matrix is calculated based on the DART product.
155
If you like, you can try to multiply the embedding matrix with its transpose to see what you get.
156
The advantage of involving Q and K is that it includes trainable weights, so that the model can learn more about the attention.
157
Next, we scale the matrix
158
which is done during the training process for numerical stability by
159
dividing the values in this matrix by the square root of the feature dimensions of q
160
and k which is 2 in our example so
161
that we get these values next
162
we apply causal masking by setting the upper triangle of the matrix to infinitely large negative values
163
i have here set these elements to negative 1 billion in a decoder only transformer
164
we like to apply a causal mask to prevent tokens from attending to future tokens
165
i will explain what this means after
166
we have applied the soft max function next we apply the soft max function this is the soft max function
167
that we will apply to every row.
168
So, if we plug in this value here, and all the corresponding values in the denominator,
169
and do the math, it will result in the value 1 that we plug in here.
170
Then we apply the softmax function to the other values in this row.
171
Once we are done, the values should span between 0 and 1.
172
The values on each row should also sum up to 1.
173
Then we apply the softmax function on the other rows, which will result in these values.
174
So, when we apply causal masking,
175
we make sure that the token only attends to itself and the previous tokens to predict the next token.
176
this explains by the token eating has non zero values for these tokens
177
and zero values for these tokens the token eating will therefore pay no attention to the words or tokens that come after
178
so this is our attention matrix
179
which is used to determine how much each word in a sentence should focus
180
or attend to every other word to understand the context of the sentence
181
The rows represent the query, whereas the columns represent the key.
182
So, you should read from left to right.
183
For example, this row shows how much the token eating attends to itself and the other tokens.
184
Since this token attends to other elements of itself, this is sometimes called self -attention.
185
For example, the token eating is focused on itself, and mainly on the word she.
186
Similarly, green is mostly focused on ö and itself.
187
However, note that I have used made -up weights in this video, so don't pay too much attention on the exact values in this matrix.
188
So, we are approaching the end of the attention block, which involves the computation of the so -called value matrix.
189
We will multiply the value matrix by the attention matrix.
190
To compute the value matrix V, we multiply the weight matrix V, which includes these four weights,
191
by the position -aware embedding matrix.
192
We are now ready to multiply the tension matrix by matrix V,
193
which will result in the so -called context matrix that includes contextual embeddings.
194
Note that this matrix looks very similar to our previous position -aware embedding matrix, but now also includes the information about the tension.
195
For example, this value comes from the following calculation.
196
Next we will apply this residual connection, which means that we will add the position aware embedding matrix,
197
with the context matrix we just computed, so that we get the following matrix.
198
By doing so, we restore some of the information from the positional aware embedding matrix
199
that might have been lost in the attention block, and also facilitate the optimization process during the training.
200
Note that these matrices have to be the same size in order to add them.
201
If the context matrix would have, for example, three dimensions, because we might have used weight matrices with three columns.
202
Then we may use an output projection matrix with three rows and two columns,
203
which we multiply with the context matrix, so
204
that we get a context matrix with the same size as
205
the position of our embedding matrix in order to add these two matrices.
206
The output projection matrix is especially useful when we have several attention heads
207
that needs to be compressed to the same dimension as x.
208
It is now time to normalize this matrix, which is done to stabilize the computation during the training.
209
This means that we subtract the mean from these values so that we get centered values around zero.
210
And then divide the centered values by the square root of
211
the variance plus a small constant to avoid the possibility of dividing by zero.
212
We will by this standardize the values.
213
This calculation also includes trainable weights that help to stabilize the computation.
214
However, we will not include these here.
215
These are our normalized values.
216
Since we only have two feature dimensions, the normalized values will either be negative one or one.
217
Using only two embedding dimensions is therefore not a good idea, because the normalization then removes a lot of information in this case.
218
We will therefore skip the normalization and instead use these values as input to the fully connected layer.
219
So, in order to predict which word that comes after green, we plug in its corresponding values in the fully connected layer.
220
These values contain information about the embedding, the position, and the attention.
221
If we want to predict what comes after eating, we would instead plug in these values.
222
Remember that the first feature dimension of the original embedding matrix was associated with food,
223
and that the second embedding dimension was associated with clothes.
224
It would therefore make sense that the network has a low
225
or negative weight that is associated with the connection between clothes and apple, and a high weight between food and apple.
226
This is a bias weight that is also optimized during the training.
227
Then we compute the output value for apple by using the bias weight,
228
the input values, and the weights that are associated with the output node apple.
229
Then we do the same calculation for the output node shirt.
230
It would make sense that the network has optimized the weights so that food gives a weak signal to shirt,
231
whereas clothes gives a strong signal.
232
Suppose that we would get these values, called logits, for all the output nodes.
233
Note that the calculations of these numbers correspond to this matrix operation, where x is the following matrix
234
and this is the matrix with the weights associated to the connections between the input
235
and output nodes whereas this is a vector with the bias weights.
236
Finally, we plug in these values into the softmax function to compute the output probabilities.
237
So, if we apply the softmax function on these values, we will get these probabilities.
238
Since the highest probability is associated with apple, the model will predict that the next word is apple.
239
Suppose that we would shift these weights.
240
Then the model would instead predict the word shirt.
241
Since the model trains on target sequences, it will try to increase this probability to make the correct prediction.
242
by changing the values of these weights, and all the other weights in the model, including the weights in the Q, K and B weight matrices,
243
and the weights in the embedding matrix.
244
Although we use a very simple example, the network has 24 weights to optimize in this layer.
245
Two weights each for gamma and beta, because we have two feature dimensions.
246
12 weights to produce the Q, V, and K matrices and 16 weights in the embedding matrix.
247
So, in total, our simple network has 56 weights to be optimized during the training.
248
When we use the network for predicting the next word after the training is done, the values of the 56 weights are fixed.
249
however the attention weights in the attention matrix are not fixed
250
because these values are recalculated depending on
251
which sentence we plug in into the network finally we will have a look at one method
252
that produces more variation in the output which is called top p sampling
253
Instead of selecting the word with the highest probability,
254
it can sample from the smallest set of words or tokens whose cumulative probability is greater than p,
255
which can for example be set to 0 .75.
256
The method might here, due to chance, therefore select the word hat.
257
By randomly selecting among the most likely words, it will produce a larger variation in its output.
258
This was the end of this video about transformer neural networks.
259
Thanks for watching.

Descargar app

Todo lo que necesitas para hablar con fluidez

Pronunciación con IAEvalúa cada oración
Práctica de IPADomina cada sonido
VocabularioConstruye tu banco de palabras
Juego de vocabularioAprende mientras juegas
FOR YOU

Recomendar Video

Vocabulario y notas de pronunciación para esta lección

Esta lección de conversación se basa en el vídeo “Transformer Networks Explained: How ChatGPT Works”. Las palabras que más se repiten: matrix, word, value, weight, embedding. Este vídeo tiene 259 frases y 3456 palabras para practicar shadowing. La parte hablada dura 29:05. El hablante mantiene un ritmo constante de unas 119 palabras por minuto, cómodo para el shadowing. Solo el 79 % de las palabras está entre las 3.000 más comunes del inglés, por lo que el vocabulario es exigente.

Vocabulario clave de este vídeo

15 palabras del vídeo que vale la pena aprender, con su pronunciación y significado:

PalabraPronunciaciónSignificado
matrix sustantivo/ˈmeɪ.tɹɪks/matriz
token sustantivo/ˈtoʊkən/seña
dimension sustantivo/daɪˈmɛn.ʃən/dimensión
predict verbo/pɹɪˈdɪkt/predecir
output sustantivo/ˈaʊtpʊt/producción, output
sequence sustantivo/ˈsiː.kwəns/secuencia
input sustantivo/ˈɪnpʊt/aportación, aporte
whereas/(h)wɛɹˈæz/mientras que (locution)
plug sustantivo/plʌɡ/enchufe, clavija
layer sustantivo/ˈleɪ̯ɚ/capa, tongada
calculation sustantivo/ˌkælkjuˈleɪʃn̩/cálculo, calculación
probability sustantivo/ˌpɹɑ.bəˈbɪl.ə.ti/probabilidad
connect verbo/kəˈnɛkt/conectar, empalmar
mask sustantivo/mæsk/máscara, mascarilla
involve verbo/ɪnˈvɑlv/involucrar

Phrasal verbs que vas a escuchar

PalabraPronunciaciónSignificado
make up verbo/ˌmeɪk ˈʌp/reponer, suplir
go through verboatravesar

Gramática en este vídeo

Las estructuras que más usa el hablante, con las palabras exactas del vídeo:

EstructuraEn el vídeo
Voz pasiva be + participio pasado — importa lo que ocurre, no quién lo haceare mainly used · been translated · can be used
Oraciones de relativo who / which + oración — información extra sobre una persona o cosatransformers, which are · kaffe, which means · encoder, which means
Present perfect have/has + participio pasado — una acción pasada que sigue importando ahorahave already been translated · has predicted · has already been trained

Pronunciación a tener en cuenta

El hablante usa 7 contracciones y formas reducidas, como we'll, don't. Dilas en su forma corta, tal como las oyes.

  • Los sonidos de “sh” y “zh”: dimension /daɪˈmɛn.ʃən/, calculation /ˌkælkjuˈleɪʃn̩/, prediction /pɹɪˈdɪkʃən/, tension /ˈtɛnʃən/, variation /ˌvɛəɹiˈeɪʃn̩/
  • Palabras largas — cuida el acento: probability /ˌpɹɑ.bəˈbɪl.ə.ti/, associate /əˈsoʊʃi.ət/, representation /ˌɹɛp.ɹə.zɛnˈteɪ.ʃən/, stability /stəˈbɪlɪti/, similarly /ˈsɪm.ə.lə.li/

Sonidos difíciles para hispanohablantes:

  • s + consonante al inicio — sin añadir una “e” delante: skip /skɪp/, spin /spɪn/, stability /stəˈbɪlɪti/, stabilize /ˈsteɪ.bəˌlaɪz/, standardize /ˈstæn.də.daɪz/
  • /v/ — no es /b/: los dientes tocan el labio inferior: involve /ɪnˈvɑlv/, divide /dɪˈvaɪd/, variation /ˌvɛəɹiˈeɪʃn̩/, vector /ˈvɛktɚ/, convert /kənˈvɝt/
  • /z/ — sonora, no /s/: whereas /(h)wɛɹˈæz/, columns /ˈkɑləmz/, translate /tɹænzˈleɪt/, representation /ˌɹɛp.ɹə.zɛnˈteɪ.ʃən/, optimization /ˌɑptəmaɪˈzeɪʃən/

Cómo practicar con este vídeo

  1. Escucha el vídeo entero una vez sin hablar y anota las palabras que no conoces.
  2. Haz shadowing frase por frase a velocidad normal, repitiendo cada una hasta que tu ritmo coincida con el del hablante.
  3. Grábate y compara con el original, prestando atención a palabras como matrix, token, dimension.

¿Qué es la Técnica de Shadowing?

Shadowing es una técnica de aprendizaje de idiomas respaldada por la ciencia, desarrollada originalmente para la formación de intérpretes profesionales y popularizada por el políglota Dr. Alexander Arguelles. El método es simple pero poderoso: escuchas audio en inglés nativo y lo repites en voz alta de inmediato, como una sombra que sigue al hablante con solo 1-2 segundos de retraso. A diferencia de la escucha pasiva o los ejercicios de gramática, el shadowing obliga a tu cerebro y músculos de la boca a procesar y reproducir simultáneamente patrones de habla reales. Las investigaciones muestran que mejora significativamente la precisión de la pronunciación, la entonación, el ritmo, el habla conectada, la comprensión auditiva y la fluidez al hablar, convirtiéndola en una de las metodologías más efectivas para la preparación del IELTS Speaking y la comunicación en inglés en el mundo real.

Técnica de shadowing: lee la guía completa paso a paso →