跟讀練習: Vanishing & Exploding Gradient explained | A problem resulting from backpropagation - 通過影片學習英語口語

選擇練習模式

每種模式都會開啟全屏練習室。

78 句

如果句子過短或過長，請點選 Edit 進行調整。

Hey, what's going on everyone?

In this video, we're going to discuss a problem

that creeps up time and time again during the training process of an artificial neural network.

This is the problem of unstable gradients, and is most popularly referred to as the vanishing gradient problem.

So let's get to it.

Alright, what do we really already know about gradients as it pertains to neural networks?

Well, for one, when we use the word gradient by itself, we're typically referring to the gradient of the loss function with respect to the weights in the network network.

We also know how this gradient is calculated, using backpropagation, which we covered in our earlier videos dedicated solely to backprop.

And finally, as we saw in our video that demonstrates how a neural network learns, we know what to do with this gradient after it's calculated.

We update our weights with it.

Well, we don't, per se, but stochastic gradient descent does, with the goal in mind to find the most optimal weight for each connection

that will minimize the total loss of the network.

So, with this understanding, we're now going to talk about the vanishing gradient problem.

We're first going to answer, well, what the heck is the vanishing gradient problem anyway?

Here we'll cover the idea conceptually.

We'll then move our discussion to talking about how this problem occurs.

Then, with the understanding that we'll have developed up to this point, we'll discuss the problem of exploding gradients, which we'll see is actually very similar to the vanishing gradient problem,

and so we'll be able to take what we learned about that problem and apply it to this new one.

So what is the vanishing gradient problem?

We'll first talk about this kind of generally, and then we'll get into the details in just a few moments.

In general, this is a problem that causes major difficulty when training a neural network.

More specifically, this is a problem that involves the weights in earlier layers of the network work.

Recall that during training, Stochastic Gradient Descent, or SGD, works to calculate the gradient of the loss with respect to weights in the network.

Now, sometimes, and we'll speak more about why this is in a bit, the gradient with respect to weights in earlier layers of the network becomes really small,

like vanishingly small.

Hence, vanishing gradient.

Okay, what's the big deal with a small gradient?

Well, once SGD calculates this gradient with respect to a particular weight, it uses this value to update that weight.

So the weight gets updated in some way that is proportional to the gradient.

If the gradient is vanishingly small, then this update is in turn going to be vanishingly small as well.

So if this new updated value of the weight has just barely moved from its original value, then it's not really doing much for the network.

This change is not going to carry through the network very well to help reduce the loss

because it's barely changed at all from where it was before the update occurred.

Therefore this weight becomes kind of stuck, never really updating enough to even get close to its optimal value,

which has implications for the remainder of the network to the right of this one weight

and impairs the ability of the network to learn.

So now that we know what this problem is, how exactly does this problem occur?

We know from what we learned about backpropagation

that the gradient of the loss with respect to any given

weight is going to be the product of some derivatives that depend on components that reside later in the network.

So given this, the earlier in the network a weight lives, the more terms will be needed in the product

that we just mentioned to get the gradient of the loss with respect to this weight.

What happens if the terms in this product, or at least some of them, are small?

And by small, we mean less than one small.

Well, the product of a bunch of numbers less than one is going to give us an even smaller number, right?

Okay, cool.

So as we mentioned earlier, we now take this result, the small number, and update our weight with it.

Recall that we do this update by first multiplying this number by our learning rate, which it itself is a small number,

usually ranging between 0 .01 and 0 .0001.

So now the result of this product is an even smaller number.

Then we subtract this number from the weight, and the final result of this difference is going to be the value of the updated weight.

Now you can think about if the gradient that we obtained with respect to this weight was already really small, i .e vanishing, then by the time we multiply it by the learning rate,

the product is going to be even smaller, and so when we subtract this teeny tiny number from the weight, it's just barely going to move the weight at all.

So, essentially, the weight gets into this kind of stuck state not moving,

not quote learning, and therefore not really helping to meet the overall objective of minimizing the loss of the network.

And we can see why earlier weights are subject to this problem.

Because as we said, the earlier in the network the weight resides, the more terms are going to be included in the product to calculate the gradient.

And the more terms we're multiplying together that are less than 1, the quicker the gradient is going to vanish.

So now let's talk about this problem in the opposite direction.

Not a gradient that vanishes, but rather a gradient that explodes.

Well, think about the conversation we just had about how the

vanishing gradient problem occurs with weights early in the network due to a product of at least some relatively small values.

Now think about calculating the gradient with respect to the same weight, but instead of really small terms, what if they were large?

And by large, we mean greater than 1.

Well, if we multiply a bunch of terms together that are all greater than 1, we're going to get something greater than 1, and perhaps even a lot greater than 1.

The same argument holds here that we discussed about the vanishing gradient, where the earlier in the network a weight lives, the more terms will be needed in the product we just mentioned.

And so the more of these larger value terms we have being multiplied together, the larger the gradient is going to be, thus essentially exploding in size.

And with this gradient, we go through the same process to proportionally update our weight with it that we talked about earlier.

But this time, instead of barely moving our weight with this update, we're going to greatly move it.

So much so, perhaps, that the optimal value for this weight won't ever be achieved

because the proportion to which the weight becomes updated with each epoch is just too large

and continues to move further and further away from its optimal value.

So a main takeaway that we should be able to gain from this discussion is

that the problem of vanishing gradients and exploding gradients is actually a more general problem of unstable gradients.

This problem was actually a huge barrier to training neural networks in the past, and now we can see why that is.

In a later video, we'll talk about techniques that have been developed to combat against this problem.

For now, let's take our discussion to the comments.

Let me know what you're thinking.

下載應用

流利口語所需的一切

AI 發音評分逐句打分

音標練習掌握每個音

詞彙學習建立你的詞庫

詞彙遊戲邊玩邊學

掃碼

Download on theApp Store Get it onGoogle Play

FOR YOU

本課的詞彙與口語要點

這節口語課以影片“Vanishing & Exploding Gradient explained”為素材。影片中反覆出現的詞有：gradient, weight, network, vanishing, product。這段影片共有 78 個句子、1279 個單詞可供跟讀。講話部分時長為 7:19。說話人語速自然，每分鐘約 175 個詞，接近日常對話。 87% 的單詞屬於英語最常用的 3,000 詞，詞彙容易理解。

影片中的重點詞彙

影片中 15 個值得學習的單詞，附發音和釋義：

單詞	發音	釋義
calculate 動詞	/ˈkælkjʊleɪt/	計算 /計算, 算
occur 動詞	/əˈkɜː/	發生 /發生
optimal 形容詞	/ˈɑptɪməl/	最佳的
explode 動詞	/ɪkˈsploʊd/	爆炸, 爆發 /爆發
refer 動詞	/ɹɪˈfɜː/	談到 /談到
essentially 副詞	/ɪˈsɛnʃəli/	本質上 /本質上, 基本上
layer 名詞	/ˈleɪ̯ɚ/	層 /層
recall 動詞	/ɹɪˈkɔl/	罷免 /罷免, 召回
minimize 動詞	/ˈmɪn.ɪˌmaɪz/	最小化
reside 動詞	/ɹɪˈzaɪd/	居住, 住
technique 名詞	/tɛkˈniːk/	技術 /技術, 技巧
obtain 動詞	/əbˈteɪn/	獲得 /獲得
hence 副詞	/ˈhɛns/	從這裡 /從這裡, 從這兒 /從這兒
objective 名詞	/ɒbˈd͡ʒɛk.tɪv/	目標 /目標, 目的
difficulty 名詞	/ˈdɪfɪkəlti/	困難 /困難

影片中出現的短語動詞

單詞	釋義
go through 動詞	通過 /通過

值得反覆跟讀的句子

影片中簡短而完整的句子，可以直接用在日常對話裡：

Hey, what's going on everyone?
Here we'll cover the idea conceptually.
For now, let's take our discussion to the comments.
Let me know what you're thinking.

需要注意的發音

說話人用了 20 次縮略和弱讀形式，例如 we'll, we're, don't。請按聽到的簡短形式來說。

“sh” 和 “zh” 音: essentially /ɪˈsɛnʃəli/, artificial /ˌɑː.tɪˈfɪʃ.əl/, proportion /pɹəˈpɔɹʃən/, implication /ˌɪmpləˈkeɪʃən/, vanish /ˈvænɪʃ/
長單詞——注意重音位置: essentially /ɪˈsɛnʃəli/, difficulty /ˈdɪfɪkəlti/, artificial /ˌɑː.tɪˈfɪʃ.əl/, derivative /dɪˈɹɪvətɪv/, implication /ˌɪmpləˈkeɪʃən/

中文母語者容易讀錯的音：

/v/ — 不要讀成 /w/，上齒輕觸下唇: objective /ɒbˈd͡ʒɛk.tɪv/, involve /ɪnˈvɑlv/, derivative /dɪˈɹɪvətɪv/, vanish /ˈvænɪʃ/
詞尾子音 — 要讀清楚，後面不要加母音: calculate /ˈkælkjʊleɪt/, explode /ɪkˈsploʊd/, descent /dəˈsɛnt/, minimize /ˈmɪn.ɪˌmaɪz/, reside /ɹɪˈzaɪd/
/r/ 和 /l/ 的區別: neural /ˈnʊɹəl/, recall /ɹɪˈkɔl/, greatly /ˈɡɹeɪ̯t.li/, proportionally /pɹəˈpɔɹʃənəli/

如何用這段影片練習

先完整聽一遍影片，不要開口，記下不認識的單詞。
先用 0.75 倍速逐句跟讀，熟練之後再回到正常速度。
錄下自己的聲音並與原聲對比，特別注意 calculate, occur, optimal 這類單詞。

什麼是跟讀法？

跟讀法 (Shadowing) 是一種有科學依據的語言學習技巧，最初開發用於專業口譯員的培訓，並由多語言者Alexander Arguelles博士普及。這個方法簡單而強大：您在聽英語母語原聲的同時立即大聲重複——就像是一個延遲1-2秒緊跟說話者的影子。與被動聽力或語法練習不同，跟讀法強迫您的大腦和口腔肌肉同時處理並模模擬實的講話模式。研究表明它能顯著提高發音準確性，語調，節奏，連讀，聽力理解和口語流利度——使其成為雅思口語備考和真實英語交流最有效的方法之一。

影子跟讀法: 閱讀完整分步指南 →

跟讀練習: Vanishing & Exploding Gradient explained | A problem resulting from backpropagation - 通過影片學習英語口語

選擇練習模式

下載應用

推薦影片

Mixture of Experts Explained Visually: How Trillion-Parameter Models Actually Work

OWASP Top 10 Explained

Discussing My Beliefs.. (Nihilism)

Vectors and 2D Motion: Crash Course Physics #4

BGP Overview

MIT Introduction to Deep Learning | 6.S191

What is Inattentional Blindness? (Easiest Explanation)

Gaussian Naive Bayes, Clearly Explained!!!

Mental Debugging: How to Build Resilience After Failure

Give me 31 minutes, I will make PyTorch click forever

The GELU, SiLU and SwiGLU activation functions, clearly explained!!!

Physics Informed Neural Networks (PINNs) [Physics Informed Machine Learning]