쉐도잉 연습: Vanishing & Exploding Gradient explained | A problem resulting from backpropagation - 영상으로 영어 말하기 배우기

Hey, what's going on everyone?

In this video, we're going to discuss a problem

that creeps up time and time again during the training process of an artificial neural network.

This is the problem of unstable gradients, and is most popularly referred to as the vanishing gradient problem.

So let's get to it.

Alright, what do we really already know about gradients as it pertains to neural networks?

Well, for one, when we use the word gradient by itself, we're typically referring to the gradient of the loss function with respect to the weights in the network network.

We also know how this gradient is calculated, using backpropagation, which we covered in our earlier videos dedicated solely to backprop.

And finally, as we saw in our video that demonstrates how a neural network learns, we know what to do with this gradient after it's calculated.

We update our weights with it.

Well, we don't, per se, but stochastic gradient descent does, with the goal in mind to find the most optimal weight for each connection

that will minimize the total loss of the network.

So, with this understanding, we're now going to talk about the vanishing gradient problem.

We're first going to answer, well, what the heck is the vanishing gradient problem anyway?

Here we'll cover the idea conceptually.

We'll then move our discussion to talking about how this problem occurs.

Then, with the understanding that we'll have developed up to this point, we'll discuss the problem of exploding gradients, which we'll see is actually very similar to the vanishing gradient problem,

and so we'll be able to take what we learned about that problem and apply it to this new one.

So what is the vanishing gradient problem?

We'll first talk about this kind of generally, and then we'll get into the details in just a few moments.

In general, this is a problem that causes major difficulty when training a neural network.

More specifically, this is a problem that involves the weights in earlier layers of the network work.

Recall that during training, Stochastic Gradient Descent, or SGD, works to calculate the gradient of the loss with respect to weights in the network.

Now, sometimes, and we'll speak more about why this is in a bit, the gradient with respect to weights in earlier layers of the network becomes really small,

like vanishingly small.

Hence, vanishing gradient.

Okay, what's the big deal with a small gradient?

Well, once SGD calculates this gradient with respect to a particular weight, it uses this value to update that weight.

So the weight gets updated in some way that is proportional to the gradient.

If the gradient is vanishingly small, then this update is in turn going to be vanishingly small as well.

So if this new updated value of the weight has just barely moved from its original value, then it's not really doing much for the network.

This change is not going to carry through the network very well to help reduce the loss

because it's barely changed at all from where it was before the update occurred.

Therefore this weight becomes kind of stuck, never really updating enough to even get close to its optimal value,

which has implications for the remainder of the network to the right of this one weight

and impairs the ability of the network to learn.

So now that we know what this problem is, how exactly does this problem occur?

We know from what we learned about backpropagation

that the gradient of the loss with respect to any given

weight is going to be the product of some derivatives that depend on components that reside later in the network.

So given this, the earlier in the network a weight lives, the more terms will be needed in the product

that we just mentioned to get the gradient of the loss with respect to this weight.

What happens if the terms in this product, or at least some of them, are small?

And by small, we mean less than one small.

Well, the product of a bunch of numbers less than one is going to give us an even smaller number, right?

Okay, cool.

So as we mentioned earlier, we now take this result, the small number, and update our weight with it.

Recall that we do this update by first multiplying this number by our learning rate, which it itself is a small number,

usually ranging between 0 .01 and 0 .0001.

So now the result of this product is an even smaller number.

Then we subtract this number from the weight, and the final result of this difference is going to be the value of the updated weight.

Now you can think about if the gradient that we obtained with respect to this weight was already really small, i .e vanishing, then by the time we multiply it by the learning rate,

the product is going to be even smaller, and so when we subtract this teeny tiny number from the weight, it's just barely going to move the weight at all.

So, essentially, the weight gets into this kind of stuck state not moving,

not quote learning, and therefore not really helping to meet the overall objective of minimizing the loss of the network.

And we can see why earlier weights are subject to this problem.

Because as we said, the earlier in the network the weight resides, the more terms are going to be included in the product to calculate the gradient.

And the more terms we're multiplying together that are less than 1, the quicker the gradient is going to vanish.

So now let's talk about this problem in the opposite direction.

Not a gradient that vanishes, but rather a gradient that explodes.

Well, think about the conversation we just had about how the

vanishing gradient problem occurs with weights early in the network due to a product of at least some relatively small values.

Now think about calculating the gradient with respect to the same weight, but instead of really small terms, what if they were large?

And by large, we mean greater than 1.

Well, if we multiply a bunch of terms together that are all greater than 1, we're going to get something greater than 1, and perhaps even a lot greater than 1.

The same argument holds here that we discussed about the vanishing gradient, where the earlier in the network a weight lives, the more terms will be needed in the product we just mentioned.

And so the more of these larger value terms we have being multiplied together, the larger the gradient is going to be, thus essentially exploding in size.

And with this gradient, we go through the same process to proportionally update our weight with it that we talked about earlier.

But this time, instead of barely moving our weight with this update, we're going to greatly move it.

So much so, perhaps, that the optimal value for this weight won't ever be achieved

because the proportion to which the weight becomes updated with each epoch is just too large

and continues to move further and further away from its optimal value.

So a main takeaway that we should be able to gain from this discussion is

that the problem of vanishing gradients and exploding gradients is actually a more general problem of unstable gradients.

This problem was actually a huge barrier to training neural networks in the past, and now we can see why that is.

In a later video, we'll talk about techniques that have been developed to combat against this problem.

For now, let's take our discussion to the comments.

Let me know what you're thinking.

✨ 추천 영상

321 문장

Mixture of Experts Explained Visually: How Trillion-Parameter Models Actually Work

Daily ConversationUniversity Students

158 문장

이 레슨의 어휘와 말하기 포인트

이 말하기 레슨은 영상 “Vanishing & Exploding Gradient explained”을(를) 바탕으로 합니다. 가장 자주 반복되는 단어는 다음과 같습니다: gradient, weight, network, vanishing, product. 이 영상에는 섀도잉할 문장 78개와 단어 1279개가 있습니다. 말하는 구간의 길이는 7:19입니다. 화자는 분당 약 175단어로, 일상 대화에 가까운 자연스러운 속도로 말합니다. 단어의 87%가 영어에서 가장 많이 쓰이는 3,000단어에 속해 어휘가 쉬운 편입니다.

이 영상의 핵심 어휘

영상에 나오는 익혀 둘 만한 단어 15개를 발음, 뜻과 함께 정리했습니다.

단어	발음	뜻
calculate 동사	/ˈkælkjʊleɪt/	정산하다, 계산하다
occur 동사	/əˈkɜː/	일어나다, 발생하다
explode 동사	/ɪkˈsploʊd/	폭발적으로 증가하다, 폭발하다
refer 동사	/ɹɪˈfɜː/	언급하다
layer 명사	/ˈleɪ̯ɚ/	층
minimize 동사	/ˈmɪn.ɪˌmaɪz/	최소화(最小化)하다
reside 동사	/ɹɪˈzaɪd/	살다, 거주하다
technique 명사	/tɛkˈniːk/	기술
obtain 동사	/əbˈteɪn/	얻다, 획득하다
objective 명사	/ɒbˈd͡ʒɛk.tɪv/	목적
difficulty 명사	/ˈdɪfɪkəlti/	어려움
component 명사	/kəmˈpoʊ.nənt/	부분
artificial 형용사	/ˌɑː.tɪˈfɪʃ.əl/	인공, 인조
depend 동사	/dɪˈpɛnd/	의존하다, ...에 달려 있다
barrier 명사	/ˈbæɹi.ə/	장벽

따라 말해 볼 만한 문장

영상에 나오는 짧고 완결된 문장으로, 일상 대화에서 그대로 쓸 수 있습니다:

Hey, what's going on everyone?
Here we'll cover the idea conceptually.
For now, let's take our discussion to the comments.
Let me know what you're thinking.

이 영상의 문법

화자가 가장 많이 쓰는 문형을 영상 속 실제 표현과 함께 정리했습니다.

문형	영상 속 표현
수동태 be + 과거분사 — 누가 하는지보다 무슨 일이 일어나는지에 초점	is calculated · will be needed · be included
현재완료 have/has + 과거분사 — 과거의 일이 지금도 관련이 있을 때	have developed · have been developed

주의할 발음

화자는 we'll, we're, don't 같은 축약형과 약화된 형태를 20번 사용합니다. 들리는 대로 짧게 발음하세요.

“sh”와 “zh” 소리: essentially /ɪˈsɛnʃəli/, artificial /ˌɑː.tɪˈfɪʃ.əl/, proportion /pɹəˈpɔɹʃən/, implication /ˌɪmpləˈkeɪʃən/, vanish /ˈvænɪʃ/
긴 단어 — 강세 위치에 주의: essentially /ɪˈsɛnʃəli/, difficulty /ˈdɪfɪkəlti/, artificial /ˌɑː.tɪˈfɪʃ.əl/, derivative /dɪˈɹɪvətɪv/, implication /ˌɪmpləˈkeɪʃən/

한국어 화자가 어려워하는 소리:

/f/ — ㅍ(/p/)으로 바꾸지 말고 윗니를 아랫입술에 대기: refer /ɹɪˈfɜː/, difficulty /ˈdɪfɪkəlti/, artificial /ˌɑː.tɪˈfɪʃ.əl/
/v/ — ㅂ(/b/)과 구별하기: objective /ɒbˈd͡ʒɛk.tɪv/, involve /ɪnˈvɑlv/, derivative /dɪˈɹɪvətɪv/, vanish /ˈvænɪʃ/
/z/ — ㅈ이 아니라 성대를 울리는 /s/: minimize /ˈmɪn.ɪˌmaɪz/, reside /ɹɪˈzaɪd/

이 영상으로 연습하는 방법

먼저 말하지 않고 영상을 끝까지 듣고 모르는 단어를 적어 둡니다.
0.75배속으로 한 문장씩 섀도잉을 시작하고, 익숙해지면 보통 속도로 돌아갑니다.
자신의 목소리를 녹음해 원본과 비교하고, calculate, occur, explode 같은 단어에 특히 주의합니다.

쉐도잉이란? 영어 실력을 빠르게 키우는 과학적 방법

쉐도잉(Shadowing)은 원래 전문 통역사 훈련을 위해 개발된 언어 학습 기법으로, 다언어 학자인 Dr. Alexander Arguelles에 의해 대중화된 방법입니다. 핵심 원리는 간단하지만 매우 강력합니다: 원어민의 영어를 들으면서 1~2초의 짧은 지연으로 즉시 소리 내어 따라 말하는 것——마치 '그림자(shadow)'처럼 화자를 따라가는 것입니다. 문법 공부나 수동적인 청취와 달리, 쉐도잉은 뇌와 입 근육이 동시에 실시간으로 영어를 처리하고 재현하도록 훈련합니다. 연구에 따르면 이 방법은 발음 정확도, 억양, 리듬, 연음, 청취력, 말하기 유창성을 크게 향상시킵니다. IELTS 스피킹 준비와 자연스러운 영어 소통을 원하는 분들에게 특히 효과적입니다.

섀도잉 방법: 단계별 전체 가이드 읽기 →

쉐도잉 연습: Vanishing & Exploding Gradient explained | A problem resulting from backpropagation - 영상으로 영어 말하기 배우기

Mixture of Experts Explained Visually: How Trillion-Parameter Models Actually Work

OWASP Top 10 Explained

Vectors and 2D Motion: Crash Course Physics #4

BGP Overview

Enhancing AV Simulation with Generative AI - DRIVE Labs Ep. 36

Gaussian Naive Bayes, Clearly Explained!!!

P2 什么是机器学习【2024公认最好的 | 吴恩达机器学习 | 教程 | Machine Learning Specialization(超爽中英!)】

What is Inattentional Blindness? (Easiest Explanation)

Bỏ học để khởi nghiệp có phải lối tắt?

The GELU, SiLU and SwiGLU activation functions, clearly explained!!!

The End of Human Mathematics? AI's Recent Breakthrough -- And What's Next

LET THERE BE... Voltage? | Maxwell's Equation #2 Explained for Beginners