쉐도잉 연습: The GELU, SiLU and SwiGLU activation functions, clearly explained!!! - 영상으로 영어 말하기 배우기
레슨 만드는 중...
1
Sometimes drop out is all you need.
2
Hooray!
3
StatQuest.
4
Hello, I'm Josh Starmer and welcome to StatQuest.
5
Today we're going to talk about the GelU, SilU, and Swiglu activation functions, and they're going to be clearly explained.
6
This StatQuest is brought to you by the letters A, B, and C.
7
A always, B be, C curious, always be curious.
8
Too long, didn't watch.
9
If you're already familiar with the REL U activation function, then the GAL U activation function,
10
which sounds like it could be some sort of expletive, but is short for Gaussian Error Linear Unit, and the CIL U activation function,
11
which is short for Sigmoid Linear Unit and thus sound sort of like an oxymoron, are almost the same.
12
One relatively subtle difference is that the RELU outputs zero for all negative input values.
13
And the GELU and CELU have this little dip for input values between negative two and zero.
14
Another subtle difference is that the RELU has this angular bend at zero.
15
And the Gell-U and Sill-U have a softer, differentiable curve at zero.
16
Lastly, for all positive input values, the Rell-U just outputs the original input value.
17
While the Gell-U and Sill-U have a slight curve
18
that makes the output values just a little smaller than the positive input values.
19
In contrast, the SWIGLU, which stands for Swish Gated Linear Unit, which sounds like a random jumble of words,
20
can look like a smooth version of the RELU, or it can look like pretty much anything else it wants to look like.
21
That said, despite the glaring differences in the shapes that the SWIGLU can take on, it's closely related to the GELU and CILU.
22
Anyway, as a result of these relatively subtle differences for the Gelu and Siyu,
23
and the relatively extreme differences for the Swiglu,
24
these activation functions do not tend to overfit training data as much as the Relu.
25
In other words, if all these dots were the training data,
26
then a large neural network using the Relu activation function would have a strong tendency to overfit the training data.
27
In contrast, a large neural network using the GelU,
28
SilU, or Swiglu activation functions does not have as strong a tendency to overfit the training data.
29
Thus, this new class of activation functions tend to result in neural networks that do a better job making predictions after training.
30
Baaam.
31
But Josh, where do these new activation functions get their different shapes from?
32
And how did anyone come up with them to begin with?
33
Well, just for you, Squatch, and anyone else who's interested, I made this whole stat quest just to answer those very questions.
34
Hooray!
35
Way back in the days before 2010, when you built a neural network, you usually did it using sigmoid activation functions.
36
At the time, sigmoid activation functions seemed like a great idea because their output is constrained to be between 0 and 1,
37
which represented the outputs of actual neurons either not firing or firing.
38
And the curve reflected how neurons transition from one state to the other.
39
Lastly, the sigmoid activation function is continuous and differentiable for all input values, making it a reasonable choice for backpropagation.
40
And, for simple neural networks like this one, sigmoid activation functions work great.
41
However, once people started scaling up their neural networks to have more hidden layers and more activation functions in each hidden layer,
42
it became harder and harder to train the model.
43
Thus, when we use the sigmoid activation function, we can't build very large neural networks.
44
The reason for this size limitation has to do with the derivative of the sigmoid shape, which is needed to fit the neural network to data.
45
Specifically, as the input for the sigmoid activation function gets further and further from zero,
46
the derivative gets closer and closer to zero.
47
And a super small value for the derivative means gradient descent, or whatever else we're using to fit the neural network to training data,
48
takes super small steps towards the optimal parameter values.
49
And that means finding the optimal parameter values can take a long, long time.
50
As a result, if these were the training data, then, ideally, we'd want to fit a squiggly shape like this to it.
51
However, because we can't easily train a large neural network using the sigmoid activation function, we might get a squiggly shape like this instead.
52
The squiggly shape from the sigmoid underfits the training data.
53
And thus, the squiggly shape created with the sigmoid activation function doesn't fit the data as well as we might like.
54
So, around 2010, the sigmoid activation function was replaced by the rel U, this bent shape.
55
And the rel U activation function proved to be fundamentally important to making deep neural networks trainable.
56
Ultimately, in 2017, the REL-U achieved peak awesomeness by being part of the first transformer.
57
Bam.
58
The REL-U was great because, among other things, it was super easy to compute.
59
If the input value, the x-axis coordinate, was negative, then the output, the corresponding y-axis coordinate, was 0.
60
And if the input value was positive, then the output, the corresponding y-axis value, was the exact same value.
61
Another advantage was that the derivative was defined to either be 0 or 1.
62
Having a derivative of 1 for large input values helped simplify backpropagation
63
and made it easier to optimize the weights and biases in the model.
64
Unlike the sigmoid activation function that had a super small derivative for input values far from zero,
65
the ReLU has a derivative of one regardless of how much greater the input value is compared to zero.
66
And that means, at least for large input values, the ReLU can optimize parameter values relatively quickly.
67
Boom!
68
Nope, not yet.
69
Unfortunately, the RELU has its own problem.
70
First, remember that we had this training data
71
and we showed the ideal squiggly shape that we wanted our neural network to fit to the data.
72
And, unfortunately, when we used the sigmoid activation function, instead of the ideal squiggly shape,
73
we got a squiggly shape that underfit the training data.
74
Well, the problem with using the RELU activation function is
75
that now we can create a huge neural network that makes a squiggly shape that overfits the training data.
76
In other words, the large neural networks
77
that we can create with the RELU activation function often overcorrect for the problem we had with the sigmoid.
78
Can't you just make a smaller neural network with the RELU?
79
Great question, Squatch.
80
In theory, we could do that, but in practice, it's next to impossible to figure out the optimal size for a neural network.
81
I mean, imagine you started with a network that had 100,000 activation functions.
82
Are you going to then test to see if 99,999 activation functions is better?
83
And then test 99,998?
84
Etc?
85
Okay, I get your point.
86
It would take way too much time.
87
So, ideally, it would be nice if we could have something
88
that didn't underfit the data like a small neural network using the sigmoid,
89
and didn't overfit the data like a large neural network that uses the ReLU.
90
One early solution to this problem was something called dropout.
91
During training, when we are fitting a neural network to a dataset,
92
Dropout randomly selects a subset of activation functions and removes them from the network.
93
And then applies backpropagation to this smaller network to update the weights and biases.
94
Once the weights and biases are updated,
95
it goes back to the original neural network and selects another random subset of activation functions to remove from the training step.
96
And then applies backpropagation to this smaller network to update the weights and biases.
97
And this process is repeated for the duration of training.
98
As a result of using Dropout during training, instead of training one big neural network,
99
we effectively train lots of smaller networks.
100
Thus, after training, when it comes time to make predictions with the entire neural network,
101
we're essentially averaging over all of the smaller neural networks that we created during training.
102
And the model averaging from the rail U combined with dropout results in a squiggly shape
103
that does a better job approximating the ideal shape.
104
Bam.
105
Nope, not yet.
106
Unfortunately, combining the rail U with dropout has its own problems.
107
The first problem is that dropout is an extra thing that we need to add to the training stage.
108
And the second, perhaps more important problem,
109
is that dropout is completely independent of the inputs to each activation function in the network. In other words,
110
dropout will remove an activation function from a neural network regardless of whether the input signal is large or small.
111
Ugh, how do we fix that?
112
Well, that's where the Galu, Silu, and Swiglu activation functions come in.
113
BAM!
114
Remember, at the start of this stat quest, we saw what the Galu, Silu, and Swiglu activation functions looked like.
115
And we learned that they did a better job fitting a squiggly shape to the data.
116
Well, now we'll learn where these activation functions come from.
117
And to do this, we'll We'll learn how to build them from scratch.
118
We'll start with a super simple neural network that just has one input, one output, and nothing else.
119
Josh, is that even a neural network?
120
It's enough to show how the Galu and Silu activation functions were created.
121
Okay.
122
Now, even though this neural network is ridiculously simple, let's imagine we apply dropout to it.
123
If we apply dropout to this neural network, then, sometimes, an input value, like negative 2, will make it to the output.
124
And sometimes, the input value will not make it to the output, and the output value will be 0.
125
Now, if we apply dropout 50% of the time, then we can calculate the average output value
126
by multiplying the top output value by 0.5 and multiplying the bottom output value by 0.5 and adding those two products together.
127
In this case, the average output value is negative 1.
128
Bam.
129
Now let's see what happens when we use the input value x to determine the probability
130
that we will avoid dropout p of x.
131
Specifically, we'll use this blue squiggle to convert the input value x into a probability
132
that we will avoid dropout p of x.
133
Note, if you're dying to know more details about this blue squiggle,
134
It's the cumulative distribution function for a standard Gaussian curve.
135
However, the important thing is that we have a squiggle that goes from 0 to 1.
136
In this case, if the input, x, is a negative number, then the probability that we'll avoid dropout, p of x, will be closer to 0.
137
And if the input is a positive number, the probability that we will avoid dropout will be closer to 1.
138
For example, if the input value is negative 2, then, according to this blue squiggle,
139
the probability we will avoid dropout is 0.02.
140
So we plug 0.02 in for P of X, the probability for avoiding dropout.
141
And we plug in 1 minus 0.02, or 0.98, into the probability that we will use dropout.
142
Note, because the probabilities we avoid dropout, 0.02, or use dropout, 0.98,
143
are different, we're now calculating a weighted average of the output values.
144
Anyway, in this case, when the input value is negative 2, the weighted average is negative 0.04.
145
So we can put a dot at
146
on a graph that has input values on the x-axis and the weighted average of the output values on the y-axis.
147
Now, if the input value is negative 1, then the probability we will avoid dropout is 0.16.
148
So we plug 0.16 into the probability for avoiding dropout.
149
And we plug 1 minus 0.16 or 0.84 into the probability that we will use dropout.
150
And the weighted average is negative 0.16.
151
So we can put a dot at negative 1 comma negative 0.16.
152
Likewise, we can put dots at 0 comma 0.
153
1, 0.84, and 2, 1.96.
154
And, in general, when we allow the probability for avoiding dropout to be determined by this blue squiggle, we end up with this function.
155
And now, instead of plugging numbers into this neural network with dropout, we can use this new function as an activation function.
156
In other words, this neural network with dropout and probabilities can be simplified to this neural network with a single activation function.
157
And that means that our new activation function has the effects of input-dependent dropout built right into it.
158
Because it was derived from the weighted averages that input-dependent dropout creates.
159
Lastly, because this blue squiggle is the cumulative distribution function for a Gaussian distribution,
160
this activation function is called GalU for Gaussian Error Linear Unit.
161
Double bam!
162
Now that we know how the GalU activation function can be derived entirely from input-dependent dropout,
163
Let's go back to the larger neural network to derive the general equation for the gelu and associated activation functions.
164
First, let's just plug in x into the input.
165
Now, when we just do the math, we get...
166
x times p of x plus 0 times 1 minus p of x.
167
And because the second term is multiplied by zero, just like an activation function that gets dropped out, it goes away.
168
And we're left with the input value, x, times the probability we avoid dropout, P . In other words,
169
input values from negative infinity to positive infinity times their corresponding
170
y-axis coordinate on a curve like this give us an activation function function.
171
Now that we have a general equation for creating activation functions with input dependent dropout,
172
one cool thing we can do is swap out the blue squiggle
173
that we're using for P of X and get a related, but still different, activation function.
174
For example, remember the sigmoid activation function we started with long ago, but ultimately replaced with ReLU?
175
Well, compared to the cumulative distribution function for a Gaussian curve, the sigmoid shape, the orange squiggle, isn't that different.
176
when we multiply input values by their corresponding y-axis coordinates on the orange squiggle, we get this new activation function.
177
And since this new activation function is based on the sigmoid shape,
178
this activation function is called the SilU for sigmoid linear unit.
179
When we plot the GelU and the SilU activation functions on the same graph, we see that the differences are not super extreme.
180
And just as we might expect by the similarity in the shapes, both can work in similar contexts.
181
That said, as of this quest creation, it seems that Google favors the GelU for its AI models.
182
Noted.
183
Now let's talk about the SWEGLU activation function.
184
To do that, let's go back to our super simple neural network with Dropout.
185
And make it fancy.
186
First, we'll add a weight.
187
Josh, that's not very fancy.
188
Don't worry, Squatch, we'll get there.
189
Anyway, now when we plug a value, X, into the input, we get WX on the output on top.
190
And, because we still have dropout on the bottom, we have zero in the output down there.
191
Now we plug the output, wx, into the sigmoid probability function.
192
And that gives us a weighted average like before.
193
And, if we stopped right now, we'd have something very close to an activation function called swish.
194
The big difference between the weighted average we have now and the swish is the swish just has the input value,
195
x, outside of the probability function instead of wx.
196
So, instead, we'll call our weighted average swishish.
197
Also, while we're at it, most people, including the maintainers of the PyTorch documentation,
198
just use SILU to refer to both SILU and swish.
199
Anyway, regardless of what it's called, we're not going to stop here.
200
Instead, we'll add another copy of our input value, x, and multiply it by a different weight, v.
201
Lastly, we multiply vx by the weighted average, the swish-ish thing, to get the final output value, y.
202
Whoa, that's pretty fancy.
203
Agreed.
204
Because this bypass multiplication term, Vx, which is used by another activation function called a gated linear unit,
205
or glue, is combined with something that is swish-ish, we call the whole thing the swiglu activation function.
206
Boom!
207
These additional weights and multiplication add a lot more flexibility to the shapes that we can get for the activation functions.
208
But Josh, why would this be better?
209
Well, to quote the authors of the SuiGlu activation function, we offer no explanation as to why these architectures seem to work.
210
Small bam.
211
That said, I have two ideas for why they work.
212
First, the SuiGlu requires a lot more computation than the ReLU.
213
As a result, we tend to make smaller neural networks with Swiglu.
214
And a smaller neural network may not overfit the training data as much as a larger one.
215
Second, and perhaps more importantly, throughout the history of neural networks, one recurring theme is that the more we,
216
as people, get out of the way and don't tell the network what to do, the better they perform.
217
And with the Swiglu activation function, with its extra trainable parameters, we're essentially saying, Neural network,
218
you figure out what shape you need for the activation function.
219
Anyway, since it seems to work, the Swiglu variation is commonly used in AI, especially AI models created by Meta.
220
In summary, the idea for these activation functions is that they are all based on the weighted average of dropout.
221
And, if anything, the big lesson here is that neural networks are still an area of very active experimentation.
222
People are trying new things all the time and seeing what happens.
223
If it works, then they publish their new ideas.
224
Triple bam!
225
Now it's time for some shameless self-promotion.
226
If you want to review statistics, machine learning, and AI offline, check out the StatQuest PDF study guides and my best-selling books on machine learning,
227
neural networks and AI, and statistics at StatQuest.org.
228
There's something for everyone.
229
Hooray!
230
We've made it to the end of another exciting StatQuest.
231
If you like this StatQuest and want to see more, please subscribe.
232
And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie,
233
or just donate.
234
The links are in the description below.
235
Alright, until next time, quest on!
📺 같은 채널
✨ 추천 영상
C11
74 문장
LangChain in 5 Minutes (Explained Clearly)
Business EnglishUniversity StudentsWorking Adults
C181 문장
Mathematician Collapses All Functions to One Weird Formula
Business EnglishUniversity StudentsWorking Adults
C1124 문장
What Makes People Successful? | Angela Duckworth
Business EnglishUniversity StudentsWorking Adults
C12
280 문장
Siemens S7-1200: Signal Modules - Wire, Program, Test
Business EnglishUniversity StudentsWorking Adults
C12
764 문장
11. Introduction to Machine Learning
Business EnglishUniversity StudentsWorking Adults
C11
132 문장
Video
Business EnglishUniversity StudentsWorking Adults
C113
202 문장
SAC | Soft Actor Critic (SAC) architecture | SAC Explained
Business EnglishUniversity StudentsWorking Adults
C12
182 문장
We Let $200 GPT-Pro Think for 7 Days on Unsolved Math
Business EnglishUniversity StudentsWorking Adults
C18
55 문장
When AIs act emotional
Business EnglishUniversity StudentsWorking Adults
C12
234 문장
Cursed Units 2: Curseder Units
Business EnglishUniversity StudentsWorking Adults
C11
506 문장
Analyzing Statements of Cash Flows 1 (2025 CFA® Level I Exam – FSA – Learning Module 4)
Cambridge Exam PrepBusiness EnglishUniversity Students
이 영상을 이용해 영어 말하기 실력을 키우는 이유
영상 영어 공부는 실제 대화의 리듬과 발음, 자연스러운 표현을 배우는 데 최적입니다. 이 비디오는 기술적 주제를 쉽게 설명하는 과정에서 명료한 논리 구조와 다양한 문장 패턴을 사용하므로, shadow speech나 shadowspeak 연습에 적합합니다. 특히 IELTS 스피킹과 같은 시험에서 필요한 '복잡한 주제를 명확히 전달하는 능력'을 기르는데 도움이 됩니다. 발화의 템포와 강세를 따라하며 말하기 연습을 하면, 자연스러운 영어 표현이 익숙해질 뿐만 아니라, 기술 용어도 스스로 사용할 수 있게 됩니다.
문맥 속 문법과 표현
- "Too long, didn't watch.": 구어체에서 자주 사용되는 축약 표현으로, 긴 내용을 간략히 요약할 때 유용합니다. "너무 길어서, 안 봤어"라는 뜻으로, 대화에서 친밀감을 주거나 빠르게 의견을 전달할 때 사용됩니다.
- "the GELU and CELU have this little dip for input values between negative two and zero": "~에 대해"라는 의미의 전치사 'for'와 범위를 나타내는 'between ~ and' 구조가 결합되었습니다. 기술적 설명에서 수치 범위나 특정 조건을 설명할 때 자주 사용되므로, 이 패턴을 익히면 복잡한 내용도 명확히 전달할 수 있습니다.
- "the reason for this size limitation has to do with the derivative of the sigmoid shape": "~와 관련이 있다"는 뜻의 'has to do with'는 원인을 설명할 때 자연스러운 표현입니다. 논리적 연결을 강조해야 하는 글 또는 말에서 자주 사용되므로, IELTS 스피킹에서 논증 구조를 강화할 수 있습니다.
발음의 함정
이 영상에서는 기술 용어의 발음이 주요 난점입니다. "GELU"는 "제루"가 아닌 /ˈdʒiːluː/, "SiLU"는 /ˈsɪluː/로 발음해야 합니다. 특히 "sigmoid"는 /sɪɡˈmɔɪd/로, 'g'가 강하게 발음되지 않고 'm'이 강세를 받습니다. 또한 "backpropagation"은 /ˌbækprɒpəˈɡeɪʃn/으로, 중간의 'prop' 부분이 /prɒp/로 발음됩니다. Shadow speak 연습 시 이런 용어들의 발음을 정확히 따라야 하며, 단어 사이의 연결음(linking)도 주의해야 합니다. 예를 들어 "neural network"는 /ˈnjʊərəl ˈnetwɜːk/가 아닌 /ˈnjʊərəlnetwɜːk/로 발음되어 두 단어가 자연스럽게 연결됩니다.
이 영상의 문법
화자가 가장 많이 쓰는 문형을 영상 속 실제 표현과 함께 정리했습니다.
| 문형 | 영상 속 표현 |
|---|---|
| 조건문 if + 절, will/would + 동사 — 조건과 그 결과 | if we stopped right now, we'd have |
| 수동태 be + 과거분사 — 누가 하는지보다 무슨 일이 일어나는지에 초점 | is brought · is constrained · is needed |
쉐도잉이란? 영어 실력을 빠르게 키우는 과학적 방법
쉐도잉(Shadowing)은 원래 전문 통역사 훈련을 위해 개발된 언어 학습 기법으로, 다언어 학자인 Dr. Alexander Arguelles에 의해 대중화된 방법입니다. 핵심 원리는 간단하지만 매우 강력합니다: 원어민의 영어를 들으면서 1~2초의 짧은 지연으로 즉시 소리 내어 따라 말하는 것——마치 '그림자(shadow)'처럼 화자를 따라가는 것입니다. 문법 공부나 수동적인 청취와 달리, 쉐도잉은 뇌와 입 근육이 동시에 실시간으로 영어를 처리하고 재현하도록 훈련합니다. 연구에 따르면 이 방법은 발음 정확도, 억양, 리듬, 연음, 청취력, 말하기 유창성을 크게 향상시킵니다. IELTS 스피킹 준비와 자연스러운 영어 소통을 원하는 분들에게 특히 효과적입니다.