Test

Why does regularization help with overfitting? Why does it help with reducing variance problems? Let's examine a few examples to develop an understanding of how it functions. So, remember that the images depicting high bias, high variance, and "just right" from our previous blog appeared as follows.

Let's consider a large and deep neural network. Although this one may not be depicted as particularly large or deep, it's worth examining whether such a neural network is currently overfitting. So you have some cost function, write J of W, b equals sum of the losses,

 And so what we did for regularization was add this extra term that penalizes the weight matrices from being too large. And we said that was the Frobenius norm.

So why is it that shrinking the L2 norm, or the Frobenius norm with the parameters might cause less overfitting?

• One piece of intuition is that if you significantly increase your regularization lambda, it will strongly incentivize setting the weight matrices to be very small or close to zero. W, to be reasonably close to zero.
• So one piece of intuition is maybe it'll set the weight to be so close to zero for a lot of hidden units that's basically zeroing out a lot of the impact of these hidden units. And if that's the case, then, you know, this much simplified neural network becomes a much smaller neural network.

In fact, it is almost like a logistic regression unit, you know, but stacked multiple layers deep. And so that will take you from this overfitting case, much closer to the left, to the other high bias case But, hopefully, there'll be an intermediate value of lambda that results in the result closer to this "just right" case in the middle.

The intuition is that by significantly increasing lambda, it will push W close to zero. However, in practice, this doesn't actually happen. Instead, it reduces the impact of many hidden units, resulting in a simpler network that behaves more like logistic regression.

 The intuition of completely zeroing out a bunch of hidden units isn't quite right. What actually happens is that all the hidden units are still used, but each of them has a much smaller effect. However, you do end up with a simpler network, which is less prone to overfitting, as if you have a smaller network.

Here's another attempt at additional intuition for why regularization helps prevent overfitting. And for this, I'm going to assume that we're using the tan h activation function, which looks like this. This is g of z equals tan h of z.

So basically, if z is really small and only varies within a small range of values, then you're just working with the linear part of the tan h function.

As the input values to an activation function become larger or smaller, the function often becomes less linear. This non-linearity is crucial because it allows neural networks to model complex relationships and learn from non-linear patterns in the data.

If the regularization parameter, lambda, is large, your parameters will be relatively small because the cost function penalizes large parameter values. This helps prevent overfitting by keeping the model simpler and more generalizable.

. And so if the weights, W, are small, then because z is equal to W, right, and then technically, it's plus b. But if W tends to be very small, then z will also be relatively small.

And in particular, if z ends up taking relatively small values, just in this little range, then g of z will be roughly linear. So it's as if every layer will be roughly linear, as if it is just linear regression.  if every layer in a neural network is linear, the entire network remains linear. This means that even a very deep network with linear activation functions can only compute a linear function in the end. So, it's not capable of fitting those highly complex, non-linear decision boundaries that lead to overfitting, as we observed in the high variance case on the previous slide.

 In summary, if the regularization parameters are very large, the parameters \( W \) will be very small, making \( z \) relatively small as well (ignoring the effects of \( b \) for now). This means \( z \) will take on a small range of values. Consequently, if the activation function is \( \tanh \), it will behave relatively linearly. As a result, your neural network will compute something close to a large linear function, which is a simpler function compared to a highly complex, non-linear function and so is also much less able to overfit.

. when implementing regularization, we took our definition of the cost function J and we actually modified it by adding this extra term that penalizes the weights being too large. When implementing gradient descent, one important debugging step is to plot the cost function \( J \) as a function of the number of iterations of gradient descent. You should observe that the cost function \( J \) decreases monotonically with each iteration, indicating that the algorithm is converging properly.

This helps ensure that your gradient descent implementation is working correctly.. And if you're implementing regularization, then please remember that J now has this new definition. If you plot the old definition of J, just this first term, then you might not see a decrease monotonically. So to debug gradient descent, make sure that you're plotting, you know, this new definition of J that includes this second term as well. Otherwise, you might not see J decrease monotonically on every single elevation. So that's it for L2 regularization, which is actually a regularization technique that I use the most in training deep learning models. In deep learning, there is another sometimes used regularization technique called dropout regularization. Let's take a look at that in the next blog.