test

Introduction 

When implementing backpropagation, there's an essential test called gradient checking that can help you ensure that your backpropagation is functioning correctly. Sometimes, after writing out all the equations, it’s hard to be 100% sure if every detail is right. Gradient checking helps you verify whether the implementation is correct or if there are hidden errors that need to be addressed.

Understanding Numerical Gradient Approximation

Before diving into gradient checking, let’s first explore how to numerically approximate gradients. This involves using a small change, or "epsilon," to compute gradients and compare them to the results from backpropagation. We’ll start with an example function to better understand this process.

Numerical Approximation with a Simple Function

Let’s take a function $f(\theta) = \theta^3$  Suppose we start with a value of $\theta = 1$. Instead of just nudging $\theta$ to the right (e.g., $\theta + \epsilon$), we’ll nudge it both to the right and left—creating $\theta + \epsilon$ and $\theta - \epsilon$. For example, with $\theta = 1$, we can calculate values for $\theta +ϵ=\mathrm{1.01<span\; style="font-family:\; arial,\; helvetica,\; sans-serif;\; font-size:\; medium;">and </span>}$$\theta - \epsilon = 0.99$, were $\epsilon = 0.01$.

 

The Two-Sided Difference Formula for Better Accuracy

Rather than using just one side (as in a one-sided difference), the two-sided difference formula provides a more accurate estimate of the gradient. This involves calculating the difference between $f(\theta +ϵ)$ and $f(\theta -ϵ)$, then dividing by $2ϵ$. Mathematically, this can be written as:

$$\frac{f(\theta +ϵ)-f(\theta -ϵ)}{2ϵ}$$

This two-sided approach takes into account both sides of the point, resulting in a more precise gradient estimate compared to the one-sided method.

it is 0.01. It turns out that rather than taking this little triangle and computing the height over the width, you can get a much better estimate of the gradient if you take this point, f of theta minus epsilon and this point, and you instead compute the height over width of this bigger triangle. So for technical reasons which I won't go into, the height over width of this bigger green triangle gives you a much better approximation to the derivative at theta.

And you saw it yourself, taking just this lower triangle in the upper right is as if you have two triangles, right? This one on the upper right and this one on the lower left. And you're kind of taking both of them into account by using this bigger green triangle. So rather than a one-sided difference, you're taking a two-sided difference.

Example Calculation

For our function $f(\theta )={\theta }^3$, if $\theta =1$ and $ϵ=0.01$, the values of $f(\theta +ϵ)$ and $f(\theta -ϵ)$ can be calculated as follows:

• $f(1.01)=(1.01{)}^3$
• $f(0.99)=(0.99{)}^3$

Using the two-sided difference formula:

$$\frac{f(1.01)-f(0.99)}{2\times 0.01}=3.0001$$

The theoretical gradient at $\theta =1$ is $g(\theta )=3{\theta }^2$, which equals 3. Comparing the results, the error is extremely small, just 0.0001, showing that the two-sided difference method is highly accurate.

So let's work out the math. This point here is F of theta plus epsilon. This point here is F of theta minus epsilon. So the height of this big green triangle is f of theta plus epsilon minus f of theta minus epsilon. And then the width, this is 1 epsilon, this is 2 epsilon. So the width of this green triangle is 2 epsilon. So the height of the width is going to be first the height, so that's F of theta plus epsilon minus F of theta minus epsilon divided by the width. So that was 2 epsilon which we write that down here. And this should hopefully be close to g of theta. So plug in the values, remember f of theta is theta cubed. So this is theta plus epsilon is 1.01. So I take a cube of that minus 0.99 theta cube of that divided by 2 times 0.01. You should get that this is 3.0001.

Whereas from the previous slide, we saw that g of theta, this was 3 theta squared so when theta was 1, so these two values are actually very close to each other. The approximation error is now 0.0001. Whereas on the previous slide, we've taken the one sided of difference just theta + theta + epsilon we had gotten 3.0301 and so the approximation error was 0.03 rather than 0.0001. So this two sided difference way of approximating the derivative you find that this is extremely close to 3. And so this gives you a much greater confidence that g of theta is probably a correct implementation of the derivative of F.

Why the Two-Sided Difference Works Better

Using the two-sided difference formula gives a much closer approximation to the true gradient, with significantly less error than the one-sided method. For a one-sided difference, the error is around 0.03, whereas for the two-sided difference, it drops to 0.0001. This method provides much greater confidence that the gradient calculation is correct.

When you use this method for grading, checking and back propagation, this turns out to run twice as slow as you were to use a one-sided defense. It turns out that in practice I think it's worth it to use this other method because it's just much more accurate.

Theoretical Insight: Why Two-Sided Differences Are More Accurate

For those familiar with calculus, the formal definition of a derivative involves taking the limit as epsilon approaches zero. The two-sided difference formula:

$$\frac{f(\theta +ϵ)-f(\theta -ϵ)}{2ϵ}$$

is a closer approximation to the derivative because the error in this method is proportional to ${ϵ}^2$, which is much smaller than the error in the one-sided difference, which is proportional to $ϵ$.

The little bit of optional theory for those of you that are a little bit more familiar of Calculus, it turns out that, and it's okay if you don't get what I'm about to say here. But it turns out that the formal definition of a derivative is for very small values of epsilon is f of theta plus epsilon minus f of theta minus epsilon over 2 epsilon. And the formal definition of derivative is in the limits of exactly that formula on the right as epsilon those as 0.

And the definition of unlimited is something that you learned if you took a Calculus class but I won't go into that here. And it turns out that for a non zero value of epsilon, you can show that the error of this approximatio

n is on the order of epsilon squared, and remember epsilon is a very small number. So if epsilon is 0.01 which it is here then epsilon squared is 0.0001. The big O notation means the error is actually some constant times this, but this is actually exactly our approximation error. So the big O constant happens to be 1. Whereas

in contrast if we were to use this formula, the other one, then the error is on the order of epsilon. And again, when epsilon is a number less than 1, then epsilon is actually much bigger than epsilon squared which is why this formula here is actually much less accurate approximation than this formula on the left. Which is why when doing gradient checking, we rather use this two-sided difference when you compute f of theta plus epsilon minus f of theta minus epsilon and then divide by 2 epsilon rather than just one sided difference which is less accurate. If you didn't understand my last two comments, all of these things are on here. 

Gradient Checking in Backpropagation

When using gradient checking in backprop

agation, the process may run slower (approximately twice as slow) due to the extra computations from the two-sided difference method. However, this trade-off is worth it for the increased accuracy. The two-sided difference ensures that your gradient calculations are precise, giving you confidence in the correctness of your backpropagation implementation.

Conclusion: The Power of Gradient Checking

By using the two-sided difference method for gradient checking, you can numerically verify whether your function $g(\theta )$ is a correct implementation of the derivative of a function $f(\theta )$. This method can be a powerful tool to ensure your backpropagation code is correct and free of bugs. In the next steps, we’ll explore how to apply gradient checking to your machine learning projects, helping you debug and improve your models with confidence.