How Gradient Checking Can Save You Time and Help Debug Neural Networks

Gradient checking is a technique that has saved me a lot of time and helped me catch numerous bugs in my backpropagation implementations. In this post, I'll explain how gradient checking works and how you can use it to debug or verify that your neural network's gradients are correct.

Sections

• Understanding Neural Network Parameters
• Reshaping the Gradients
• How to Perform Gradient Checking (Grad Check)
• Comparing the Gradients
• Interpreting the Results
• Real-World Example
• Python Code Example
• Conclusion

Understanding Neural Network Parameters

In a neural network, you have parameters such as W1, B1, all the way to WL and bL, which represent the weights and biases in each layer. To perform gradient checking, the first thing you need to do is reshape all these parameters into a single large vector called theta.

For example, take each W matrix and reshape it into a vector. Do this for all W’s, then concatenate them to form the giant vector theta.

Once you’ve done this, your cost function J, which originally depended on the W’s and B’s, will now just depend on this single vector, theta.

Reshaping the Gradients

Next, you'll apply the same reshaping and concatenation process to the gradients. You’ll take the gradients dW[1], db[1], and so on, and reshape them into a giant vector called d_theta, which has the same dimensions as theta.

For instance, dW[1] has the same dimensions as W1, and db[1] is already a vector. By reshaping and concatenating all the derivatives, you get the giant vector d_theta, which represents the gradients of the cost function J with respect to theta.

How to Perform Gradient Checking (Grad Check)

So the question is, now, is the theta the gradient or the slope of the cos function J? So here's how you implement gradient checking, and often abbreviate gradient checking to grad check

Now that you have theta and d_theta, you can start gradient checking, often abbreviated as "grad check." The idea is that the cost function J is now a function of the components of theta (theta1, theta2, theta3, etc.). To perform the check:

1. For each component theta_i of the vector, compute an approximation of the gradient using a two-sided difference:

   • First, increase theta_i by a small value epsilon and compute J(theta).
   • Then, decrease theta_i by epsilon and compute J(theta) again.
   • Use these two values to approximate the gradient at theta_i.

2. Repeat this process for each component of theta to get a vector of approximate gradients, d_theta_approx, which will have the same dimensions as d_theta.

So to implement grad check, what you're going to do is implements a loop so that for each I, so for each component of theta, let's compute D theta approx i to b. And let me take a two sided difference. So I'll take J of theta. Theta 1, theta 2, up to theta i. And we're going to nudge theta i to add epsilon to this. So just increase theta i by epsilon, and keep everything else the same. And because we're taking a two sided difference, we're going to do the same on the other side with theta i, but now minus epsilon. And then all of the other elements of theta are left alone. And then we'll take this, and we'll divide it by 2 theta.

Comparing the Gradients

Once you have both d_theta and d_theta_approx, you’ll need to check if they are similar. To do this, calculate the Euclidean distance (L2 norm) between the two vectors: $\frac{{||d\_theta\_approx - d\_theta||}}{{||d\_theta\_approx|| + ||d\_theta||}}$

This formula helps you normalize the difference, making it easier to compare the two vectors. The smaller this value, the more accurate your gradient approximation is likely to be.

And what we saw from the previous post is that this should be approximately equal to d theta i. Of which is supposed to be the partial derivative of J or of respect to, I guess theta i, if d theta i is the derivative of the cost function J. So what you going to do is you're going to compute to this for every value of i. And at the end, you now end up with two vectors. You end up with this d theta approx, and this is going to be the same dimension as d theta. And both of these are in turn the same dimension as theta. And what you want to do is check if these vectors are approximately equal to each other.

Interpreting the Results

So, in detail, well how you do you define whether or not two vectors are really reasonably close to each other? What I do is the following. I would compute the distance between these two vectors, d theta approx minus d theta, so just the o2 norm of this. Notice there's no square on top, so this is the sum of squares of elements of the differences, and then you take a square root, as you get the Euclidean distance. And then just to normalize by the lengths of these vectors, divide by d theta approx plus d theta. Just take the Euclidean lengths of these vectors. And the row for the denominator is just in case any of these vectors are really small or really large, your the denominator turns this formula into a ratio.

• If the result is less than 10^-7, your gradient computation is likely correct.
• If it’s around 10^-5, you may want to review the gradients carefully to ensure everything is okay.
• If the result is larger than 10^-3, this indicates a potential bug, and you should inspect your implementation closely.

After debugging and refining your code, if you get a small value from this check, it’s a strong indication that your implementation is correct.

So we implement this in practice, I use epsilon equals maybe 10 to the minus 7, so minus 7. And with this range of epsilon, if you find that this formula gives you a value like 10 to the minus 7 or smaller, then that's great. It means that your derivative approximation is very likely correct. This is just a very small value.

If it's maybe on the range of 10 to the -5, I would take a careful look. Maybe this is okay. But I might double-check the components of this vector, and make sure that none of the components are too large.

And if some of the components of this difference are very large, then maybe you have a bug somewhere. And if this formula on the left is on the other is -3, then I would wherever you have would be much more concerned that maybe there's a bug somewhere. But you should really be getting values much smaller then 10 minus 3. If any bigger than 10 to minus 3, then I would be quite concerned. I would be seriously worried that there might be a bug. And I would then, you should then look at the individual components of data to see if there's a specific value of i for which d theta across i is very different from d theta i. And use that to try to track down whether or not some of your derivative computations might be incorrect. And after some amounts of debugging, it finally, it ends up being this kind of very small value, then you probably have a correct implementation. So when implementing a neural network, what often happens is I'll implement foreprop, implement backprop. And then I might find that this grad check has a relatively big value. And then I will suspect that there must be a bug, go in debug, debug, debug. And after debugging for a while, If I find that it passes grad check with a small value, then you can be much more confident that it's then correct. 

Real-World Example

When I’m implementing a neural network, I typically implement forward propagation and backpropagation first, and then run gradient checking. If the value from the check is too large, I know there’s a bug, and I go back to debug. Once the gradient check produces a small value, I can be confident that my network is working as intended.

Conclusion

Gradient checking is an invaluable tool that has helped me catch bugs in my neural network implementations, and it can help you too. It’s a simple but effective way to verify the correctness of your gradients, especially when dealing with complex models.

In the next post, I’ll share some tips on how to implement gradient checking more efficiently. Stay tuned!