Introduction 

In this article, we'll cover some essential tips for implementing gradient checking, a useful debugging technique for neural networks. If you’re new to gradient checking or looking for insights on effective use, this guide will help you understand best practices for incorporating this approach into your workflow.

First, don't use grad check in training, only to debug. 

1. Use Gradient Checking for Debugging Only

Gradient checking is a great tool for ensuring the accuracy of your gradient computations, but it’s crucial to use it only for debugging, not during training. Calculating the approximate gradient $d\theta_{\text{approx}}$ for every parameter can be extremely time-consuming, making it unsuitable for real-time training. Instead, rely on backpropagation to compute the gradients during training, and only use gradient checking to confirm your results. Once you’ve verified that $d\theta$ (backprop-computed gradients) is close to $d\theta_{\text{approx}}$, you can disable gradient checking.

So what I mean is that, computing d theta approx i, for all the values of i, this is a very slow computation. So to implement gradient descent, you'd use backprop to compute d theta and just use backprop to compute the derivative. and don't run this during every iteration of gradient descent, because that's just much too slow.

2. Isolate Bugs by Examining Gradient Components

If your gradient check fails—meaning $d\theta_{\text{approx}}$ differs significantly from $d\theta$—investigate the individual components of $d\theta$. For example, check if certain elements of $d\theta_{\text{approx}}$ are much further from their corresponding $d\theta$ values. what I would do is look at the different values of i to see which are the values of d theta approx that are really very different than the values of d theta. If discrepancies appear mainly in components of $d\theta_b$ (related to biases) rather than $d\theta_w$ (related to weights), the issue may be with your bias calculations. By narrowing down which components fail, you can hone in on potential bugs more efficiently. So for example, if you find that the values of theta or d theta, they're very far off, all correspond to dbl for some layer or for some layers, but the components for dw are quite close

Remember, different components of theta correspond to different components of b and w. When you find this is the case, then maybe you find that the bug is in how you're computing db, the derivative with respect to parameters b. And similarly, vice versa, if you find that the values that are very far, the values from d theta approx that are very far from d theta, you find all those components came from dw or from dw in a certain layer, then that might help you hone in on the location of the bug. This doesn't always let you identify the bug right away, but sometimes it helps you give you some guesses about where to track down the bug.

3. Remember the Regularization Terms

If you’re using regularization, include the regularization term in your gradient calculations. For example, if your cost function includes a regularization term such as:

$$J(\theta )=\frac{1}{m}\sum \text{losses}+\lambda \sum W_l^2,$$

then your gradient $d\theta$ should include this term. Omitting it can lead to incorrect gradients and, consequently, failed gradient checks

4. Gradient Checking with Dropout

Gradient checking doesn’t work well with dropout since dropout randomly deactivates different neurons in each iteration, making it impossible to have a stable cost function $J$ for consistent gradient checking. Instead, temporarily disable dropout by setting keep_prob = 1.0 during gradient checking. Once the gradients are verified, you can re-enable dropout to continue training.

5. Check Gradient After Initial Training

Occasionally, a gradient check may pass at initialization but fail later as the weights and biases increase. If gradient calculations are only accurate near initialization (when parameters are close to zero), inaccuracies may appear as the network trains. To avoid this, consider running a gradient check after some training iterations to verify accuracy at non-zero parameter values.

References