Description

So far, we have learned about the sum rule, the power rule, and the product rule. In this tutorial, we will be discussing our fourth and final tool for this module, which is called the chain rule.

In the intricate world of machine learning,  the chain rule stands as a fundamental principle, guiding the optimization of complex models and enabling them to learn from data. Understanding this mathematical concept is crucial for delving into the intricacies of neural networks and various machine learning algorithms.

Following this, our toolbox will then be sufficiently well-stocked that we'll be ready to start tackling some heftier, more interesting problems. Sometimes, we use functions as the inputs of other functions. As you can possibly imagine, describing this can get a little bit confusing. So, what we're going to do is give each of our functions meanings, so that we can hopefully keep track of what's going on.

Section 1- The Essence of the Chain Rule

The chain rule, a cornerstone of calculus, provides a framework for calculating the derivative of composite functions. A composite function is a combination of two or more functions, where the output of one function serves as the input for another. The chain rule allows us to break down the derivative of this complex function into manageable steps.

Section 2- The Chain Rule in Machine Learning

In the realm of machine learning, the chain rule plays a pivotal role in training neural networks. Neural networks, inspired by the structure of the human brain, consist of interconnected layers of neurons, each performing a specific computation. These computations involve transforming input data into meaningful outputs, such as predicting the probability of a certain class or generating creative text formats. To effectively train neural networks, we need to adjust the connections between neurons, known as weights, in a way that minimizes the error between the network's predictions and the true labels. This process is called optimization, and it relies heavily on the chain rule.

Chain Rule- The chain rule is a fundamental concept in calculus that allows you to find the derivative of a composite function. In other words, it tells you how to find the derivative of a function that is made up of two or more functions nested inside each other. The chain rule in calculus plays a crucial role in machine learning, such as gradient computation, backpropagation in neural networks, and understanding the relationships between model parameters and the loss function[2].

Section 3- The Chain Rule in Action: Backpropagation

Backpropagation, the heart of neural network optimization, utilizes the chain rule to efficiently compute the gradients of the loss function with respect to the weights of the network. Gradients provide information about how much each weight contributes to the overall error, allowing us to update these weights in a direction that reduces the error.

The chain rule enables the propagation of error backward through the network, layer by layer, ensuring that each weight is adjusted according to its contribution to the overall error. This iterative process, guided by the chain rule, leads to the refinement of the network's parameters, ultimately improving its ability to make accurate predictions.

Section 3- Chain Rule in Action

Consider the nested function h of p of m. We have the function h, which describes how happy I am, as a function of p, how many pizzas I've eaten that day. Then, the pizzas I get to eat per day is itself a function of m, which is how much money I make. So, we're still ultimately relating money to happiness, but via the concept of pizza. This nested function scenario comes up a whole lot in science and engineering, as you relate chains of concepts together.

And I suppose this particular example I've chosen here also gives you some insight into my priorities in life. So, first I'm going to give you the function relating happiness and pizza, which has the following polynomial form: h of p equals minus a third p squared, plus p, plus 1 over 5, which is easily understandable from a plot. What we can see is that, although without any pizza it's still possible to be happy, in principle, my peak happiness is with about one and a half pizzas. Any more pizza than this and I become less happy, and then, beyond about three pizzas, my happiness becomes rapidly negative.

Next is our function relating pizza and money. P of m equals e to the power of m minus 1, which is also fairly straightforward to understand by looking at a plot. If you've got no money, you can't buy any pizza. But the more money you have, your pizza purchasing power increases exponentially. As at first you can take advantage of bulk-buy discounts, but as you start getting really rich, you can buy your own pizza oven, and eventually even build your own pizza factory.

What we'd like to know is, by considering how much money I have now, how much effort should I put into making more, if my aim is to be happy? To work this out, we're going to need to know what the rate of change of happiness is, with respect to money, which is of course just dh by dm. For now, this relatively simple example, we could just directly substitute our pizza-money function into our happiness-pizza function, which would give us this,

and then differentiate this thing directly. However, the chain rule provides us with a more elegant approach, which importantly will still work even for complicated functions, where direct substitution like this may not be an option.

Consider the derivative of h with respect to p, and of p with respect to m. You'll notice that in this particular notation convention, where the derivatives are represented by quotients, the product of these two quantities looks like it would give you the desired function dh by dm.And in actual fact, this is a perfectly sensible way to think about what's going on. This approach is called the chain rule, because, in a sense, we are making a chain of derivative relationships.

Now, this is certainly not what you might describe as a formal derivation, but it is already good enough to enable you to make use of the chain rule effectively.

So, let's now apply this rule to our function. Firstly, let's differentiate our two functions, which give us dh by dp equals 1 minus 2 over 3p. dp by dm equals e to the power of m. And then multiplying these together is simple enough. However, we just need to remember that if we don't want pizzas, p, to appear in our final expression, then we just need to sub in our expression for p in terms of m. And then finally, by simply rearranging the terms, we recover the expression that we saw at the start of the video. So, dh by dm equals one third e to the m times five minus 2 e to the power of m. And there we have it. Now, don't forget, that although for this simple example the chain rule didn't save us a huge amount of time, compared to substituting in before the differentiation, don't let that fool you.

What's magic about the chain rule is that for some real-world applications, we may not have a nice analytical expression for our function. But we may still have the derivatives. So, being able to simply combine them with the chain rule becomes very powerful, indeed. Before we finish, let's have a quick look at our money-happiness function and its derivative on a graph. As we can see, if you're broke, and it's really worthwhile making some money, but the benefit of getting more, especially once you have enough pizza, decreases dramatically and quickly becomes negative. We now have a fourth and final tool for the module, and in the next video we'll be putting them all to use in an example.

References

2- Calculus that every Machine Learning Engineer should know